Improve UI and workflow for GMF confidence modifiers

[npit]

While annotating an incident with GMF, reading early listed reports may lead to assigning a 'known' or 'potential' classification, along with relevant snippets and discussion. Subsequent reports however (especially ones published at a much later date, where additional data may be available) may change that confidence modifier.

Our current UI setup for confidence levels makes that change tedious. E.g. converting the assigned label(s) L from known to potential, you have to manually:

• delete L from the 'known' section, navigate to the 'potential' field and recreate them
• Cut-paste the classification discussion text relevant to L from the 'known' to the 'potential' section.
• For each snippet with one or more labels in L:
  • If the same snippet doesn't exist in the 'potential' section, copy the snippet text and discussion from the 'known' section. Else, locate the existing 'potential' snippet.
  • Assign the created 'potential' labels L to the snippet.
  • Back in the 'known' section, if the snippet was only labelled with L, delete it. If not, only remove the L assignments.

To vastly simplify the above, I propose the following UI refactor:

• Reduce the UI to a single, confidence-agnostic set of elements, one for each G/M/F category
• Set a default confidence modifier (e.g. known), which is applied to each new label assignment
• The user may switch the confidence modifier by interacting with the label graphical box. E.g. clicking it toggles the confidence from known -> potential (or cycles / shows a dropdown, if more than two modifiers are available in the future).
• Switching the confidence level somewhere in the annotation form updates the confidence of that label assignment for the incident. E.g. the user may do so at the incident labels or snippet relevant labels, to an equivalent effect.
• Visual hints (e.g. font / background color, super/sub scripts, etc.) on the label box indicate what confidence level it represents. Changing a confidence level updates the label visuals everywhere that label appears.

Example: the incident has a single UI for 'methods'. It is classified with a default (=known) Language Modeling label, and a potential Regression label. Label box subscripts indicate each label's confidence modifier.

The above setup reduces the task of converting a label confidence to a single click (if only two levels exist, as is the case currently), and enables future support fore more confidence levels with no changes to the GMF annotation UI.

[lmcnulty]

I think this may call for a data model change rather than just a different UI. I can think of two basic approaches for how we would store Known/Potential as modifiers for array item values.

Option 1 – Have some kind of "modifiers" field that can reference existing field values and apply modifiers like confidence levels. Pro: We would be able to query GMF classifications easily and without changes. Con: It will be tricky to make the UI comfy.

attributes: [
  {
    field_name: 'AI Goals',
    value_json: '["Content Search", "Classification"]'
  },
  { 
    field_name: 'Field Modifiers', 
    value_json: `
      [
        {
          "field_name": "AI Goals",
          "value": "Content Search", 
          "modifier": "Known"
        },
        {
          "field_name": "AI Goals",
          "value": "Classification", 
          "modifier": "Potential"
        },
      ]
    `
  }
],

Option 2 – Make Goals, Methods, and Failures into object-lists like the snippets currently are. Pro: We could mostly reuse the existing object-list UI. Con: Querying GMF classifications would be tricky, this would require changes to checklists.

attributes: [
  {
    field_name: 'AI Goals',
    value_json: `
      [
        { 
          "goal": "Content Search", 
          "confidence": "Known"
        },
        {
          "goal": "Classification",
          "confidence": "Potential"
        }
      ]
    `
  }
],

I think my preference is for (1) even though it's less clean, because (2) will make things difficult on the checklists side.

[lmcnulty]

In Slack, Nikiforos wrote:

[G]iven that you're using keyword-ey fields for the JSONs, are these schemas fixed? If so, why is that? Can't we use whatever schema fits our use case and convert it as we like for the UIs?

We need a fixed schema because one is enforced by the GraphQL server – we can't just query arbitrary fields. We can (and in some places do) stringify arbitrary JSON, but that makes it difficult to query from the contents. We also want all of the taxonomies to share some basic structure so we can have them all work the same way in e.g. the spatial visualization, the search UI, the checklists UI, etc. We could keep the Known vs Potential items separate and paper over it in the UI, but that separation is also troublesome in places where we use the taxonomy labels, so I'd prefer to change the data model rather than just the UI.I think we would benefit from discussing in the GitHub thread where others can see and provide their opinions. Is it ok with you if I quote these last messages there?