Classification export CSV - process drawing subtasks similar to if individual task

mcbouslog commented 5 years ago

In the classification export CSV individual task annotations appear as follows (note *labels):

question task: {"task":"T2","task_label":"Individual question task","value":"Answer2"}
text task: {"task":"T3","value":"testing individual text task 2","task_label":"Individual text task"}
dropdown task: {"task":"T0","value":[{"select_label":"Individual Dropdown","option":true,"value":"c6f1890c69ab9","label":"feline"}]}

A drawing task with sub-tasks annotation is:

{
    "task": "T1",
    "task_label": "Explain what to draw.",
    "value": [{
        "x": 214.140625,
        "y": 257.04296875,
        "tool": 0,
        "frame": 0,
        "details": [{
            "value": 1
        }, {
            "value": "testing sub-task text 2"
        }, {
            "value": [{
                "value": "0806ccfc6feae",
                "option": true
            }]
        }],
        "tool_label": "Tool With Sub-tasks"
    }]
}

Highlighting the drawing sub-tasks (details property):

question sub-task: {"value":1}
text sub-task: {"value":"testing sub-task text 2"}
dropdown sub-task: {"value":[{"value":"0806ccfc6feae","option":true}]}

Could we run the drawing sub-task annotations (details) through the same processing they would get if they were individual task annotations?

relevant Panoptes code
examples are from this staging project/workflow

camallen commented 5 years ago

I just created a classification for that project (see extracted payload below). The sub task details aren't encoded in the payload at all, the only information we have on them is the implicit array ordering of the values from the details array.

Looking at the csv export code, i can't see how sub tasks are exporting correctly for other task types as well. I assume that any existing projects that are using using sub tasks are dealing with this somehow most likely on their own. Strange this hasn't come up before, i assume that folks don't really use sub tasks much?

To solve this we will need to match the sub-task details payload values to the matching task definition in the workflow subtask details array. Once we have the task definitions we can re-use the export code we have to handle the different task types.

{
    "classifications": [{
        "id": "96743",
        "annotations": [
                   ....
          {
            "task": "T1",
            "value": [{
                "x": 436.3262023925781,
                "y": 147.688232421875,
                "tool": 0,
                "frame": 0,
                "details": [{
                    "value": 0
                }, {
                    "value": "cat eye"
                }, {
                    "value": [{
                        "value": "0806ccfc6feae",
                        "option": true
                    }]
                }]
            }]
        }],
    }]
}

workflow task and sub task definition

{
    "workflows": [{
        "id": "3291",
        "display_name": "Individual-DrawingSubtask",
        "tasks": {
            "T1": {
                "help": "",
                "type": "drawing",
                "tools": [{
                    "type": "point",
                    "color": "#00ff00",
                    "label": "Tool With Sub-tasks",
                    "details": [{
                        "help": "",
                        "type": "single",
                        "answers": [{
                            "label": "answer1"
                        }, {
                            "label": "answer2"
                        }, {
                            "label": "answer3"
                        }],
                        "question": "Sub-task Question"
                    }, {
                        "help": "",
                        "type": "text",
                        "required": false,
                        "instruction": "Sub-task Text"
                    }, {
                        "help": "",
                        "type": "dropdown",
                        "selects": [{
                            "id": "1c5264d6aecb1",
                            "title": "Sub-task Dropdown",
                            "options": {
                                "*": [{
                                    "label": "sub-task cat",
                                    "value": "36d75a794d768"
                                }, {
                                    "label": "sub-task feline",
                                    "value": "0806ccfc6feae"
                                }, {
                                    "label": "sub-task deity",
                                    "value": "6680c83c8e84f"
                                }]
                            },
                            "required": true,
                            "allowCreate": false
                        }],
                        "instruction": "Sub-task Dropdown"
                    }]
                }],
                "instruction": "Explain what to draw."
            }
        }
    }]
}

trouille commented 5 years ago

Posting comments from a slack thread here, so that we continue the conversation and bring @mcbouslog as well:

@mcbouslog Could you share how NFN has been dealing with this in the past? Did they just add subtasks to their new workflows?

@camallen noted in the slack DM: "I’m wondering longer term if we should think about not using complex workflow structures but rather collaborative workflows and caesar instead (knowing that it isn’t easy to setup these either)."

@camallen Could you clarify what you mean by 'collaborative' workflows?

I think after Zach's TESS effort ends (early Feb), we'll want to have him explore solutions for this issue, before starting the CSSI work.

srallen commented 5 years ago

Our idea with the rewrite is to have the drawing sub-tasks to also adopt a similar structure to what we'll be using workflow steps for, so instead of nesting the full task within the drawing task, it'll just point to the task key as a reference:

"T1": {
  type: "drawing",
  tools: [{
    type: 'single',
    details: ['T2', 'T3']    
 }]
}

In this example, T2 and T3 would just be immediate children under tasks just like T1 is.

trouille commented 5 years ago

Good point Sarah.

@mcbouslog Do you know if there's a way for the NfN team to get the values manually for now knowing that there's a more systematic fix/change coming in 2019?

camallen commented 5 years ago

Could you clarify what you mean by 'collaborative' workflows?

I should have said 'Collaborative filtering workflows' e.g. camera catalogue, snapshots at sea, where answers from 1 workflows push data into another one. Not in the sense of the transcription projects.

Ali was working on a paper showing that it is more efficient, not sure if it's been published yet but here code is here https://github.com/aliburchard/SnapsAtSea

In this example, T2 and T3 would just be immediate children under tasks just like T1 is.

I like this idea and it's a much nicer solution. Could the subtask array key name be changed from details, perhaps to just subtasks? Noting that the old and new style of encoding the payloads / workflow subtask definitions will both need csv export solutions.

srallen commented 5 years ago

I like this idea and it's a much nicer solution. Could the subtask array key name be changed from details, perhaps to just subtasks? Noting that the old and new style of encoding the payloads / workflow subtask definitions will both need csv export solutions.

Yeah I like that renaming and it would make a clear delineation between the methods of definition. A drawing task with details would be the old version and one with subtasks would be the new one.

bw4sz commented 4 years ago

Can I get a recent summary of the subtask export in the classifications.csv? I have a drawing task that labels a bounding box by species, and then a subtask to assign a behavior to that box. In the export the annotations row looks like

[{"task":"T0","task_label":"Species","value":[{"x":803.00439453125,"y":506.8141174316406,"tool":5,"frame":0,"width":7.96014404296875,"height":15.994110107421875,"details":[{"value":[5]}],"tool_label":"White Ibis"},{"x":495.5355224609375,"y":221.092529296875,"tool":5,"frame":0,"width":8.74884033203125,"height":16.839523315429688,"details":[{"value":[5]}],"tool_label":"White Ibis"}]}]

I see a slightly mysterious "details" value json that i'm guess is what was mentioned above. Is this the subtask selection? I can hard-code the order, its the same for every bounding box.

lcjohnso commented 4 years ago

Can I get a recent summary of the subtask export in the classifications.csv?

I see a slightly mysterious "details" value json that i'm guess is what was mentioned above. Is this the subtask selection?

Subtasks annotations are currently passed via details field in annotations where question subtasks (as used in above example) yield the index of the selected answer(s).

Note that this export style will change significantly as a result of the new [Front End Monorepo]{https://github.com/zooniverse/front-end-monorepo} effort. For information about the annotation JSON structure for subtasks in rebuild, please see FEM ADR 25 and https://github.com/zooniverse/front-end-monorepo/issues/1315

zooniverse / panoptes

Classification export CSV - process drawing subtasks similar to if individual task #3001