Workflow Assignment: if current assigned workflow is complete, locked/unlocked split fails

[lcjohnso]

Package

app-project

Describe the bug

Multiple Gravity Spy volunteers reported that leveling up was no longer working in two ways:

1. While they previously had leveled up and unlocked multiple workflows, only the default (Level 1) workflow was unlocked.
2. Even when they tried clicking on Level 1 to start classifying, the classifier would not load.

Workflow assignment (aka: leveling up) relies on the assignment of a user according to workflow_id (in UserProjectPreferences.settings), looking up that assigned workflow's level (in workflow.configuration), and splitting the project's active workflows into locked vs. unlocked based on whether each workflow has a level value that is less than or greater than the level of the assigned workflow.

What appears to be happening: when a user's current assigned workflow (e.g., GS 2.0 - Level 3, ID=24189, with workflow.configuration.level: 6) is complete (100% retired), that workflow is not loaded by the fetchWorkflowData() function. As a result, assignedWorkflow is undefined in WorkflowSelectButtons, the locked/unlocked split can not be determined, and the code falls back to default settings (only Level 1 unlocked; same as if user was not logged in and had no UserProjectPreferences entry).

To Reproduce

Steps to reproduce the behavior:

1. Login as zootester1, who has been assigned UserProjectPreference with 'settings':{'workflow_id': '24189'} (aka: Gravity Spy 2.0 - Level 3)
2. Visit Gravity Spy landing page: https://www.zooniverse.org/projects/zooniverse/gravity-spy
3. Observe that only Level 1 is unlocked, though Levels 1-4 & GS 2.0 Levels 1-3 should all be unlocked.
4. Try to enter Level 1; observe that the classifier does not load

Expected Behavior

Even if the top / assigned level of a user is complete, and therefore that workflow is not accessible (i.e., its button is not shown on the project landing page), the locked/unlocked levels should still be assigned correctly and the user should be able to load and classify on all unlocked workflows.

Possible Solutions

Add a fallback call of fetchSingleWorkflow() as part of WorkflowSelectButtons in cases where:

1. assignedWorkflowID exists (workflow_id is set in UPP.settings)
2. the assigned workflow is not found in the workflows array, which contains workflows that are active + not complete, or individually requested via URL
3. the assigned workflow exists, is linked to the project, and is active (i.e., is in project.links.active_workflows)

Basically, load the assigned workflow so we can pull the level value from workflow.configuration, but otherwise leave the array of loaded workflows (and therefore dictating what workflow buttons will appear on the project landing page) unchanged.

There are some similarities to https://github.com/zooniverse/front-end-monorepo/issues/6188 as they both center around what workflows are fetched and loaded.

[goplayoutside3]

Looking into this bug, and can confirm the described issue where zootester1’s workflow selector shows only workflow 1610 (level 1) as unlocked. By clicking the button I can’t access the classifier either.

Just to make sure I’ve got a good understanding of the issue since I’m not very familiar with the setup process for levels:

1. Workflow 24189 is configured as level 3.
2. Volunteers unlocked that level, but did not unlock level 4 yet.
3. Workflow 24189 became 100% complete (retired).
4. A different workflow (1935) is now configured as level 3.
5. Gravity Spy would like volunteers who unlocked level 3 via workflow 24189 to be shown workflow 1935 instead?
   • The conclusion of this list is where I’m a little confused. Hindsight is always 20/20, but during development of the leveling-up feature why wasn’t the level integer saved in a user’s project preferences instead of a workflow id? By recording a specific workflow id, it throws a wrench in project teams who may want to re-assign levels to different workflows, or in this case, switch out one workflow for another at a certain level. What’s the intended behavior if project team deactivates a workflow that volunteers are currently assigned?

Solution:

When the WorkflowSelectButtons component renders, on the project home page or classify page, assignedWorkflowID is an id that doesn’t exist in the list of workflows fetched on page load. The ideal solution would be if the level integer was saved in project preferences, workflows could be sorted based on level, but since that’s not the case I agree we’d like to fetch the config of the workflow assigned to the user. However, I don’t think it’s as simple as fetchSingleWorkflow() in the selector buttons.

If a user navigates directly to a specific workflow, the classify page also double checks if the user is allowed to see it. This code would also need modification and allow for project teams to switch out one workflow for another at a certain level. https://github.com/zooniverse/front-end-monorepo/blob/cb60f0eb7e09dcd92c549cb28d8805241860018d/packages/app-project/src/screens/ClassifyPage/ClassifyPageContainer.js#L26-L30

I’m leaning more toward checking if a project has workflow assignment enabled, and then fetching all active workflows on page load regardless of completion status, but again what’s the intended behavior if the project team deactivates a workflow that volunteers are currently assigned?

[goplayoutside3]

Wait, now I'm even more confused. @lcjohnso in the solution where fetchSingleWorkflow() is used to get workflow.configuration.level for a volunteer's assigned workflow, for zootester1 this means workflow 24189's level is actually 6. The frontend code can't assume the unlocked level of zootester1 is supposed to be 3 🤔 I'm doubly confused about what the intended behavior is for Gravity Spy in this scenario.

[goplayoutside3]

I have a draft PR https://github.com/zooniverse/front-end-monorepo/pull/6201 open that solves this Issue, as long as my comment above about level 6 confusion is cleared up. The PR essentially checks if assignedWorkflowID exists, and always fetches data about the workflow's config because sometimes a workflow will retire (or be deactivated?) when a volunteer is assigned to it.

[lcjohnso]

The missing detail is due to Gravity Spy's complex level setup.

24189 is "GS 2.0 Level 3", but level 6 overall. Here is the current level setup:

# Level WorkflowID WorkflowDisplayName
1 1610 level 1: Neutron Star Mountain
2 1934 level 2: Galactic Supernova
3 1935 level 3: Binary Neutron Star Merger
4 7765 level 4: Neutron Star-Black Hole Merger
4 24187 GS2 level 1
5 24188 GS2 level 2
6 24189 GS2 level 3
7 7766 level 5: Binary Black Hole Merger
8 7767 level 6: Inflationary Gravitational Waves

Note: GS 1.0 Level 4 and GS 2.0 Level 1 are "Level 4" so that both levels are unlocked at that point.

And here is the corrected description of the desired outcome:

1. Workflow 24189 is configured as level 6.
2. Volunteers unlocked that level, but did not unlock level 7 or 8 yet (i.e., GS 1.0 level 5: Binary Black Hole Merger or GS 1.0 level 6: Inflationary Gravitational Waves).
3. Workflow 24189 became 100% complete (retired).
4. ~A different workflow (1935) is now configured as level 3.~
5. Gravity Spy would like volunteers who unlocked level 6 via workflow 24189 to be shown levels 1-5 as unlocked -- same as if level 6 = 24189 was Active+NotComplete. In this case, volunteers can continue working on GS 1.0 "level 4: Neutron Star-Black Hole Merger" (WorkflowID=7765, level = 4), and if they contribute enough, their classifications performance could unlock GS 1.0 "level 5: Binary Black Hole Merger" (WorkflowID=7766; level = 7).

Additional responses:

During development of the leveling-up feature why wasn’t the level integer saved in a user’s project preferences instead of a workflow id? By recording a specific workflow id, it throws a wrench in project teams who may want to re-assign levels to different workflows, or in this case, switch out one workflow for another at a certain level.

My guess: it was deemed more important to be able to shift or rearrange existing workflows and levels without needing to edit individual UserProjectPreferences entries. For example: when the three GS 2.0 levels were added, all we needed to do was activate new workflows and reassign workflow.configuration.level values, as opposed to needing to edit thousands of individual UPP entries to reassign level index values. Neither strategy is perfect, but this one has proven to be useful.

What’s the intended behavior if project team deactivates a workflow that volunteers are currently assigned?

This is a clear weakness of the Workflow Assignment feature: currently, we assume that all workflow levels will remain active and not complete indefinitely. If a level is completed and no longer available for classification, the level-to-level promotion chain may be broken. I don't think there's a good answer to your question -- users assigned to deactivated workflows will need to be reassigned (this is possible via fetch_settings() and save_settings() Python Client functions).

In this specific case, 24189 lies at the top of the GS 2.0 series and was a "terminating" level, in that volunteers are never promoted onward from that workflow. Therefore, users who had leveled up into GS 2.0 Level 3 (WorkflowID=24189) can either continue working on any of the unlocked levels, or work toward promotion to GS 1.0 "level 5: Binary Black Hole Merger" (WorkflowID=7766; level = 7).

[goplayoutside3]

Thank you Cliff! That clears things up and confirms that PR #6201 is the correct fix. That PR branch is deployed to https://fe-project-branch.preview.zooniverse.org/projects/zooniverse/gravity-spy, so feel free to test Gravity Spy using that url. I'll request immediate review from the frontend team so we can get the fix deployed this week.

[crowston]

I agree with the intended behaviour: if the workflow a user is on is not available, they should still see all of the previously unlocked levels.

Hmm, another edge case: suppose level 1 becomes unavailable: where do new users go? Presumably to level 2?

[goplayoutside3]

The fix for blocked volunteers assigned to 24189 is deployed 👍

@crowston Re:

another edge case: suppose level 1 becomes unavailable: where do new users go? Presumably to level 2?

At the moment, if the workflow where configuration.level = 1 retires, volunteers who are brand new to the project would not be able to classify anything. This discussion makes me wonder if for leveling-up projects specifically, all workflows buttons should remain visible even if 100% complete? (While still retaining locked/unlocked behavior)

[crowston]

Yay, thanks!

Maybe if a new volunteer arrives they should be put in the lowest unlocked level, which would usually but not always be level 1?

[lcjohnso]

Current practice: in the case that the Level 1 workflow (configuration.level: 1) retires, the Level 2 workflow should edit its configuration.level parameter to also equal 1. That way there is a not-complete Level 2 workflow that is automatically unlocked.

[crowston]

Makes sense though it needs someone to notice and to make a manual change.