Can't create Assignments with over 4000 Subjects

[shaunanoordin]

Issue: When creating an Assignment that consists of over 4000 subjects, the process fails.

Analysis:

• Creating an Assignment consisting of 4000 Subjects will cause the app to "pause" for 60 seconds before dying, with a response from the Education API saying 504 Gateway Timeout.
• Creating an Assignment consisting of 3000 Subjects will "pause" the app for 50 seconds before a "success" mess is returned.
• Creating an Assignment consisting of 100 Subjects will succeed after a 5 second pause.
• In short:
  • (1) if you have a large number of Subjects, it takes an inordinately large amount of time to submit the Create Assignment request to the Education API
  • (2) if your request exceeds 60 seconds, it dies.
  • (3) the cutoff appears to be at just above 3500 Subjects, but of course this depends on the connection and the Education API server availability.

Experimental results:

Subjects | OK? | Size* | Response 
---------+-----+-------+---------
100      | Yes |       |  4.90s
1000     | Yes |       | 20.95s
2000     | Yes |       | 38.08s
3000     | Yes | 129KB | 51.40s
4000     | NO  | 172KB | 60.00s - 504 Gateway Timeout
10000    | NO  |       | 60.00s - 504 Gateway Timeout

*Size of the Request body, if extracted as a .JSON file.

Question: Why does the submission process take so long? In theory an equivalent 129KB "file upload" shouldn't even take 50s to process.

Pinging @marten, @simoneduca and @eatyourgreens for your thoughts.

[marten]

I'd love to take a moment tomorrow to watch the logs while you're doing such a request. See if the bottleneck is the EduAPI or it's subsequent call to Panoptes. Maybe we need to perform some stuff in the background.

[shaunanoordin]

👍 Can do! I've got plenty o' Assignments waiting to be created.

[shaunanoordin]

Thanks to Marten, we have an update on this bug.

Problem: the Panoptes-side processing is a bottleneck.

Solution: The Education API needs to return a response quickly to the user (with new "status" variable saying "still processing..."), while simultaneously processing the rest of the subjects in the background.

Caveat: this means it'll take a short while before Teachers/Students can see all subjects, but this is acceptable for the Assignment use cases.

[shaunanoordin]

We've just received another report from Bridget confirming that teachers are encountering issues with Create Assignment functions. Strangely though, in the screenshots she provided, she encountered the issue's symptions when creating Assignments with only seventeen Subjects, not thousands.

[shaunanoordin]

This was fixed by zooniverse/education-api#33 and zooniverse/education-api#34 , but has a follow up bug in #305 . Welcome to the rabbit hole!