Closed nathanielrindlaub closed 2 months ago
I think that is a good idea. I have a similar design with my Lens to Salesforce workflow.
@ingalls, I was just made aware that the process of creating, updating, or deleting deployments on cameras with large numbers of images (like 20k) will timeout, and the deployment update will fail. This is because ProjectModel.reMapImagesToDeps()
has to iterate over every image in the camera to make sure it's assigned to the right deployment.
So once we get the task tracking in place I think our two priorities should be:
getStats
asyncreMapImagesToDeps
(or all deployment CRUD ops) asyncI think this is just about done. We've migrated getStats
, exportData
, exportImageErrors
, andcreate/update/deleteDeployments
to the new async task lambda. We decided against using the task pattern for batches
as they are much more involved and deeply integrated in the rest of the code base. It's worth, however, looking at whether we can now move some of the other time-consuming operations (Deleting large numbers of images, Deleting large numbers of labels, Merging large numbers of labels) to async and increasing or removing the allowable operation threshold.
Final punchlist:
getTaskFailure
action handler so it works for all task types.
We now have a growing number of potentially long-running tasks that Lambdas are not well suited to support, especially if the user is expecting a synchronous response. These include:
@ingalls recommends that we consider creating a consistent pattern for tracking these async tasks in the DB (creating a collection or collections that we update when the state of one of these processes changes and a consistent query pattern for accessing them). I think it's a great idea... right now we have 2 entirely different ways of checking the bulk upload state and the annotation state, and for most of the others we haven't yet implemented spinning the tasks off on separate infrastructure; instead, we only support them at pretty low thresholds.
For the tasks we haven't yet broken out to run async, we have a couple options: