search

ImageMarkup / isic-archive

International Skin Imaging Collaboration: Melanoma Project
https://isic-archive.com/
Apache License 2.0
47 stars 10 forks source link

Build TIFF and superpixel files asynchronously #159

Closed brianhelba closed 8 years ago

brianhelba commented 8 years ago

Image.createImage should create new girder_worker jobs for both TIFF generation (which can just be done via the large_image plugin) and superpixel generation. At the moment, both processes running synchronously is causing the upload of new datasets to take way too long.

However, this means adding another software dependency, and it will make the install process more complicated, since the worker plugin must be enabled and setup manually. Eventually, we can fully automate this process, but it's going to require us to get the Girder Ansible module upstream. In the meantime, we should ensure that the provisioning installs at least girder_worker and that the plugin setup is fully documented.

Once this infrastructure is set up, we could reconfigure our deployment to spin up additional worker machines only when a new dataset is expected to be uploaded. This would increase the new dataset processing throughput, but allow us to use a lower-cost machine configuration for the rest of the time.

brianhelba commented 8 years ago

@aadikalloo Once this is in place, we should test the dataset upload process with real datasets of significant size.