search

lsst-epo / citizen-science-notebooks

A collection Jupyter notebooks that can be used to associate Rubin Science Platform data to a Zooniverse citizen science project.
3 stars 1 forks source link

Retry 10k synchronous and asynchronous transfers and assess kernel performance and processing time #112

Closed ericdrosas87 closed 1 month ago

ericdrosas87 commented 1 month ago

Enough time has passed since I last tried to run 10k cutouts through the butler service and lsst.afw.display module that it's worth retrying to see if any recent updates to either software package provide material progress on the issues I've been facing.

Questions worth reassessing:

  1. Is it possible to run 10k images through the butler and lsst.afw.display synchronously without the kernel dying? If the kernel doesn't die how long does it take?
  2. Is it possible to run 10k images through the butler and lsst.afw.display asynchronously using four processes or threads without the kernel dying? If the kernel doesn't die how long does it take to complete?

These questions need to be answered before I start working with CST to find a clear path forward - either using the Image Cutout Service or making do with the original workflow (butler + lsst.afw.display)

ericdrosas87 commented 1 month ago

Currently, the butler takes about ~7 hours to produce ~3915 cutouts before the kernel crashes due to the memory limit being reached. I wasn't able to track how long it took just the butler to do its part because the kernel kept crashing and the timer would keep going.