search

geminiplanetimager / gpi_pipeline

Gemini Planet Imager Data Reduction Pipeline
Other
8 stars 6 forks source link

Storing recipes into FITS headers is very slow for large numbers of files #68

Closed dsavransky closed 8 years ago

dsavransky commented 8 years ago

Right now, the pipeline stores the recipe into all FITS headers in memory, before starting to do reduction on any of them. If you're reducing a large batch of several hundred files, this can lead to tens of seconds at the start of a reduction where it's apparently doing nothing. There is no on-screen indication of why it's taking so long. This is an annoyance but not a bug.

It would be better to reorder this so that the header is updated one at a time, when the FITS data are loaded into memory that happens separately for each file. Better user experience, same result in the end.

semaphoreP commented 8 years ago

Who even reduces hundreds of FITS files at once anymore? Data Cruncher user experience does not need this improvement