Feature Request: efficient reading of large log files

HiPERCAM / hipercam

Python package for handling HiPERCAM data

4 stars 4 forks source link

Feature Request: efficient reading of large log files #40

Closed StuartLittlefair closed 6 years ago

StuartLittlefair commented 6 years ago

It would be nice if the HLog object supported lazy loading so that we could read in a large log file without using an enormous amount of memory.

This might take some redesign of the Hlog object though

trmrsh commented 6 years ago

I have altered the Hlog loading to reduce its memory footprint. Not exactly what you were looking for, but it should help. For really large files, I will at some point make a converter to FITS with one HDU per CCD. Again not perfect, but it would have to be fairly extreme to make these too large. Closing for the moment.