search

LM-SAL / aiapy

Python library for AIA data analysis
https://aiapy.readthedocs.io/en/stable/
BSD 3-Clause "New" or "Revised" License
7 stars 3 forks source link

Saved prepped maps are very large #52

Closed nabobalis closed 10 months ago

nabobalis commented 4 years ago

In GitLab by @dstansby1 on Apr 16, 2020, 03:13

I'm currently trying to prep a bunch of maps, and then save them to .fits files so I don't have to keep prepping them. I'm calling the following functions on a map:

smap = fix_observer_location(smap)
smap = update_pointing(smap)
smap = correct_degradation(smap)
smap = register(smap)

then saving the resulting map to file using smap.save('mymap.fits'). Starting with a L1 193A file, which is 11 MB big, the resulting saved file ends up being 134.2 MB big, and I wanted to check if there is any way to reduce this filesize.

My initial investigation shows that the L1 map is 16 bit integer, and the registered map is 64 bit floating point, which might explain the large difference filesize.

nabobalis commented 4 years ago

In GitLab by @Cadair on Apr 16, 2020, 03:17

The input files are RICE compressed as part of the FITS standard. I would imagine saving them as compressed would reduce the file size, you can do this from SunPy Map and astropy http://docs.astropy.org/en/stable/io/fits/usage/unfamiliar.html#compressed-image-data

I would say that unless you need to copy those files around, you are going to get a lot better performance if you don't compress them.

nabobalis commented 4 years ago

In GitLab by @Cadair on Apr 16, 2020, 03:18

You can set the hdu_type in GenericMap.save to save it as a compressed HDU: https://docs.sunpy.org/en/stable/api/sunpy.map.GenericMap.html#sunpy.map.GenericMap.save

nabobalis commented 4 years ago

In GitLab by @markcheung on Apr 16, 2020, 08:47

Level 1 images are saved as RICE compressed FITS, as @Cadair already mentioned. Pixels has integer values and compressed really well. After prepping it's stored in memory as float64 which is 4 times larger, and don't compress as well. So even when you save as a compressed HDU it's likely a few times larger than the original FITS file. I don't think you'll lose a lot by converting to float32 first before saving.

nabobalis commented 4 years ago

In GitLab by @markcheung on Apr 16, 2020, 08:49

134.2 MB seems a few MB larger than I'd expect. Is the metadata in the Map object so big?

nabobalis commented 4 years ago

In GitLab by @dstansby1 on Apr 16, 2020, 14:56

I guess the data should come in at 4096 4096 64bits / 8bits/byte / 1024 / 1024 = 128 MB, so there's ~6MB of metadata hanging around? Does seem like a lot... Anyway, thanks for the replies and suggestions all!

nabobalis commented 4 years ago

In GitLab by @dstansby1 on Apr 16, 2020, 14:56

closed