search

swagnercarena / paltas

Conduct simulation-based inference on strong gravitational lensing systems.
MIT License
15 stars 12 forks source link

Cache consolidated COSMOS catalog #4

Closed JelleAalbers closed 3 years ago

JelleAalbers commented 3 years ago

Initializing the COSMOSCatalog takes a bit of time, because it loops over all images to find their pixel sizes.

This lets COSMOSCatalog save its catalog to a ~30 MB file in the COSMOS directory (with ~6 GB other files). When COSMOSCatalog is loaded a second time on a machine, it just loads that one file directly.

For me, this speeds up the catalog initialization from ~24 seconds to ~6 milliseconds. (Except for the first time you load it, of course.)

swagnercarena commented 3 years ago

The changes being developed on the branch sampling include this update. I was just about to merge them to main. I'll assign you as a reviewer so you can make sure this matches what you had in mind.

JelleAalbers commented 3 years ago

Ah yes, I see: https://github.com/swagnercarena/manada/blob/sampling/manada/Sources/cosmos.py#L53. Looks like we had the same thought :-)