search

oda-hub / gw-backend

0 stars 1 forks source link

reduce size of the skymaps (and hence the container image) #8

Closed volodymyrss closed 2 years ago

volodymyrss commented 2 years ago

needs https://github.com/oda-hub/nb2workflow/issues/26

volodymyrss commented 2 years ago

actually, I found another solution: gzipping all the maps. It reduces the size from 1.7Gb to 50Mb. It seems to work, it's here https://renkulab.io/gitlab/vladimir.savchenko/gw-backend and is deployed on staging. Please @dsavchenko verify if it works as intended.

Its rather transparent, since astropy, and even fitsio, can open gzipped fits directly.

dsavchenko commented 2 years ago

Great, it works! I couldn't even imagine the degree of compression possible, so didn't try to gzip.

volodymyrss commented 2 years ago

the files are mostly empty. But it's a bit extreme, yeah.

but, I can not query gw170814. gw170817 and others work. It takes long time to run the last cell, which "glues" the results to notebook. Is the result so big?.. Can it be reduced a bit?

dsavchenko commented 2 years ago

I've realized the problem. Previously, I made the sky maps for the GWTC-1 myself from the parameter estimation samples, and their unzipped size was below 1M. Then I found the official release of the skymaps https://dcc.ligo.org/LIGO-P1800381/public so decided to use it. But the sky maps there seem to be encoded as standard high-resolution Healpix, not MOC so the unzipped size is huge. The fits tables of the sky maps are returned by the backend, causing the problem. Possible fixes are:

  1. Get back to manually produced maps of GWTC-1
  2. Pass binary of gzipped fits from backend instead of table
  3. Convert official maps to MOC

I think the third option is the best, but I have to check, how to do it.

volodymyrss commented 2 years ago

Great, makes a lot of sense. Yes, it should be possible to convert them to MOC somewhat easily.

dsavchenko commented 2 years ago

https://renkulab.io/gitlab/dsavchenko/gw-backend/-/merge_requests/2

dsavchenko commented 2 years ago

Done

dsavchenko commented 2 years ago

@volodymyrss aren't Renku-based backends deployed automatically to staging? The problem persists after merging the fix to master in the Renku repo

volodymyrss commented 2 years ago

hm, it did not work. it's ok now, but I need to sort out this automation better.

volodymyrss commented 2 years ago

please confirm if it's ok.

dsavchenko commented 2 years ago

please confirm if it's ok.

Works now