search

lusee-night / luseepy

A set of python utilities for performing various LuSEE Night related calculations.
MIT License
0 stars 1 forks source link

Add packages to docker image #2

Closed slosar closed 2 years ago

slosar commented 2 years ago

Add the following to docker image: ARES - https://ares.readthedocs.io/en/latest/index.html PERSES - https://github.com/CU-NESS/perses fitsio healpy

slosar commented 2 years ago

Make sure those maps mentioned in perses are available (i.e let the image creation download them)

buddhasystem commented 2 years ago

Would it make sense to keep the "base" image with just atropy and lunarsky around, or is it useless without these other packages?

slosar commented 2 years ago

The question is why would anyone use the base package when there are others available with bells and whistles? I would like to prevent a proliferation of images (unless you give me a good argument)

buddhasystem commented 2 years ago

It depends on the use case and the size. If it's all tiny and builds fast, then it's not an issue. If it becomes a chore to build and large in size, this may impact the development (in the former case) and reduce performance if deployed at scale since worker nodes will be pulling images repeatedly. The issue really comes to head with Singularity. It is slow to start Docker containers -- it is building it's own layers based on Docker. It also tends to quickly fill up disk space with its cache. So if Singularity is in the picture (which I believe it should be if we want to use batch facilities at BNL -- eventually) this may be a concern.

If the scenario is that there is just a person on a decent machine running notebooks then it's perhaps not an issue. In PHENIX, I built tiny images containing a legacy version of ROOT, for compatibility, to just quickly go over data files and make plots.

But you know, we can just skip these details for now and revisit them later, so yes, let's stick with the minimal set.

NB. I noticed that on some Python platforms fitsio won't have a valid wheel and will try to compile C when building an image. I don't like that. That was the reason I started updating my Python and accidentally nuked 3.5.

slosar commented 2 years ago

Ok, fitsio will sooner or later become a core dependency; fits is a standard format in astrophysics (https://en.wikipedia.org/wiki/FITS) healpy will also likely become core.

I think astropy and is a big guy, everyone else is kinda small, so whether we have them or not should not make too much of a difference? I suggest a single image for now and then if we decide it is too big we think about how to slim it down to core depencies for running on a clustert.

buddhasystem commented 2 years ago

I agree.