search

aiidateam / aiida-prerequisites

Docker image that contains all prerequisites allowing to run AiiDA.
2 stars 2 forks source link

size increase after upgrade to ubuntu 20.04 #35

Open ltalirz opened 3 years ago

ltalirz commented 3 years ago

I notice that the size of the aiida-prerequisites docker image increased from 460MB to 569MB for version 0.4.0 (both compressed).

Part of it may come from the increased size of the phusion baseimage (which we can't do much about): bionic-1.0.0: 54MB focal-1.1.0: 80MB

However, several other steps in aiida-prerequisites also take more space, see 0.4.0 layer size vs 0.3.0 layer size

E.g. the main apt-get install line results in a ~70MB larger layer than in the previous image.

We should keep an eye on this and possibly start thinking about where we could shave something off.

P.S. We see a similar size increase in the aiida-core docker image (from 841MB to 993MB)

yakutovicha commented 2 years ago

So, I looked at the sizes and what I noticed that for the release 0.5 the image size went magically back to 460 MB. The only sensible difference between 0.4 and 0.5 is the Ubuntu version (20.04 vs 22.04). So not sure what to say here..

yakutovicha commented 2 years ago

At the same time, the next version of aiida-prerequisites will gain some weight due to the use of separate conda environments for PGSQL and RMQ.

The increase for arm64 architecture is ~450 -> ~520, while for amd64 is going to be a bit more ~460 -> 630. Fix of https://github.com/aiidateam/aiida-prerequisites/issues/58 will allow to reduce the size of amd64 image a little bit.

However, the biggest question (and compromise) here is whether to use separate environments for PGSQL and RMQ. It gives more flexibility, but it gives more size. I have no answer to this, just an open thought.

yakutovicha commented 2 years ago

Of course, the hope we have is that docker-compose approach will solve all the problems :)

ltalirz commented 2 years ago

Of course, the hope we have is that docker-compose approach will solve all the problems :)

As far as a user-level aiida-core deployment is concerned, I guess it already does (e.g. pg+rmq or just rmq)

As for the AiiDAlab, I wouldn't bet that docker-compose support at the JupyterHub level will ever come to pass. As discussed here the model they suggest for this scenario is to run a service separate to the JupyterHub, which the JupyterHub containers can connect to and request resources from using their JupyterHub token. I.e. for us this would be a rabbitmq / postgres service where a JupyterHub container can come and request a queue / a database. This service would be managed separately from the JupyterHub.

yakutovicha commented 2 years ago

As for the AiiDAlab, I wouldn't bet that docker-compose support at the JupyterHub level will ever come to pass.

We are testing a kubespawner for this purpose. It supports "extra_containers" option that will allow us to run multiple containers in the same pod. So we are hoping to have "pgsql" and " rabbitmq" as those optional containers.