Docker image with home directory tar

superstar54 commented 1 month ago

Docker image with home directory tar

When the user launches the container, a lot of time is spent on preparing the environment, setting up codes, pseudopotentials, etc. In order to reduce the launch time of the container, we tar already prepared home directory and keep it in the image, and then untar it when the container is launched. This way, the user can start working with the container immediately.

To build the image, you can use the following command:

docker build -t aiidalab/qe-tar-home .

To run the container, you can use the following command:

docker run --rm -it -p 8888:8888 aiidalab/qe-tar-home

To compare with the image without the home directory, you can use the following command:

docker run --rm -it aiidalab/qe:amd64-latest /bin/bash

Image size and time

Test in my laptop:

Type	Data Size	Start Time
Standard (aiidalab/qe:amd64-latest)	3.9 GB	60 s
home tar (Compress)	5.2 GB	12 s
home tar (Not compress)	6.4 GB	6 s

superstar54 commented 1 month ago

Hi @giovannipizzi , this is the PR for the new image with home tar. The time indeed reduced a lot!

The only problem is that when deploying the image on the demo server, it has a timeout issue, and we are working on it.

giovannipizzi commented 1 month ago

Thanks! Did you compare time for the first start and for the second start (when untarring is not necessary)? For the timeout maybe the image is too big? How big is it before (the standard aiidalab one) and after?

superstar54 commented 1 month ago

Hi @giovannipizzi . The comparison of size and time is shown in the description of the PR. I copied it here:

Image size and time

Test in my laptop:

Type	Data Size	Start Time
Standard (aiidalab/qe:amd64-latest)	3.9 GB	60 s
home tar (Compress)	5.2 GB	12 s
home tar (Not compress)	6.4 GB	6 s

I tested both compressed and uncompressed images; all showed timeout errors. I also increased the timeout from 5 minutes to 10 minutes, but I had no success. It could be that Kubernetes handles the home folder incorrectly. We need to investigate more.

giovannipizzi commented 1 month ago

Ok sorry, on the phone I didn't see properly the table. For the time, however, this I guess is the time for the first startup when the directory has to be untarred, right? I was curious to see the time of the second and following startups, where only the services (psql, rmq, aiida daemon) need to be restarted, but the data is already in the home. Should be even shorter right? For k8s, the new size is not way bigger than before, so maybe the timeout problem is due to something else?

danielhollas commented 1 month ago

Btw if you want to to speedup the compression / uncompression, you can try decreasing the compression level. (or switching the compression algorithm for a faster one)

giovannipizzi commented 1 month ago

Btw if you want to to speedup the compression / uncompression, you can try decreasing the compression level. (or switching the compression algorithm for a faster one)

Indeed - what are you currently using? GZ, BZ2, ...? GZ is faster, and I'd try with the lowest level (1, on a scale 1-9 - 0 means no compression IIRC). Then we can discuss if the time loss is worth the space reduction or not

unkcpz commented 2 weeks ago

Thanks a lot!! @superstar54

I fix the deployment on Azure, the problem is the image name you set was not correct. The name should be superstar54/qe-tar-home-z and tag is latest, instead of superstar54 and qe-tar-home-z as tag. Please see this commit [1].

However, the startup still goes to pseudo libraries setting and it still takes 2 mins. I did a test with the image and run with the docker locally, and I didn't see the speed up as well. So I think it is rather a image problem not just for k8s deployment.

Can you have a look at it? Please make sure you didn't use aiidalab-launch to test since it will create a volume and the next time it will skip the home set.

The way I used to test is by always put --rm -it to run from image to make sure it start a new container. Or maybe you have a new image build but not pushed to the docker.io?

[1] https://github.com/aiidalab/aiidalab-demo-server/commit/d412b9e786cd13fd7a849287fd97f72e84724773 NOTE! this change will be revert back to use the official image from QeApp. It is now only for test purpose.

superstar54 commented 1 week ago

Thanks for the fix! @unkcpz .

However, the startup still goes to pseudo libraries setting and it still takes 2 mins. I did a test with the image and run with the docker locally, and I didn't see the speed up as well. So I think it is rather a image problem not just for k8s deployment.

I forget the push the latest version to the docker hub. I will try to use the github action to do this automatically. Will let you know when the new image is ready to test.

superstar54 commented 1 day ago

I updated the image, and tested on https://staging-demo.aiidalab.io:

Type	Data Size	Start Time (demo server)	my laptop
Standard (aiidalab/qe:amd64-latest)	3.9 GB	2 min 35 s	60 s
home tar (Compress)	4.36 GB	38 s	12 s

aiidalab / aiidalab-qe