download size of qm 20.11.2a

[ltalirz]

@chrisjsewell I see that QM size increased from 3.4GB (9.9GB extracted) at 20.6.1 to 4.4GB (13.2GB extracted) at 20.11.2a. I.e. it takes an extra 3.3GB on disk.

Do you happen to know why?

[chrisjsewell]

Well the distribution has increased from 6 simulation codes to 8, plus siesta now has more dependencies, plus an extra python virtual environment for jupyter.

20.6.1:

• Quantum Espresso 6.5
• Yambo 4.5.1
• fleur 0.30 MaX4
• siesta v4.1-rc1
• cp2k 7.1
• Wannier90 3.1.0

20.11.2a:

• Quantum ESPRESSO 6.5
• Yambo 4.5.2
• fleur 0.30 MaX4
• cp2k 7.1
• Wannier90 3.1.0
• siesta MaX-1.2.0
• bigdft 4855f2cd
• abinit 9.2.1

But obviously if you can identify any ways to "trim the fat" that would be great 😄

[chrisjsewell]

Note the "Disk Usage Analyzer" application is included in the VM, so can be used to look at this.

A number of things of note:

• There are some (identical) generic clean-up tasks in roles/marvel-nccr.simulationbase/tasks/cleanup.yml and roles/marvel-nccr.ubuntu_desktop/tasks/cleanup.yml, but these feel like a bit of an odd place for them (and easy to overlook). I would just remove this and add a separate playbook with clean tasks including:

  • Clear apt cache with apt-get clean and rm -rf /var/lib/apt/lists/*
  • Clear pip cache: pip cache purge (but should be taken care of by removing ~/.cache)
  • The other things that clean=True are used for is in roles/marvel-nccr.wannier90/tasks/clean.yml and roles/marvel-nccr.quantum_espresso/tasks/main.yml, but they can be removed because they are cleaning things in the /tmp folder that gets removed anyway, and in roles/marvel-nccr.quantum_mobile_customizations/tasks/folders.yml to clean .pyc files in the (empty) examples folder which is also not necessary

• Remove Python 2.7 install (210 Mb in /usr/lib/python2.7). It's installed in roles/marvel-nccr.ansible_prerequisites/tasks/main.yml, but I imagine this is no longer required by ansible. In fact I'm not sure if ansible_prerequisites is required at all now.

• The jupyter virtual env can be reduced by ~500Mb with jupyter lab clean (see jupyter lab docs)

• The biggest stuff in the aiida virtual env are bokeh and plotly, which come from the aiidalab install

• The binaries take up 1.6 Gb, I imagine this is unavoidable:

• The psuedopotentials take up >300 Mb: the downloads made in roles/marvel-nccr.aiida/tasks/aiida-pps.yml to the aiida folder can be removed, once they are imported to reclaim ~80Mb (actually moving the downloads to /tmp, or also moving to use of aiida-psuedo). Then the question is whether we really need to also have the "raw" psuedo files available - downloaded in roles/marvel-nccr.quantum_espresso/tasks/pps.yml and roles/marvel-nccr.siesta/tasks/psml-pps.yml - or if they are just stored as families in the aiida database. I imagine you could want them, if you were just using the codes without aiida, but do we need to account for that.

• Also, minor thing, can do e.g. sudo journalctl --vacuum-size=2M(https://ma.ttias.be/clear-systemd-journal/)

[ltalirz]

thanks for looking into this!

Then the question is whether we really need to also have the "raw" psuedo files available

I think we want to keep them. QM is not just for AiiDA use.

[ltalirz]

By the way, if we were to use conda to install cp2k, qe, siesta, ... some dynamically linked libraries could be reused. E.g. the cp2k binary from conda-forge is just 38MB instead of 304MB

[chrisjsewell]

how much space does Conda take up in general?

[chrisjsewell]

Remove Python 2.7 install (210 Mb in /usr/lib/python2.7). It's installed in roles/marvel-nccr.ansible_prerequisites/tasks/main.yml,

Implemented in: https://github.com/marvel-nccr/quantum-mobile/commit/d27a90d2bb317d6c551e6535a799a4430fad08ae

The psuedopotentials take up >300 Mb: the downloads made in roles/marvel-nccr.aiida/tasks/aiida-pps.yml to the aiida folder can be removed, once they are imported to reclaim ~80Mb

Implemented in: https://github.com/marvel-nccr/ansible-role-aiida/commit/cbb17a5c330fcd433e0d05006013c8458f320cb2

The jupyter virtual env can be reduced by ~500Mb with jupyter lab clean

Implemented in: https://github.com/marvel-nccr/ansible-role-aiidalab/commit/7403438bc290b1e105e818afc698869ea7e4bce0

With all these in place I have created https://quantum-mobile.readthedocs.io/en/latest/releases/both/21.03.18-qe.html, from https://github.com/marvel-nccr/quantum-mobile/blob/main/playbook-build-qe.yml, which only contains the quantum espresso code and aiida plugin (plus aiidalab etc)

This produces a 3.5 GB VM (10.4 GB extracted) and 1.52 GB Docker image on Docker hub (4.4 Gb extracted)

[ltalirz]

great, thanks! I consider this issue addressed then; we can still come back to it in the future for further ideas on reducing image size (there are others as well https://github.com/marvel-nccr/quantum-mobile/issues/84 ).

[chrisjsewell]

As a reference baseline, here also is some snapshots of the VM, created with no simulation code or aiidalab roles (just the aiida role), and after running cleanup:

[ltalirz]

you may want to consider using something like filelight that gives a multi-level breakup in one view

libreoffice might be a candidate for removal, but of course there is also a benefit to not worrying about the ubuntu defaults.