Major-update: Ubuntu 20.04, aiida-2.0, Mamba

chrisjsewell commented 2 years ago

This update makes some critical updates to the VM creation, to transition towards the use of conda. Its a "half-way house", whereby aiida is now installed with conda, but not any services/codes.

This is the link to the uploaded image: https://drive.google.com/file/d/1xaZ4AZuyXoJ-sLKkaZpJQwvmWpkiykMq/view?usp=sharing

Moves the tasks in both marvel-nccr.quantum_mobile_customizations and marvel-nccr.simulationbase into this repository, and removes their use.

I found both of these roles to be very opaque in what they were actually doing, and also difficult to change/customise. If you now look in playbook-build.yml, its a little longer, but I feel more understandable about what each step is doing.

I have then also moved the aiida setup task into this repo, and moved to the use of conda/mamba for the conda installation, and both aiida-core 2.0 and aiida-quantumespresso 4 are installed.

Again it is easier to have the code here to iterate on and to see what is going on, also to integrate with the installation of codes, I've also moved some of the ansible "tasks" to ansible "modules" (i.e. python code, which is easier to control) This could potentially be moved back out to a separate ansible collection at some point. Also, the SSSP pseudo-potentials are installed using aiida-pseudo. For now, I have left the installation of postgres and rabbitmq to apt, not conda. note workon aiida still works to activate the environment (I added an alias to conda activate)

Currently, only Quantum Espresso 7.1 is installed; just via compilation at present, rather than Conda.
The VM is updated to Ubunut 20.04. This all seems to work fine. It required one change to the GUI install, and there is a change to how to get the desktop shortcuts to work (which already required manual steps):
- First install the file with https://manpages.ubuntu.com/manpages/focal/en/man1/desktop-file-edit.1.html:
```
$ sudo desktop-file-install --delete-original ~/Desktop/homepage.desktop
```
- Now navigate to the /usr/share/applications in the folder app, find the file and drag/drop it to the Desktop
- Right click on the file on the Desktop and select Allow Launching

The generated VM image is 4.8Gb and imported is 13.1 GB

this is the generated release notes:

[Quantum Mobile]
version = 22.09.21
Operating System = Ubuntu 20.04.4 LTS
Login credentials = max / moritz
[Apt packages]
grace = 1:5.1.25-7build1
gnuplot = 5.2.8+dfsg1-2
xcrysden = 1.6.2-3build1
jmol = 14.6.4+2016.11.05+dfsg1-4build1
default-jre = 2:1.11-72
rabbitmq-server = 3.8.2-0ubuntu1.3
postgresql-client = 12+214ubuntu0.1
[Conda aiida environment]
aiida-core = 2.0.4-pyh1a96a4e_0@conda-forge
aiida-pseudo = 0.7.0-pyhd8ed1ab_0@conda-forge
aiida-quantumespresso = 4.0.0-pyhd8ed1ab_0@conda-forge
cif2cell = 2.0.0a3-pyhd8ed1ab_0@conda-forge
ipykernel = 6.15.3-pyh210e3f2_0@conda-forge
jupyterlab = 3.4.7-pyhd8ed1ab_0@conda-forge
pip = 22.2.2-pyhd8ed1ab_0@conda-forge
python = 3.9.13-h9a8a25e_0_cpython@conda-forge
[Quantum ESPRESSO]
version = 7.1
components = pw.x, cp.x, pp.x, ph.x, neb.x, hp.x, wannier90.x, epw.x, tddfpt.x
[Pseudopotentials]
SSSP/PBE/efficiency/1.1 = /usr/local/share/pseudo_sssp_PBE_efficiency_1.1

ltalirz commented 2 years ago

Cheers!

I'm just reading through your PR description; most changes look good to me, just some comments below.

I've also moved some of the ansible "tasks" to ansible "modules" (i.e. python code, which is easier to control) This could potentially be moved back out to a separate ansible collection at some point.

That's nice. Indeed I imagine these modules could be useful to anyone who wants to manage AiiDA environments through ansible.

Currently, only Quantum Espresso 7.1 is installed

I thought the whole point of the Quantum Mobile was to have all the major open-source codes & aiida plugins preinstalled(?) Or is this a version specifically for the tutorial? Maybe I'm misunderstanding your comment

Just as a last comment: I saw that the VASP and Gaussian logos were copied into the repo. I guess that comes from the common workflows; just wanted to mention that these will probably never ship directly with the Quantum Mobile.

chrisjsewell commented 2 years ago

Or is this a version specifically for the tutorial?

Yes, so first and foremost this build was focussed on the tutorial. Obviously, it takes hours to include all the codes via "direct" compilation, so I didn't want to go through all that whilst experimenting with this 😅 Plus obviously, most of the other aiida plugins, except aiida-quantumespresso, are yet to support aiida-core v2, so these certainly cannot be installed (there is a day planned to rectify this)

Ideally, the final outcome is that all codes are installed in their own conda environments (hopefully "sharing" most dependency versions, like mpi, to keep the VM size down)

The sticking points for this, for me are:

This would require that directly using any code, e.g. pw.x, requires, one to always first run conda activate env_name. I feel I need to first get "buy-in" from people using Quantum Mobile in this way first, e.g. for Quantum Espresso tutorials.

And related to this, is the fact that using conda activate env_name is problematic if you want to run the code via a script. As I think we have discussed before, in my testing and reported in e.g https://github.com/conda/conda/issues/7980, if you simply create a script:

#!/bin/bash
conda activate env_name

Then run it as $ ./script.sh or bash script.sh, even if it works interactively (i.e. you have already run conda init), you still get CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'., which is due to way conda actually works, also see: https://docs.conda.io/projects/conda/en/latest/dev-guide/deep-dive-activation.html

there is a workaround for this

#!/bin/bash
eval "$(conda shell.posix hook)"
conda activate env_name

also you can run e.g. conda run -n env_name pw.x ..., but I feel both of these are not as intuitive for users, and the latter, is not currently possible with AiiDA code setup.

Weirdly though, @unkcpz pointed me towards: https://github.com/aiidalab/aiidalab-qe/blob/86607fbab00475cde3c8c65bca18cebf0298aa3b/aiidalab_qe/setup_codes.py#L85, where they are indeed directly using conda activate env_name in the script. I have no idea how this is working 🤷 so need to look into it

edit: the last important one is the support of ARM64 for code on conda, and that QE 7.1 is currently missing, see https://github.com/conda-forge/qe-feedstock/issues/33 and https://github.com/conda-forge/qe-feedstock/pull/32

but yeh arm64 support is a whole mess of its own

chrisjsewell commented 2 years ago

Just as a last comment: I saw that the VASP and Gaussian logos were copied into the repo. I guess that comes from the common workflows; just wanted to mention that these will probably never ship directly with the Quantum Mobile.

yep cheers, indeed I just copied all of them from there, VASP and Gaussian don't actually show up in the documentation so could be removed

unkcpz commented 2 years ago

Then run it as $ ./script.sh or bash script.sh, even if it works interactively (i.e. you have already run conda init), you still get CommandNotFoundError: Your shell has not been properly configured to use 'conda activate'.,

Maybe it is related to the login_shell of computer setup?

superstar54 commented 2 years ago

I downloaded the VM from the link. In the VM, the SSSP pseudo-potentials version is still 1.0. In the release note, it should be version 1.1.

(aiida) max@qmobile:~/aiida-tutorial$ aiida-pseudo  list
Label                               Type string                Count
----------------------------------  -------------------------  -------
SSSP/1.0/PBE/efficiency             pseudo.family.sssp         85

chrisjsewell commented 2 years ago

SSSP pseudo-potentials version is still 1.0. In the release note

yeh thanks that was just a mistake by me that I fixed already (https://github.com/marvel-nccr/quantum-mobile/pull/200/commits/e87bfec592a1b2d8c01e7e3b42760d32cb6be3c7) but after creating the VM 😉

chrisjsewell commented 2 years ago

Maybe it is related to the login_shell of computer setup?

Ah yeh thats one, if you do bash -l script.sh it works 🎉 I guess it should be ok, to just document that people should always run any scripts involving conda activate like this 🤔

superstar54 commented 2 years ago

Could you provide the latest image? I am runing the AiiDA turoial on the VM. I want to make sure I am using the right image.

One more question. The remote absolute path of pw code is not consistent with the actual path of pw.x.

(aiida) max@qmobile:/$ verdi code show 86
--------------------  ------------------------------------
PK                    86
UUID                  a7653ec9-6cad-4753-a614-9f39d42e3a54
Label                 qe.pw
Description
Default plugin        quantumespresso.pw
Type                  remote
Remote machine        local_slurm
Remote absolute path  /usr/local/pw.x
Prepend text          ulimit -s unlimited
                      export OMP_NUM_THREADS=1
Append text
--------------------  ------------------------------------

Here is the path of pw.x in the VM.

(aiida) max@qmobile:/$ which pw.x
/usr/local/bin/pw.x

chrisjsewell commented 2 years ago

Could you provide the latest image?

Oh I don't have another one, the pseudo version is the only difference, and I didn't have time to rebuild just for that

chrisjsewell commented 2 years ago

One more question. The remote absolute path of pw code is not consistent with the actual path of pw.x.

Yep that requires a fix 👍

unkcpz commented 2 years ago

Ah yeh thats one, if you do bash -l script.sh it works 🎉 I guess it should be ok, to just document that people should always run any scripts involving conda activate like this 🤔

But in the computer configuration, there is an option for use_login_shell, the default is True, maybe you change it in ansible, do you check it is set to True?

unkcpz commented 1 year ago

🥳

marvel-nccr / quantum-mobile

Major-update: Ubuntu 20.04, aiida-2.0, Mamba #200