Reorganise computers/codes setup folders.

[yakutovicha]

fixes #35

This is how I would propose to reorganise the computer's folder:

$ tree
.
├── codes
│   ├── cp2k-6.1-cpu.yaml
│   ├── cp2k-6.1-gpu.yaml
│   ├── cp2k-7.1-cpu.yaml
│   ├── cp2k-7.1-gpu.yaml
│   ├── cp2k-8.1-cpu.yaml
│   ├── cp2k-8.1.yaml
│   ├── pp-6.5.yaml
│   ├── projwfc-6.5.yaml
│   └── pw-6.5.yaml
├── default -> ./hybrid
├── hybrid
│   ├── computer-configure.yaml
│   └── computer-setup.yaml
├── hybrid-s1005
│   ├── computer-configure.yaml
│   └── computer-setup.yaml
├── hybrid-s904
│   ├── computer-configure.yaml
│   └── computer-setup.yaml
└── multicore
    ├── computer-configure.yaml
    └── computer-setup.yaml

The main features:

1. The codes are decoupled from the computer setup. This is to let the user freely chose the computer they want to run on.
2. Different computer configurations are kept in separate folders. Most of them will be removed when we introduce templating mechanism in the computer setup.

[yakutovicha]

The reason why I prefer to keep different computers in different folders is that it is not guaranteed, that the computer-setup will remain the same for all the configurations. Some might change the proxy server, some might change the safe_interval.

[ltalirz]

I'm happy with this proposal. Thoughts @mbercx ?

[yakutovicha]

I would not merge it in the current state, as, to be sure things are fine, I need to adapt the deployment scripts.

[yakutovicha]

@mbercx, please let me know if you agree with the proposed change. If yes, I will adapt the deployment scripts and modify the other computers/codes.

[mbercx]

Alright, I can agree with this. ^^ A couple of final notes:

I see how some codes can be run on all partitions, and so it's better to have them in a separate directory, else we're introducing some duplication again. I do agree with @ltalirz that the codes for specific partitions should match the folder names, but would vote to have hybrid and multicore in the filename instead of changing the folder names. Regarding the labels though, I'm not sure if it makes sense to add the partition to the code label, since it will typically already be in the computer label. So cp2k-6.1-mc@daint-mc seems redundant.

I'm not sure if I fully understand this remark:

The reason why I prefer to keep different computers in different folders is that it is not guaranteed, that the computer-setup will remain the same for all the configurations. Some might change the proxy server, some might change the safe_interval.

How is keeping them in different folders versus just having different setup files in one partition folder an improvement in this regard? Now we're also just duplicating the configuration files.

[mbercx]

Alright, after hijacking the LSMO meeting for a bit, @yakutovicha and I have reached an accord:

• Folders for partitions are fine!
• Change the filenames so the specified partition name matches the partition folder.

I would still vote to remove the partition name from the label of the Code, since it will already be in the label of the Computer, but I won't press this issue. ^^

[ltalirz]

Just mentioning that it would be a possibility to keep the partition label in the name of the yaml file (differentiating codes from different partitions I guess is the reason why Sasha introduced it), but remove it from the code label in the yaml file.

It's true that keeping it in the code label involves some duplication; on the other hand using a different label from the filename can also be confusing. No strong opinion

[mbercx]

Just mentioning that it would be a possibility to keep the partition label in the name of the yaml file (differentiating codes from different partitions I guess is the reason why Sasha introduced it), but remove it from the code label in the yaml file.

Indeed, this is what I was suggesting. ^^

It's true that keeping it in the code label involves some duplication; on the other hand using a different label from the filename can also be confusing. No strong opinion

Oh! how about using filenames like:

cp2k-7.1@hybrid.yaml
cp2k-7.1@multicore.yaml

This is actually what I do with my Code YAML files. 😁

EDIT: and then removing the partition from the code label, of course. ^^

[yakutovicha]

Okay, thanks a lot @mbercx and @ltalirz for your input. I will adapt the PR and request your opinion just to make sure I understood everything correctly.

[yakutovicha]

@mbercx, @ltalirz, I adapted the PR to what we agreed on. Please have a look and let me know if anything is still missing.

The only point that is still a bit unclear, is whether to keep the partition name in the code label. I am slightly in favour of doing that, as it would allow us to have:

cp2k-7.1-mc@gpu.yaml
cp2k-7.1-gpu@gpu.yaml

But I have no strong opinion either. In the end, the user can adapt the code name while setting it up.

Let me know.

[mbercx]

Thanks @yakutovicha!

The only point that is still a bit unclear, is whether to keep the partition name in the code label. I am slightly in favour of doing that, as it would allow us to have:

cp2k-7.1-mc@gpu.yaml
cp2k-7.1-gpu@gpu.yaml

I'm not sure which code cp2k-7.1-mc@gpu.yaml would represent? I would:

1. Use <code>-<version>@<partition>.yaml for all code file names, e.g. cp2k-6.1@hybrid.yaml.
2. Make sure the <partition> in the code file name matches the folder of the corresponding partition.
3. Not put the partition in the code label, since if you do verdi code list you see the computer the code is installed on anyways.

For daint, this would look like:

$ tree
.
├── codes
│   ├── cp2k-6.1@hybrid.yaml
│   ├── cp2k-6.1@multicore.yaml
│   ├── cp2k-7.1@hybrid.yaml
│   ├── cp2k-7.1@multicore.yaml
│   ├── cp2k-8.1@hybrid.yaml
│   ├── cp2k-8.1@multicore.yaml
│   ├── pp-6.5@multicore.yaml
│   ├── projwfc-6.5@multicore.yaml
│   └── pw-6.5@multicore.yaml

Also, I think @ltalirz prefered to have computer-setup.yaml and computer-configure.yaml instead of setup.yaml and configure.yaml. I think removing computer- is fine, but have no strong opinion on the issue.

[yakutovicha]

And what if the code from multicore partition is used in the GPU partition?

cp2k-6.1-multicore@daint-hybrid
cp2k-6.1-hybrid@daint-hybrid

In the example above I wanted to demonstrate this, but wrote something confusing, sorry.

[ltalirz]

thanks for putting the computer back into the name.

I let you and Marnik battle out the remaining details, no further review required from my side.

[ltalirz]

@yakutovicha I see that you now made these changes in the master branch rather than in a new develop branch.

Will moving the codes to a different folder not break stuff in the AiiDAlab?

[yakutovicha]

Will moving the codes to a different folder not break stuff in the AiiDAlab?

No-no, I made sure to generate database.json file (used by AiiDAlab) of the same format.

See the changes in generate_json.py

[ltalirz]

I see - great! Then let's bury the idea of the develop branch and keep it simple.