Install pyiron on Vera (C3SE HPC@Chalmers)

[aageo25]

I want to install pyiron on the HPC at Chalmers. Some remarks:

• The HPC has slurm as the queue system
• Everything is available by modules (python, lammps, vasp, etc)
• According to their documentation, conda environments should be installed using containers (Apptainers)
• $HOME is limited. All calculations should be submitted in/from the group folder.
• My intention is to run pyiron and submit the calculations using executables.

I prepared my container based on the pyiron docker image. I succesfully launched JupyterLab and created a first project database. Can that be used in production?

What I want to do now is to launch an example calculation (lammps) to the slurm system. I want to know if my basic configuration is correct. Probably it is not.

[aageo25]

I noticed that pyrion does not recognize my configuration file (.pyiron)

Running "s._configuration" outputs:

{'user': 'pyiron',
 'resource_paths': ['/opt/conda/share/pyiron'],
 'project_paths': [],
 'connection_timeout': 60,
 'sql_connection_string': None,
 'sql_table_name': 'jobs_pyiron',
 'sql_view_connection_string': None,
 'sql_view_table_name': None,
 'sql_view_user': None,
 'sql_view_user_key': None,
 'sql_file': '/cephyr/users/ageo/Vera/pyiron.db',
 'sql_host': None,
 'sql_type': 'SQLite',
 'sql_user_key': None,
 'sql_database': None,
 'project_check_enabled': False,
 'disable_database': False,
 'credentials_file': None,
 'write_work_dir_warnings': True,
 'config_file_permissions_warning': True}

[aageo25]

I managed to run a LAMMPS calculation and generate a trajectory, but not through squeue. I guess it used the lammps from the container.

[jan-janssen]

Summarise the answers we discussed during the meeting:

• Containers can in principle be used with pyiron but this requires pyiron to have access to the queuing system from within the container. It is interesting to learn more about such a containerised HPC setup, but for now the installation without containers is recommended for production use and the use of containers is experimental.
• The config file not being recognised might also depend on the location of the config file. Is it included in the container?
• Is there any documentation on how to connect to the queuing system from a container for your specific HPC?

[aageo25]

Friday afternoon follow-up:

• According to C3SE support, accessing slurm inside a container might very likely crash. So I decided to go on and install pyiron in a virtual environment.
• The virtual environment I chose was virtualev, the recommended for the cluster. Thus, I install pyiron via pip (not recommended according to pyiron docs).

Pyiron is installed. Now I can move on.

• The first issue I encountered was on how to make pyiron recognize the .pyiron configuration file that it is not allocated in $HOME.

  • Issue solved by exporting the PYIRONCONFIG variable

• The current issue is now when I try creating the structure. It raised an error ValueError(Was not able to locate a periodic table).

  • I looked into the source code of pyiron_atomistics/atomistics/structure/periodic_table.py, but I did not find how this error is raised.
  • In the JupyterLab log, I notice that kite was not found. Does this have something to do with this error?

[jan-janssen]

• I looked into the source code of pyiron_atomistics/atomistics/structure/periodic_table.py, but I did not find how this error is raised.

This is similar to the issue discussed in https://github.com/pyiron/FAQs/issues/6 can you try to download the potential files as explained in https://pyiron.readthedocs.io/en/latest/source/installation.html#download-pyiron-parameter-files ?

> import pyiron
> pyiron.install()
>>> It appears that pyiron is not yet configured, do you want to create a default start configuration (recommended: yes). [yes/no]:
> yes
> exit()

[aageo25]

• I looked into the source code of pyiron_atomistics/atomistics/structure/periodic_table.py, but I did not find how this error is raised.

When I run pyiron.install(), I get "pyiron is already installed"

[jan-janssen]

At the moment this function only works when pyiron is not yet configured: https://github.com/pyiron/pyiron_base/blob/main/pyiron_base/state/install.py#L97

Still it anyway does nothing else than downloading https://github.com/pyiron/pyiron-resources/releases and creating an initial pyiron resources file.

[aageo25]

• I looked into the source code of pyiron_atomistics/atomistics/structure/periodic_table.py, but I did not find how this error is raised.

This is similar to the issue discussed in #6 can you try to download the potential files as explained in https://pyiron.readthedocs.io/en/latest/source/installation.html#download-pyiron-parameter-files ?

> import pyiron
> pyiron.install()
>>> It appears that pyiron is not yet configured, do you want to create a default start configuration (recommended: yes). [yes/no]:
> yes
> exit()

After cleaning up my environment, I was able to succesfully run pyiron.install().

I created the queues folders and files. For now I only have the work queue.

I continued to run my notebook. I prepared my queue, defining the number of cores, and modifying the run_lammps_2020.03.03_mpi.sh to submit a job to the cluster. When I fire job.run(run_mode='work', delete_existing_job=True), I get The job Al_T800K was saved and received the ID: 2 followed by the error message starting with 2023-11-30 18:27:09,968 - pyiron_log - WARNING - Job aborted and in the end ValueError: run_queue.sh crashed.

[aageo25]

• I looked into the source code of pyiron_atomistics/atomistics/structure/periodic_table.py, but I did not find how this error is raised.

This is similar to the issue discussed in #6 can you try to download the potential files as explained in https://pyiron.readthedocs.io/en/latest/source/installation.html#download-pyiron-parameter-files ?

> import pyiron
> pyiron.install()
>>> It appears that pyiron is not yet configured, do you want to create a default start configuration (recommended: yes). [yes/no]:
> yes
> exit()

After cleaning up my environment, I was able to succesfully run pyiron.install().

I created the queues folders and files. For now I only have the work queue.

I continued to run my notebook. I prepared my queue, defining the number of cores, and modifying the run_lammps_2020.03.03_mpi.sh to submit a job to the cluster. When I fire job.run(run_mode='work', delete_existing_job=True), I get The job Al_T800K was saved and received the ID: 2 followed by the error message starting with 2023-11-30 18:27:09,968 - pyiron_log - WARNING - Job aborted and in the end ValueError: run_queue.sh crashed.

The reason was a misconfigured sbatch submission template file. After adjusting the keywords, I was able to submit a lammps job to the queue.