Move NOMAD examples to plugin

[lukaspie]

• [x] build entrypoints
• [x] include in MANIFEST.in
• [x] include in pyproject.toml
• [x] check if all plugins are really up-to-date -> there may be updates in other NOMAD branches that are not included here.
• [x] decide whether some of the examples or at least some of the data shall be downloaded instead of stored in Git
• [x] can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be tricky EDIT: we can just have the data and the nomad plugin entrypoint within the plugins themselves
• [x] test for example data and ExampleUploadEntryPoint
• [x] test within NOMAD

[rettigl]

• can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

[lukaspie]

• can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

[rettigl]

• can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

[lukaspie]

• can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

I think what we could do is have the pynxtools plugins be the SSOT place that have these folders with the example files that shall be used in NOMAD. Then instead of having the files also in NORTH, we can just get them from GitHub in the NORTH CI/CD for testing. And for the NORTH tools that don't have a directly correspondant plugin, we just don't use this.

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

[lukaspie]

• can we have a mechanism so that the pynxtools plugins have their own nomad examples and these are brought into the central pynxtools only if the plugin is installed? This will likely be trickier than

Such a solution would from my point of view be the most logical place to locate these examples, i.e. together with each reader. In any case, we should follow a SSOT approach for these examples.

Absolutely, because as of now, we not only need to update them in NORTH and in nomad-FAIR, but also in all distribution branches in nomad-distro. Can we not try to download them from NORTH? We need them there anyway because the CI/CD checks that the examples work with the installed image.

It feels a bit strange having them in NORTH at all, because there is not a 1:1 mapping of NORTH tools and pynxtools plugins, i.e. there is not a tool for every example/plugin, no? Maybe one could create another example repo with sub-repos for each example, which then gets submoduled where needed? However, then one still needs to update the submodule commits all the time, so also not ideal.

I think what we could do is have the pynxtools plugins be the SSOT place that have these folders with the example files that shall be used in NOMAD. Then instead of having the files also in NORTH, we can just get them from GitHub in the NORTH CI/CD for testing. And for the NORTH tools that don't have a directly correspondant plugin, we just don't use this.

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

I still think we need to define the ExampleUploadEntryPoint instances directly in pynxtools and cannot delegate that to the plugins because you need to declare the nomad plugin entry-points in the pyproject.toml of the nomad plugin itself, like so:

[project.entry-points.'nomad.plugin']
apm_example = "pynxtools.nomad.entrypoints:apm_example"
ellips_example = "pynxtools.nomad.entrypoints:apm_example"
em_example = "pynxtools.nomad.entrypoints:em_example"

[lukaspie]

Then for the NOMAD examples, we use the url option in ExampleUploadEntryPoint (see docs) and point this to the URL of the examples in the repo.

I think we can just use https://download-directory.github.io to directly get the examples from a different repo as a zip file. Or we use git subtree.

[rettigl]

This sounds like a reasonable approach. With the requirements for the entry points, I am not sure I understand this well enough

[lukaspie]

What I hadn't figured out: we can just have the entry points for the examples in the individual plugins, see https://github.com/FAIRmat-NFDI/pynxtools-mpes/pull/32/. That makes it so much cleaner because the data can just live in the plugin repo and we don't need to use the url for the entry point.