search

nomad-coe / nomad

NOMAD lets you manage and share your materials science data in a way that makes it truly useful to you, your group, and the community.
https://nomad-lab.eu
Apache License 2.0
69 stars 16 forks source link

Problems with nexus parser #106

Open KGoetz91 opened 1 month ago

KGoetz91 commented 1 month ago

We have installed a nomad oasis in our lab environment and are trying to upload nexus format data.

The upload works and the parser recognises the data files, we are able to view the data in the oasis. However none of the metadata included in the nexus file is incorporated in a searchable way. I included some screenshots of a simple nxs file we created from the nexus homepage to include an instrument and chemical formula.

However the two metadata entries are not incorporated. Are we doing something wrong on our side or is there something missing in the parser/normalisers? nexus1 nexus2 nexus3 nexus4 nexus5

lauri-codes commented 1 month ago

Hi @KGoetz91!

In the default search interface, only the properties under results will be searchable. Since the nexus parser is probably not filling up anything in results, you will also not be able to search anything.

There are two approaches to solving this:

  1. You could define a completely new app that loads the interesting bits from your nexus data into the search interface. The default app is not configured to automatically read anything from nexus application definitions (or from any custom Python schemas), because it would get awfully crowded otherwise.
  2. You could introduce an additional normalizer or we could modify the nexus parser to fill the results section with the data. This way the entry would become more cross-compatible with other types of data, but there are constraints set by the rather rigid structure of the results section.

I think option 1 will be an easier. Maybe you could provide me with an example of the nexus files that you are interested in, and I could draft you an initial app to get started? I could then also add a page about this procedure in our online docs.

@domna: Do you have any comments on this? I guess it would quite a hard task to map each application definition to the results section consistently in the nexus parser.

domna commented 1 month ago

Hi @KGoetz91,

I agree with @lauri-codes's view that creating an app for nexus would be the way to make the data searchable. However, we will provide a general NeXus search app eventually and currently also support extracting the chemical formula into the results section (and hence make your data searchable in the periodic table viewer). We may also further extend which informations we compile into the results section. If there is something you urgently need we can try to find the correct place to put it into the results section - but as Lauri mentioned the results are a rigid structure and we may not find an appropriate place.

Additionally, I see that your data is not even parsed into the nexus metainfo (which is not searchable but browsable). This is a prerequisite to make any data searchable (even for an app). I think this is due to the fact that your file is not following any application definition (it would need to have at least an /entry/definition pointing to a valid application definition, e.g., NXoptical_spectroscopy - only then nomad is able to attach it to the correct metainfo). You could try to add the term NXoptical_spectroscopy into /entry/definition (as from the data it looks like that its an optical spectroscopy experiment). The metainfo will only parse the documented fields according to this application definition.