Instrument database

[CFGrote]

hi @aljosahafner , if you find the time, please provide a rough sketch of your ideas and plans here.

[mads-bertelsen]

Reading the description for internal milestone 5.4 I get the impression we will store serialized version of the simulations in some database instead of simple python scripts. Can see the advantage in this for simple use cases, but it will be difficult to use this as examples for how to use the simulation codes. Perhaps a database with both options could be envisioned?

[aljosahafner]

Good point! For Oasys there is more or less no other way than making the whole Oasys workspaces (binary files) available. Otherwise it is impossible to have a certain beamline in the GUI.

[aljosahafner]

After discussion with @srio , we think that a documented repository of (documented) Oasys workspaces should suffice for this. There is already built-in capability in Oasys to open remote workspaces.

https://github.com/PaNOSC-ViNYL/Oasys-PaNOSC-Workspaces

[CFGrote]

After discussion with @srio , we think that a documented repository of (documented) Oasys workspaces should suffice for this. There is already built-in capability in Oasys to open remote workspaces.

https://github.com/PaNOSC-ViNYL/Oasys-PaNOSC-Workspaces

That's great news. Such a repo would be more than sufficient, also for other RIs and simulation frameworks.

[shervin86]

Summary of the outcome of the first meeting:

• use git repositories for storing instrument description
• separate repositories for Oasys and the rest (SimEx and McStasScripts), the former in binary format, the latter in plain text
• plain text format to be decided
• there is the possibility to separate the instrument description in multiple files:
  • source
  • beam line
  • sample
  • detector
• possible directory structure:
  1. simulation code: McStas or SimEx
  2. institute name
  3. instrument name
  4. HEAD (for the most recent configuration) or a date range for the validity of the configuration (YYYY_MM_DD-YYYY_MM_DD). This is to take into account upgrades of the instruments and if someone wants to run a simulation for old data.
• it is important to be able to embed in the openPMD output file all the information useful to reproduce the simulated data, taking into account the instrument description provided by the repository and all possible changes performed by the user (instrument parameters).

[JunCEEE]

I feel the definitions in NeXus could be a good reference: https://manual.nexusformat.org/classes/base_classes/NXentry.html#nxentry

INSTRUMENT:
    name: simulation calculator (e.g. pysingfel) or the name of the instrument simulated by SIMEX (e.g. virtual SPB/SFX)
        @short_name: simulation calculator acronym
    DETECTOR: 
        sensor_material: Ideal
        sensor_thickness: 0
    DETECTOR_MODULE: Convert from crystfel .geom?
BEAM: 
    incident_wavelength: simulation wavelength
    total_flux: 
SOURCE:
        name: SIMEX

[shervin86]

In order to clone without having an account on github, this should work:

git clone git://github.com/PaNOSC-ViNYL/instrument_database.git

[shervin86]

Here's the repository that have been created and I started to populate it on a branch "shervin"

[shervin86]

@mads-bertelsen @JunCEEE can you please draft a possible todo list here from your perspective?

Here's the notes I showed during our last meeting. The conversion to Markdown is not perfect, I will fix it.

Repository structure

<simulation code >/<institute>/<instrument|source|beamline>

versioning is done as follows:

• HEAD is the most recent configuration, and also currently existing
• dismissed instruments should go under a different subdirectory with the end-of-life date in the format YYYY_MM_DD
• for each institute one or more sources are available
• for each institute one or more beamlines are available
• one instrument should include the corresponding source and beamline configurations
• whenever one of the three components is dismissed, upgraded, etc., one end-of-life entry should be created for the three of them, making sure none of them is <HEAD>

Example

• mcstas/
  • components/
  • ILL/
    • sources/
      • HEAD/source.py: need just one source
      • 2021/12/31/source.py
    • beamlines/
      • H51/HEAD/H51.py: some of the beamlines are common to multiple instruments
      • H51/2020~1231~/H51.py
    • D22/
      • HEAD/D22.py:
        • components/
        • share/
        • include(mcstas/ILL/sources/HEAD/source.py)
        • include(mcstas/ILL/beamlines/HEAD/H51.py)
      • 2019_12_31/D22.py
        • components/
        • share/
        • include(mcstas/ILL/sources/2019~1231~/source.py)
        • include(mcstas/ILL/beamlines/2019~1231~/H51.py)
  • examples/
• simex/

pseudo code

.init(url=the default repository, local_repo=some default local directory)

• url: url of the github central repository
• local_repo: local checkout directory. The user need the possibility to set it, but for the average user a default is ok

git clone $url $local_path if it does not exist git pull to update the local repo

.load_instrument (instrument_sample, version=HEAD)

• instrument: can be a short name or the full path of the file the instrument file as include of
  • source
  • beamline
  • effective instrument with detector
• version:
  • HEAD: for the most recent
  • YYY_MM_DD: for a given old version
  • if HEAD does not exist, it automagically pick up the most recent version? or return error to make sure the user knows?

.load_sample()

.set_sample(instrument_obj, sample_obj)

• instrument_obj

.ls_instrument()

prints the names of the available instruments

.ls_samples()

prints the names of the available samples

.ls_versions(instrument)

prints the available versions for the required instrument

.dump_instrument(outputfile)

saves in .py format the instrument description

.print_parameters(outfile)

prints in json format the list of parameters for the instrument

class pyVinylSample

Is there a general way to deal with the sample+sample environment that can be used by both McStasScripts and SimEx? Having a common base class as for the parameters would make it simpler to define an API for a sample database/repository.

class pyVinylParameters

A stupid basic example:

{
   "counts": [value, min, max, step], 
  "wavelength": , 
 "collimation":,
 "detector position" , 
}

• each parameter can have:
  • an assigned default value
  • a list of ranges [min:max] of acceptable values I just discovered that for D22 at ILL there are several forbidden ranges
  • a step suggested for scans, to help the user to decide a reasonable step size
  • some parameters might accept only discrete values, such a predefined list, it would be good to support it
  • the parameter class should be exportable in json format so a read and write methods should be implemented
  • make openPMD raytracing extension API know about pyVinlyParamters in order to correctly save such information in the openPMD file

The question is how to track changes in the non-parameter variables in the component definitions?

myparams = pyVinylParameters()

myparams = pyVinylParameters(instrument_file) # here instrument_file is in the original mcstas format

myinstrument, myparams = load_instrument()

#mysource = myinstrument.add_component("source", "Source_simple") mysource.xwidth = 0.12 mysource.yheight = 0.12 mysource.lambda0 = myparams.wavelength.value # copy by reference, not value mysource.dlambda = 2.2 mysource.focus_xw = 0.05 mysource.focus_yh = 0.05

myparams.dump("running_values.json")

myOpenPMDoutput <-> myinstrument #myOpenPMDoutput.set_parameters(myparams)

other option: my_source.xwidth = param(isParam=false, value=0.12) mysource.lambda0 = param(isParam=true, value=myparams.wavelength)

myinstrument.dump_nonparameters(): this loops over the components and prints in json format the values of the variables that are 'isParam=false'

if the instrument is not in the repository or modified w.r.t. to that, just dump the pickle in the openPMD?

it is not safe to include a pickle... json is more appropriate

Simulation request.json

{ "--ncount": "5000000000", "instrument": { "name": "D22" }, "mcpl": {}, "return": "COUNTS", "sDETECTOR": {}, "sFULL": { "lambda": 4.51 }, "sSAMPLE": { "D22~collimation~": 2.0, "material": "H2O", "sample~sizer~": 0.0051, "sample~sizey~": 0.0052 }, "sSOURCE": {}, "type": 2 }

simulation stages:

Information saved in openPMD

• instrument information: * existing instrument in repository

  • instrument repository url
  • instrument file path in repository
  • git commit hash
  • instrument parameters dump use json format or code it into openPMD somehow?
  • non existing instrument
    • blob of instrument file

• sample information:

[JunCEEE]

Todo:

• [x] Discuss if we like the current API desgin.
• [x] Discuss if develop the repository API as a part of libpyvinyl?
• [x] Discuss the dump json method with Carsten.
• [x] Create an instrumentation based on SIMEX.

[shervin86]

• the default branch for the repo is main instead of master, it is not nice to have a non standard definition of the default branch
• any convension for the name of the instruments? should instrument objects always have the same name? should the name of the instrument object be the same as the file in which it is defined?
• add a tools/ or scripts/ directory with a python script that converts the py into an instr as follows

  ./scripts/py2instr mcstas/ILL/instruments/D22/HEAD/D22_quick.py
  # and generates a D22_quick.instr

• CMakeLists.txt to compile all the instruments defined for mcstas and provide .deb packages for the instruments
• ctest for testing instrument simulations and differences
  • compile and run for HEAD commit
  • compile and run for master commit
  • make simulation vs simulation plots and chi2
  • report in a reasonable way for automatic tests
  • put in place a CI workflow with tests for commits and merge requests
• define labels for issues:
  • compilation: error, warning or other issue concerning the compilation
  • installation: error, warning or any other issue concerning the installation
  • feature: feature request or improvement request
  • doc: change, improvement or clarification on the documentation
  • simulation_result: issue, disagreement, etc. about the outcome of the simulation, usually supported by some comparison with real data

[mads-bertelsen]

I was tasked with creating a prototype system for parameters that would go well with the instrument database. It consists of the following:

• Parameter class that can handle legal and illegal intervals along with discrete options
• Parameters class that can hold Parameters, this would be associated with a calculator
• ParametersCollection class that can hold Parameters objects for a number of calculators
• ParametersCollection can have master parameters that links to individual parameters of several calculators

The code includes a few example files that show how the code is supposed to be used.

ParameterPrototype_package.zip

[shervin86]

This can be pushed in a new branch in the PyVynil repository and @JunCEEE and me can see how to merge it into it next week. After that, I can modify the prototype of the D22 instrument description providing the parameters

[JunCEEE]

I have checked @mads-bertelsen 's parameters class. I see its potential to be the baseParamters class in libpyvinyl. My question is:

• How can we write/read the instrumentParameters in .json?

If there can be answered, we are in good shape.

[JunCEEE]

A way to implement Mads' design in this example: https://github.com/PaNOSC-ViNYL/libpyvinyl/blob/master/doc/source/include/notebooks/example-01.ipynb

def setCalculatorParameters(photon_energy=6e3, pulse_energy=1.0e-6, grid_size_x=128, grid_size_y=128):
    param1 = Parameter("photon_energy", unit="eV", comment="Photon energy")
    param2 = Parameter("photon_energy", unit="joule", comment="Energy per pulse")
    param3 = Parameter("grid_size_x", unit="", comment="Grid size in x direction")
    param4 = Parameter("grid_size_y", unit="", comment="Grid size in y direction")
    parameters = Parameters([param1, param2, param3, param4])

    return parameters

CalculatorParameters = setCalculatorParameters(photon_energy=6e3, pulse_energy=1.0e-6, grid_size_x=128, grid_size_y=128)
calculator  = RandomImageCalculator(CalculatorParameters, output_path="out.h5")

[shervin86]

I think we can start defining its json format and then we can write a toJson fromJson methods. I will try to draft their forms later

[JunCEEE]

We agree that we should have the parameters class in libpyvinyl.

[JunCEEE]

Hi @shervin86 and @mads-bertelsen, the instrument.json file generated by the parameters class which is going to be implemented in libpyvinyl here looks like this: https://gist.github.com/JunCEEE/94189879c3593d6511c30c5f57cd151e The parameters class is based on Mads's development. How do you like it?

[mads-bertelsen]

I am now back from vacation, happy to see the parameter system has proven useful and is being implemented into pyvinyl. I wrote it with the restriction that it should be easy to dump/read from json, but didn't actually write these methods. Didn't know about the dict method, seems like a really neat way to accomplish it!

[shervin86]

Instrument database prototype available here: https://github.com/PaNOSC-ViNYL/instrument_database