map experiments data to OMAS

[orso82]

This PR brings into OMAS the ability to dynamically load machines experimental data directly from their native databases (eg. MDS+ for DIII-D). This is accomplished by defining mappings from these data storages to OMAS.

The mapping information is stored in a json file (see for example d3d.json) and can be defined with the MDS+ TDI language or in Python (but can easily extended to other languages).

OMAS can use these local mappings files or download their latest version from any branch on GitHub. OMAS will fallback on downloading the latest mappings on the master when the user requests a mapping that is not known. This should allow us to keep adding new mappings without being too concerned about keep having to release new OMAS versions.

Why this is nice:

1. simple OMAS syntax to load experimental data (see this example)
2. remove complexity and dependencies of all intermediary layers between OMAS and the original data (see figures below)
3. remove dependency on OMFIT to fill out experimental data in OMAS (the OMFIT functions that used to do that can be easily moved to OMAS and inherit all the benefits, see for example d3d.py)
4. mappings defined with compact json syntax instead of XML
5. mappings defined in the clients not the server. This means:
   • no need to run/update other servers/services in addition to original data server/service
   • let users easily define their own mappings!
6. all in one place, all in Python
   • easy to extend the mapping engine
   • engine and mappings are version controlled together
   • engine can take full advantage of OMAS functionalities (like automatic COCOS translations, info about data coordinates, data types, documentation, etc.).
7. support for logic within the mapping functions (eg. what EFIT tree to use)
8. play well with OMFIT
   • use OMFIT ssh tunneling capabilities to access databases behind firewalls
   • OMFIT interface for defining mappings, testing them, updating mapping file or suggesting them to OMAS developers

NOTE: The mapping info could at a later stage be used also by other technologies (such as UDA) to perform the same kind of mappings (from machines native storages to UDA, for example). The reverse is also true, one could envision parsing the data mapping info used in UDA (defined in XML files) directly in OMAS.

This is OMAS accessing experimental data via IMAS

This is OMAS accessing experimental data via UDA (eg. as demonstrated for MAST)

This is OMAS accessing experimental data directly via its own machine mappings (eg. as done for DIIII-D)

This is the OMFIT GUI that is being designed to ease and guide the definition of new OMAS mappings:

[codecov[bot]]

Codecov Report

Merging #140 (a527a8d) into master (db30c1e) will decrease coverage by 0.78%. The diff coverage is 60.34%.

@@            Coverage Diff             @@
##           master     #140      +/-   ##
==========================================
- Coverage   79.53%   78.74%   -0.79%     
==========================================
  Files          50       53       +3     
  Lines        9811    10201     +390     
==========================================
+ Hits         7803     8033     +230     
- Misses       2008     2168     +160     

Impacted Files	Coverage Δ
omas/examples/omas_dynamic_nc.py	59.37% <ø> (ø)
omas/omas_imas.py	22.53% <0.00%> (ø)
omas/omas_nc.py	95.18% <ø> (ø)
omas/machine_mappings/python_tdi.py	37.14% <37.14%> (ø)
omas/tests/test_omas_examples.py	81.92% <50.00%> (-0.79%)	:arrow_down:
omas/omas_machine.py	55.97% <55.97%> (ø)
omas/omas_core.py	84.33% <61.22%> (-0.88%)	:arrow_down:
omas/tests/test_omas_machine.py	95.00% <95.00%> (ø)
omas/omas_s3.py	52.56% <100.00%> (ø)
omas/omas_setup.py	77.89% <100.00%> (+0.72%)	:arrow_up:
... and 6 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update db30c1e...a527a8d. Read the comment docs.