search

gafusion / omas

Ordered Multidimensional Array Structure
http://gafusion.github.io/omas
MIT License
30 stars 14 forks source link

Fill out more of the EFIT mapping to MDS+ #255

Closed torrinba closed 11 months ago

torrinba commented 1 year ago

This may not be 100% complete, but should be pretty close.

torrinba commented 1 year ago

I am leaving it up to you to test this out @AreWeDreaming

There several areas that need more attention still

  1. constraints.pressure.:.position.psi needs to have an absolute value applied (some additional logic could expand the applicability further)
  2. constraints.pf_current needs to have e-coils, f-coils, and a-coils appended (I just put slashes for now so this is most likely why the regression build fails)
  3. constraints.mse_polarisation_angle needs to have an atan applied
  4. constraints.chi_squared_total is an OMAS extension that is not yet part of IMAS (up to you whether it should be included now or not)
  5. constraints.j_tor needs to be converted to OMAS units (some additional logic could expand the applicability of the position.psi as well)
AreWeDreaming commented 1 year ago
  • [x] constraints.pressure.:.position.psi needs to have an absolute value applied (some additional logic could expand the applicability further)
  • [x] constraints.pf_current needs to have e-coils, f-coils, and a-coils appended (I just put slashes for now so this is most likely why the regression build fails)
  • [x] constraints.mse_polarisation_angle needs to have an atan applied
  • [x] constraints.chi_squared_total is an OMAS extension that is not yet part of IMAS (up to you whether it should be included now or not)
  • [x] constraints.j_tor needs to be converted to OMAS units (some additional logic could expand the applicability of the position.psi as well)

I think I got this covered in my next commit (not yet pushed because it needs to be tested)

torrinba commented 1 year ago

I also forgot that the constraints.j_tor will need to be added as an OMAS extension until the new IMAS version gets released and updated. Hopefully you resolved this already, but if not I can help with that

AreWeDreaming commented 11 months ago

After some modifications of OMAS I got most of the fields to load, but the following do not have data for my test case 17408219 created with CAKE. Notably I didn't test all of these but rather these are the ones I get MDS+ exception for. The first one ZSEPS is kinda weird because RSEP is set correctly.

equilibrium.time_slice.:.boundary.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.DSEP)/100.
equilibrium.time_slice.:.boundary_separatrix.geometric_axis.z: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZOUT)/100.
equilibrium.time_slice.:.boundary_separatrix.strike_point.1.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSID/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSID/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.1.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSID/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSID/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.2.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSOD/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSOD/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.2.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSOD/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSOD/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.3.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSIU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSIU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.3.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSIU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSIU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.4.r: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.RVSOU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.RVSOU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.strike_point.4.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZVSOU/100.','\{EFIT_tree}::TOP.RESULTS.GEQDSK.ZVSOU/100.',-0.89)
equilibrium.time_slice.:.boundary_separatrix.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.constraints.bpol_probe.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGMPI)
equilibrium.time_slice.:.constraints.diamagnetic_flux.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIDFLUX)
equilibrium.time_slice.:.constraints.diamagnetic_flux.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTDIA)
equilibrium.time_slice.:.constraints.flux_loop.:.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.SAISI)
equilibrium.time_slice.:.constraints.flux_loop.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGSIL)
equilibrium.time_slice.:.constraints.ip.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIPASMA)
equilibrium.time_slice.:.constraints.ip.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGPASMA)
equilibrium.time_slice.:.constraints.ip.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTPASMA)
equilibrium.time_slice.:.constraints.j_tor.:.measured: data(\{EFIT_tree}::TOP.MEASUREMENTS.VZEROJ)*data(\{EFIT_tree}::TOP.RESULTS.GEQDSK.CPASMA)/(data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.AREA)/10000.)
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.chi_squared: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.CHIECC','\{EFIT_tree}::TOP.MEASUREMENTS.CHIFCC')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pf_current.:.measured_error_upper: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.SIGECC','\{EFIT_tree}::TOP.MEASUREMENTS.SIGFCC')
equilibrium.time_slice.:.constraints.pf_current.:.reconstructed: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.CECURR','\{EFIT_tree}::TOP.MEASUREMENTS.CCBRSP')
equilibrium.time_slice.:.constraints.pf_current.:.weight: py2tdi(stack_outer,'\{EFIT_tree}::TOP.MEASUREMENTS.FWTEC','\{EFIT_tree}::TOP.MEASUREMENTS.FWTFC')
equilibrium.time_slice.:.constraints.pressure.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.RPRESS','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
AreWeDreaming commented 11 months ago

@bechtt Could you take a look at this list and note any fields that should be set by a standard EFIT run but aren't. If none of these are expected to be set then that's fine, we'll add them on CAKE side of things.

AreWeDreaming commented 11 months ago

@orso82 Any idea why the following TDI expression would cause trouble:

"equilibrium.time_slice.:.convergence.iterations_n": {
    "eval2TDI": "py2tdi(nan_max_where,'size(\\{EFIT_tree}::TOP.MEASUREMENTS.CERROR)','\\{EFIT_tree}::TOP.MEASUREMENTS.CERROR',0)",
    "treename": "{EFIT_tree}"
 }

with

def nan_max_where(a, b, n):
    import numpy as np

    a = a.data()
    b = b.data()
    a[b == n] = np.NaN
    return np.nanmax(a)

MDS+ just returns an error with no info on what's actually wrong.

torrinba commented 11 months ago

After some modifications of OMAS I got most of the fields to load, but the following do not have data for my test case 17408219 created with CAKE. Notably I didn't test all of these but rather these are the ones I get MDS+ exception for. The first one ZSEPS is kinda weird because RSEP is set correctly.

Most of the issues from MEASUREMENTS were missing or conflicting names between what was setup in MDS+ for rtEFIT and the offline version of EFIT. I fixed the EFIT outputs to match, but you will have to rerun the case to see that.

The AEQDSK failures are harder to diagnose. It's possible there are are problems with efitloader. That's the only reason I can think RSEPS would be set but not ZSEPS (which I see in other cases as well)

AreWeDreaming commented 11 months ago

With the latest set of changes the following fields are still not loading for EFIT04 shot 170325. @bechtt are all of these expected to not work?

equilibrium.time_slice.:.boundary.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.boundary_separatrix.geometric_axis.z: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZOUT)/100.
equilibrium.time_slice.:.boundary_separatrix.x_point.:.z: py2tdi(nan_where,'\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS','\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZSEPS',0)
equilibrium.time_slice.:.constraints.bpol_probe.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGMPI)
equilibrium.time_slice.:.constraints.diamagnetic_flux.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIDFLUX)
equilibrium.time_slice.:.constraints.diamagnetic_flux.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTDIA)
equilibrium.time_slice.:.constraints.flux_loop.:.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGSIL)
equilibrium.time_slice.:.constraints.ip.chi_squared: data(\{EFIT_tree}::TOP.MEASUREMENTS.CHIPASMA)
equilibrium.time_slice.:.constraints.ip.measured_error_upper: data(\{EFIT_tree}::TOP.MEASUREMENTS.SIGPASMA)
equilibrium.time_slice.:.constraints.ip.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTPASMA)
equilibrium.time_slice.:.constraints.j_tor.:.measured: data(\{EFIT_tree}::TOP.MEASUREMENTS.VZEROJ)
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.chi_squared: py2tdi(stack_outer_2,'\{EFIT_tree}::TOP.MEASUREMENTS.CHIECC','\{EFIT_tree}::TOP.MEASUREMENTS.CHIFCC')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer_3,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pf_current.:.measured_error_upper: py2tdi(stack_outer_2,'\{EFIT_tree}::TOP.MEASUREMENTS.SIGECC','\{EFIT_tree}::TOP.MEASUREMENTS.SIGFCC')
equilibrium.time_slice.:.constraints.pressure.:.weight: data(\{EFIT_tree}::TOP.MEASUREMENTS.FWTPRE)
torrinba commented 11 months ago

With the latest set of changes the following fields are still not loading for EFIT04 shot 170325. @bechtt are all of these expected to not work?

Yes, I would expect those all to work. But the MEASUREMENT failures (which are a result of different names being output by EFIT and stored in MDS+) were only fixed recently so you should probably add exceptions for older EFIT runs.

The AEQDSK failures seem to point to an issue with efitloader, but I haven't looked into that further yet (which again suggests these errors will be present for many cases in MDS+ even if it is fixed so adding an exception would be necessary)

torrinba commented 11 months ago

@AreWeDreaming all the variables should be working now. You will just need to add exception handling for the MEASUREMENT variables from older EFIT runs where the names didn't match

torrinba commented 11 months ago

Here is a test with the MEASUREMENTS fixed: OMFITmds(server='DIII-D',shot=16330309,treename='EFIT',subtree='',caching=True)

AreWeDreaming commented 11 months ago

Here are the fields that are still broken

equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZEPLIM)/100.
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer_3,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pressure.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.RPRESS','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
AreWeDreaming commented 11 months ago

Notably I did have 

equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZEPLIM)/100.

working by taking the minimum of the 4 gap sizes.

torrinba commented 11 months ago

Here are the fields that are still broken

equilibrium.time_slice.:.boundary_separatrix.closest_wall_point.distance: data(\{EFIT_tree}::TOP.RESULTS.AEQDSK.ZEPLIM)/100.
equilibrium.time_slice.:.constraints.j_tor.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.SIZEROJ','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')
equilibrium.time_slice.:.constraints.pf_current.:.measured: py2tdi(stack_outer_3,'\{EFIT_tree}::TOP.MEASUREMENTS.ECCURT','\{EFIT_tree}::TOP.MEASUREMENTS.FCCURT','\{EFIT_tree}::TOP.MEASUREMENTS.ACCURT')
equilibrium.time_slice.:.constraints.pressure.:.position.psi: py2tdi(efit_psi_to_psi,'\{EFIT_tree}::TOP.MEASUREMENTS.RPRESS','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIMAG','\{EFIT_tree}::TOP.RESULTS.GEQDSK.SSIBRY')

I fixed the first and third. I can confirm the quantities in the second and fourth are in MDS+ so I expect the problem is with the TDI (possibly the dimensions)

AreWeDreaming commented 11 months ago

I'll retest.

I fixed the first and third. I can confirm the quantities in the second and fourth are in MDS+ so I expect the problem is with the TDI (possibly the dimensions) My favorite kinda error.

AreWeDreaming commented 11 months ago

There is no more broken fields now for @bechtt's test case.

AreWeDreaming commented 11 months ago

@orso82 If you could take a look at my changes to OMAS machine it would be much appreciated. The main change is that you can now load all available mappings by using a wildcard in the location.

torrinba commented 11 months ago

@AreWeDreaming do you understand why regression is failing? It passed prior to 02fef133e1b494b4f488e13cc2881ce9c24e2c83

AreWeDreaming commented 11 months ago

I'll have to run the regression tests locally. The error before was related to OMFIT things, now it doesn't seem to give us any information at all. The lines that were changed in the hash you pointed out don't exist anymore...

orso82 commented 11 months ago

Looks good to me

I am happy to see that someone else other than me was able to generate the updated IMAS JSON data dictionary files! 👍

smithsp commented 11 months ago

I may have merged https://github.com/gafusion/OMFIT-source/pull/6708 prematurely, since this one is not passing regressions. It was passing regression on the OMFIT side...

smithsp commented 11 months ago

If I run make test on my own computer, I get

======================================================================
ERROR: test_omas_dynamic_machine (test_omas_examples.TestOmasExamples)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/Users/smithsp/omas/omas/omas_core.py", line 1303, in __getitem__
    value = self.dynamic.__getitem__(location)
  File "/Users/smithsp/omas/omas/omas_machine.py", line 984, in __getitem__
    return self.cache[o2u(key)][key]
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  [Previous line repeated 3 more times]
  File "/Users/smithsp/omas/omas/omas_core.py", line 1314, in __getitem__
    self.__setitem__(key[0], self.same_init_ods())
  File "/Users/smithsp/omas/omas/omas_core.py", line 872, in __setitem__
    raise ValueError('`%s` has no data' % location)
ValueError: `pf_active.coil.2.element.6.geometry.geometry_type` has no data

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/smithsp/omas/omas/tests/test_omas_examples.py", line 143, in test_omas_dynamic_machine
    from omas.examples import omas_dynamic_machine
  File "/Users/smithsp/omas/omas/examples/omas_dynamic_machine.py", line 30, in <module>
    ods.plot_overlay(wall=True, magnetics=True, pf_active=True)
  File "/Users/smithsp/omas/omas/omas_plot.py", line 2443, in overlay
    overlay_function(ods, ax, **overlay_kw)
  File "/Users/smithsp/omas/omas/omas_plot.py", line 2748, in pf_active_overlay
    ods['pf_active.coil'][c]['element'][e]['geometry.geometry_type'], 'pf_active', ods.imas_version
  File "/Users/smithsp/omas/omas/omas_core.py", line 1324, in __getitem__
    return value.__getitem__(key[1:], cocos_and_coords)
  File "/Users/smithsp/omas/omas/omas_core.py", line 1305, in __getitem__
    raise OmasDynamicException(f'Error dynamic fetching of `{location}` for {self.dynamic.kw}: {repr(_excp)}')
omas.omas_core.OmasDynamicException: Error dynamic fetching of `pf_active.coil.2.element.6.geometry.geometry_type` for {'machine': 'd3d', 'pulse': 168830, 'options': {'EFIT_tree': 'EFIT02'}, 'branch': '', 'user_machine_mappings': None}: ValueError('`pf_active.coil.2.element.6.geometry.geometry_type` has no data')

----------------------------------------------------------------------
Ran 42 tests in 17.585s

Note that this does not seem to be the same problem as reported in the automatic tests.

torrinba commented 11 months ago

Thanks for helping test this @smithsp. We didn't make changes to any of the DIII-D machine mappings and I don't see any changes to the IMAS schema for pf_active so I don't understand why you are seeing that error pf_active

From what I can tell _common.py only sets the geometry properties for element 0, which could be related, but when I run make test on Iris I can't reproduce the issue. Instead it seemed like the only problem was that COCOs needed to be updated (last commit). Hopefully that resolves everything and this merge can go forward!

AreWeDreaming commented 11 months ago

This shouldn't be merged yet. I just implemented the changes we do have into omas-viewer and this is what the equilibrium plot looks like: image

I'll try and figure out what went wrong. I think psi is just a transposition error, but I do not understand what's going on with

equilibrium.time_slice.:.constraints.pressure.:.position.psi
AreWeDreaming commented 11 months ago

Much better: image

smithsp commented 11 months ago

Need to check omas_from_imas.

AreWeDreaming commented 11 months ago

Midway of testing omas_from_imas I realized how incredibly unrelated that is to this PR. I did it anyways and found no problems. Looking for someone that could test _efit.json for non-DIII-D machine. Looped in @jmcclena to test something NSTX related.

torrinba commented 11 months ago

Midway of testing omas_from_imas I realized how incredibly unrelated that is to this PR. I did it anyways and found no problems.

It is related to the orientation of the flux matrix loaded into OMAS from the MDS+ machine mapping. There is an issue with the g-file to OMAS conversion being resolved in this PR https://github.com/gafusion/OMFIT-source/pull/6684

I'm surprised, but happy to hear that the orientation in MDS+ must be different

torrinba commented 11 months ago

@orso82 or @smithsp could you merge this if you approve? Neither Severin or I have permission to, but both agree it is ready