Closed Karel-van-de-Plassche closed 4 years ago
Merging #79 into master will not change coverage. The diff coverage is
n/a
.
@@ Coverage Diff @@
## master #79 +/- ##
=======================================
Coverage 76.64% 76.64%
=======================================
Files 24 24
Lines 6055 6055
=======================================
Hits 4641 4641
Misses 1414 1414
Continue to review full report at Codecov.
Legend - Click here to learn more
Δ = absolute <relative> (impact)
,ø = not affected
,? = missing data
Powered by Codecov. Last update f70734c...ebf9cda. Read the comment docs.
- For eigenvalue runs,
time_norm
has no meaning. In the GKDB JSON format, we allow a field being null (or None in Python). This should be interpreted as that the coordinate has size 1. If you think the field should be [null] instead of null for easier integration, we can change that
According to the IMAS specification time_norm
should be an array of floats. Possibly we could use [0.0]
for eigenvalue runs?
phi_potential_perturbed_norm_real
and related fields can be 1D-ish, as the timebase can have length one. Seems like OMAS instead squeezes out this length-1 dimension, resulting in a wrongly shaped array (1D instead of 2D), thus failing the test
The data itself is saved as 1D array, rather than 2D array. On the other hand, I agree that OMAS should check for the coordinates, but it used not to do this when you were loading from data file. I have now fixed this with 94df05b . I have released version 0.49.1
. With the update, now running the connect_gkdb.py
example leads to the following warnings:
- Looks like the load instance methods ODS.load and ODS.load_structure disappeared, so the example currently fails
Uhm, I am not sure what you mean by ODS.load
and ODS.load_structure
disappeared.
According to the IMAS specification time_norm should be an array of floats. Possibly we could use [0.0] for eigenvalue runs?
NaN is also a float right? [null]
which is interpreted as [NaN]
seems to be accepted by OMAS at least.
With the update, now running the connect_gkdb.py example leads to the following warnings
Did a small test, seems we can solve that on our side to indeed save everything as a 2D array (with [[val, .., val]]
, will change that accordingly.
Uhm, I am not sure what you mean by ODS.load and ODS.load_structure disappeared
Must've done something weird, as I also cannot reproduce. You can ignore this comment.
I've merged a small modification of the GKDB.json into master. This updates the .json file you copied accordingly.
Some background how this happend GKDB data format was originally explicitly not IMAS-compatible, as we wanted to be more flexible, as well have a structure that can be easily stored in noSQL/SQL databases. As such, the GKDB JSON specification was born. After it was 'finished', IMAS took the structure and adopted it as their 'gyrokinetic IDS'. Unfortunately this adoption was not 1-to-1, and as such the GKDB JSON was not IMAS compatible anymore (it was never intended as such)
What I've done As the GKDB JSON is so extremely close to OMAS-style IMAS compatibility, I've just changed the GKDB JSON specification. Now it's IMAS compatible, but not OMAS-IMAS compatible yet. I'm adding OMAS compatibility tests in here: https://gitlab.com/gkdb/gkdb/merge_requests/2, combined with a directory cleanup. Feel free to join in.
What needs to be done still (In this pull request or in a separate one): Looks like the data shapes are not correctly handled in the pythonization of the JSON. Specifically two issues that fail the coordinate shape test here omas_core.py#L622, which I don't think should fail:
time_norm
has no meaning. In the GKDB JSON format, we allow a field beingnull
(orNone
in Python). This should be interpreted as that the coordinate has size 1. If you think the field should be[null]
instead ofnull
for easier integration, we can change thatphi_potential_perturbed_norm_real
and related fields can be 1D-ish, as the timebase can have length one. Seems like OMAS instead squeezes out this length-1 dimension, resulting in a wrongly shaped array (1D instead of 2D), thus failing the testODS.load
andODS.load_structure
disappeared, so the example currently failsWhat I hope to do in the very long term Open source the IMAS-DD and associated tools. Discussion ongoing, but slow, as it is subject to the ITER rules and legislation.