DIII-D pandas MergeError for shots since 177985

[yumouwei]

I was checking some more recent shots and noticed pandas raises a MergeError for every physics method. e.g.

ERROR tests/test_against_cache.py::test_data_columns[p_ech] - pandas.errors.MergeError: Incompatible merge dtype, dtype('O') and dtype('float64'), both sides must have numeric dtype

The first "long" shot for this to happen is 177985 following a long maintenance & ramp up period. This error occurs up to the most recent shot 200693.

[gtrevisan]

I tried running:

poetry run python tests/test_against_cache.py --shot 177985 --data-column p_ech

and got first:

15:31:35,764 disruption_py INFO | [Shot 177985]: Populating get_power_parameters
15:31:35,919 disruption_py INFO | [Shot 177985]: No NBI power data found in this shot.
15:31:36,75 disruption_py INFO | [Shot 177985]: Failed to open ECH node. Setting to zeros
15:31:36,77 disruption_py DEBUG | [Shot 177985]: Traceback (most recent call last):
  File "disruption-py/disruption_py/machine/d3d/physics.py", line 135, in get_power_parameters
    p_ech, t_ech = params.mds_conn.get_data_with_dims(
  File "disruption-py/disruption_py/inout/mds.py", line 190, in get_data_with_dims
    data = self.conn.get("_sig=" + path).data()
  File "/usr/local/mdsplus/python/MDSplus/connection.py", line 319, in get
    return self.conn.get(exp, *args, **kwargs)
  File "/usr/local/mdsplus/python/MDSplus/connection.py", line 213, in get
    return self._get_answer(timeout)
  File "/usr/local/mdsplus/python/MDSplus/connection.py", line 142, in _get_answer
    _exc.checkStatus(_GetAnswerInfoTO(self.conid, dtype, length, ndims,
  File "/usr/local/mdsplus/python/MDSplus/mdsExceptions.py", line 94, in checkStatus
    raise exception
MDSplus.mdsExceptions.TreeNODATA: %TREE-E-NODATA, No data available for this node

and then:

15:31:43,665 disruption_py INFO | [Shot 177985]: Completed get_power_parameters, time_elapsed: 7.9010865688323975
15:31:43,666 disruption_py INFO | completed 177985
Traceback (most recent call last):
  File "disruption-py/tests/test_against_cache.py", line 120, in <module>
    data_differences = eval_against_cache(
  File "disruption-py/tests/utils/eval_against_sql.py", line 194, in eval_against_cache
    cache_data = get_cached_from_fresh(tokamak, shotlist, fresh_data, test_columns)
  File "disruption-py/tests/utils/eval_against_sql.py", line 88, in get_cached_from_fresh
    shot_data[shot_id] = pd.merge_asof(
  File "site-packages/pandas/core/reshape/merge.py", line 691, in merge_asof
    op = _AsOfMerge(
  File "site-packages/pandas/core/reshape/merge.py", line 1999, in __init__
    _OrderedMerge.__init__(
  File "site-packages/pandas/core/reshape/merge.py", line 1911, in __init__
    _MergeOperation.__init__(
  File "site-packages/pandas/core/reshape/merge.py", line 786, in __init__
    self.left_on, self.right_on = self._validate_left_right_on(left_on, right_on)
  File "site-packages/pandas/core/reshape/merge.py", line 2072, in _validate_left_right_on
    raise MergeError(
pandas.errors.MergeError: Incompatible merge dtype, dtype('O') and dtype('float64'), both sides must have numeric dtype

this seems a testing issue, though, not a broader disruption-py or physics issue (phew!). I'm assigning Amos so he can investigate and fix this, when he gets the chance.

[yumouwei]

I just checked DBeaver and the stored dataset only covers shot numbers between 160942—177061 (13,245 shots in total), which explains why the test failed for the shots I listed on top. Still I think test_against_cache should indicate that there's no data to compare to in this case.

[gtrevisan]

closed by:

• 309