search

ukaea / fair-mast

A data management system for Tokamak data
MIT License
2 stars 2 forks source link

Use NaN or None for unset data instead of -999. #75

Open Simon-McIntosh opened 6 days ago

Simon-McIntosh commented 6 days ago

When accessing the radial location of the primary x-point for shots 30420 and 30471 at time_index=20 I find 0.4551602 and -999. respectively.

I assume that a value of -999. indicates unset or missing data. Would it be possible to update this value with NaN or None so that these values may be automatically ignored when visualizing the data.

samueljackson92 commented 6 days ago

Yes. I believe that is correct and that -999 was originally used as a placeholder in the old EFIT data. I agree that NaN would be a more sensible choice. We can get this updated 👍

Simon-McIntosh commented 6 days ago

Thanks, I had just realized this too. In fact there are -999. values in both shots that I mention along with NaNs also.

<xarray.DataArray 'xpoint2_rc' (time: 64)> Size: 256B
array([           nan,            nan,            nan,            nan,
                  nan,            nan,            nan,            nan,
                  nan,            nan,            nan, -9.9900000e+02,
       -9.9900000e+02, -9.9900000e+02, -9.9900000e+02, -9.9900000e+02,
       -9.9900000e+02, -9.9900000e+02, -9.9900000e+02,  4.4878340e-01,
        4.5516020e-01,  4.7110128e-01,  4.9078003e-01,  5.0676501e-01,
        5.1959819e-01, -9.9900000e+02,  5.4396850e-01,  5.5237395e-01,
        5.5786538e-01,  5.6552464e-01,  5.6224066e-01,  5.5595833e-01,
        5.5617350e-01,  5.5559403e-01,  5.5321759e-01,  5.5510604e-01,
        5.4928404e-01,  5.5733383e-01,  5.6607568e-01,  5.7135868e-01,
        5.7491392e-01,  5.7675600e-01,  5.7752812e-01,  5.7968068e-01,
        5.8110207e-01,  5.8135033e-01,  5.7985163e-01,  5.8129978e-01,
        5.8152688e-01,  5.8445030e-01,  5.8395177e-01,  5.8924192e-01,
        5.9030801e-01,  5.9117389e-01,  5.9019601e-01,  5.8214122e-01,
        5.7863563e-01,  5.7821310e-01,  5.7855153e-01,  5.7805151e-01,
        5.7672918e-01,  5.7718468e-01,  5.7620615e-01,  5.7676393e-01],
      dtype=float32)
samueljackson92 commented 6 days ago

We regridded EFIT data was onto the same time axis. Some data points won't have values defined for all time steps if they were not defined for that part of the EFIT run, hence the NaN's.

It looks like some slices were "defined" in the original dataset but set to the nominal -999 value in original data. I think we map -999 o NaN and then it's consistent.