Replace NaN values

[mmaelicke]

Problem

Our Test data seems to contain numeric substitutes for NaN values. E.g. with one of the testfiles:

from bridget.io import read_TK2_file
df = read_TK2_file('./data/Fendt_TK2_result_2014.csv')
df.CO2.where(df.CO2 < 0).dropna()

yields:

and the plot of the time series looks like:

Suggestion

We should verify, that this is actually a NaN (@sihassl ?) and check the other files and variables for other values (@AlexDo1 ). Is it always the same? Then we need to implement this as a default NaN value option to convert it properly to numpy.NaN.

[mmaelicke]

Could confirm these are NaNs. @AlexDo1, you can now check the other columns and files, if always the exact value is used.

[AlexDo1]

Okay I'll do that and I guess that I will also replace the -9999 values with numpy.NaN values.

[AlexDo1]

It looks like -9999.9003906 is unique for these NaN values (at least in the column CO2), but there are different negative values for CO2, which should be wrong, but I guess these values shouldn´t be replaced with NaN.

[mmaelicke]

Yes, following the header-file the unit is [mmol/m³]. I would guess this is not a concentration, but a flux. So, + and - is direction, which I would guess is upward (+) and downward (-).

Just to be clear: The -9999.9003906 should not be replaced in the sample files, but rather set in the read_*_file function as a default value for nan like:

def read_TK2_file(fname, strict=True, na_values=-9999.9003906):
    [...]

Then passed to the pandas.read_csv function. The correct argument name shoudl be na_values can be found in their docs: https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.read_csv.html

[AlexDo1]

Okay, I did that on the sample data :/ But at least the plotted data looks good when replacing -9999.9003906 with NaN.

I´ll look into the read_TK2_file function now.

[mmaelicke]

I think the only issue we could run into is, when a whole coulmn is NaN only. Then it's removed during data import and the merging with default column names will not work anymore, because of shape mismatch. We should test this....