WIP: csv: Fix reading empty columns as float #1612

Closed SGeeversAtVortech closed 1 month ago

In GitLab by @vreeken on Nov 13, 2019, 16:04

We used to use np.genfromtxt with dtype=None, which means that it would guess. If a column did not have any value specified, it would guess the dtype boolean with values False. This would eventually be turned into a Timeseries of 0.0. We however want a Timeseries full of np.nan.

It is difficult to make np.genfromtxt read in floats, when the first column can sometimes be a string. Therefore, we switch to using the stdlib module 'csv', and build the structured/named array ourselves. That way we can make sure that all data values are read in as floats, and that they are NaN when a value is missing.

Note that performance is about the same.

Closes #1128

In GitLab by @vreeken on Nov 13, 2019, 17:03

added 1 commit

13001037 - csv: Fix reading empty columns as float

Compare with previous version

In GitLab by @vreeken on Nov 13, 2019, 17:05

added 1 commit

3b1038e7 - csv: Fix reading empty columns as float

Compare with previous version

In GitLab by @vreeken on Nov 13, 2019, 17:12

added 2 commits

c138a581 - 1 commit from branch master
8475fe17 - csv: Fix reading empty columns as float

Compare with previous version

In GitLab by @vreeken on Nov 15, 2019, 11:38

marked as a Work In Progress

In GitLab by @vreeken on Jan 20, 2020, 14:11

closed

Deltares / rtc-tools

WIP: csv: Fix reading empty columns as float #1612