https://github.com/astropy/astropy/issues/13331 highlighted that the ECSV specification formally allows for flexibility in the marker used to indicate missing data, but there is no way within the ECSV standard to communicate to readers what that marker is.
In particular, the Gaia project has generated ECSV files that use null for missing data. This requires a minor workaround to read with astropy. Reading the Gaia DR3 files with TOPCAT/STILTS is possible since it accepts null for missing values in numeric columns (with a performance penalty in the current implementation), but those null values are passed through normally for string columns.
This issue is intended to discuss options for updating the ECSV specification to provide unambiguous treatment of missing values.
One idea is to add an optional table-level keyword that specifies the string value that signifies missing data. This same optional keyword could also be included in the column keywords to provide per-column specification of the missing data marker for each column.
https://github.com/astropy/astropy/issues/13331 highlighted that the ECSV specification formally allows for flexibility in the marker used to indicate missing data, but there is no way within the ECSV standard to communicate to readers what that marker is.
In particular, the Gaia project has generated ECSV files that use
nullfor missing data. This requires a minor workaround to read with astropy. Reading the Gaia DR3 files with TOPCAT/STILTS is possible since it acceptsnullfor missing values in numeric columns (with a performance penalty in the current implementation), but thosenullvalues are passed through normally for string columns.This issue is intended to discuss options for updating the ECSV specification to provide unambiguous treatment of missing values.
One idea is to add an optional table-level keyword that specifies the string value that signifies missing data. This same optional keyword could also be included in the column keywords to provide per-column specification of the missing data marker for each column.