PASTAplus / dex

Explore and subset CSV tables using associated EML metadata
Apache License 2.0
3 stars 0 forks source link

Check if invalid integer cells still causes columns to be converted to floating point #72

Open rogerdahl opened 8 months ago

rogerdahl commented 8 months ago

IEEE floating point values have an embedded NaN representation. That is, there is a bit pattern for NaN. However, plain integers do not have an NaN representation (all available bit patterns are assigned to valid integers). Pandas deals with this by converting integer columns to floating point columns if any of the cells in the column do not contain valid integers, and then using the floating point NaN representation for those cells. A change already implemented in the integer parser in DeX may have fixed this. We'll need to set up a specific combination of EML with column declared as integer, and a table with invalid integers to check.