WHOIGit / ifcbdb

IFCB dashboard
MIT License
8 stars 10 forks source link

return NaN, not zero, in metadata export for bins w/o depth information #381

Open joefutrelle opened 1 year ago

joefutrelle commented 1 year ago

During metadata upload, a depth value for a bin is not set if there is no value in the uploaded CSV:

https://github.com/WHOIGit/ifcbdb/blob/36cac457b31b614b7eda2c15e28cbd2fbba7d388/ifcbdb/dashboard/accession.py#L368-L371

During metadata download, there are three cases: 1) depth is set on the bin, 2) depth is set on the dataset, and 3) depth is NaN.

https://github.com/WHOIGit/ifcbdb/blob/36cac457b31b614b7eda2c15e28cbd2fbba7d388/ifcbdb/dashboard/accession.py#L521-L526

joefutrelle commented 1 year ago

This code may be relevant as well, since it's used in metadata upload and relies on pd.isnull.

https://github.com/WHOIGit/ifcbdb/blob/36cac457b31b614b7eda2c15e28cbd2fbba7d388/ifcbdb/dashboard/accession.py#L287-L296

mike-kaimika commented 5 months ago

@joefutrelle On the master branch, I'm not seeing things as outline in the original description here. I have a dataset where a few of the bins do not have depths set, and the value that comes out in the CSV file is an empty string, rather than a zero. Is that the expected outcome of this ticket (in which case, no changes are needed)?

My other thought would be that you might want an explicit NaN value in the CSV - we can do that with an argument to represent NaN values with some other string: df.to_csv(csv_buf, mode='wb', na_rep='NaN', **kw)

However, should that affect other columns besides depth as well? For example, the above change worked on the depth column, but you can see it also produced NaN values for latitude, longitude an some other columns as well image

joefutrelle commented 4 months ago

I'm removing this from 4.2 until we can do more investigation.