search

Riverscapes / riverscapes-tools

Open-source Python 3.0 tools for the Riverscapes organization
https://tools.riverscapes.net/
GNU General Public License v3.0
11 stars 11 forks source link

CSV Import Doesn't Respect NULLs #1006

Closed philipbaileynar closed 2 months ago

philipbaileynar commented 2 months ago

Reviewing this RME project I noticed that in the metrics database table there are some values that should be NULL. DataGrip makes this clear to see.

Screenshot 2024-08-23 at 1 28 39β€―PM

These data are imported from this CSV:

metric_id,name,machine_code,data_type,field_name,description,small,medium,large,method,metric_group_id,is_active,docs_url
1,Ownership,AGENCY,TEXT,rme_dgo_ownership,,200,400,1400,,3,1,
2,State,STATE,TEXT,rme_dgo_state,,200,400,1400,,3,1,
3,County,COUNTY,TEXT,rme_dgo_county,,200,400,1400,,3,1,
4,Primary Channel Gradient,STRMGRAD,REAL,RME_IGO_prim_channel_gradient,Change in elevation along Flowline length,200,400,1400,(π‘‘π‘œπ‘ π‘œπ‘“ π‘Ÿπ‘’π‘Žπ‘β„Ž π‘’π‘™π‘’π‘£π‘Žπ‘‘π‘–π‘œπ‘› βˆ’π‘π‘œπ‘‘π‘‘π‘œπ‘š π‘œπ‘“ π‘Ÿπ‘’π‘Žπ‘β„Ž π‘’π‘™π‘’π‘£π‘Žπ‘‘π‘–π‘œπ‘›)/(π‘›π‘’π‘‘π‘€π‘œπ‘Ÿπ‘˜ π‘™π‘’π‘›π‘”π‘‘β„Ž),1,1,"Hack, J. T. (1973). Stream-profile analysis and stream-gradient index. Journal of Research of the U.S. Geological Survey, 1(4), 421–429. Nagel, D., Buffington, J., & Isaak, D. (4006). Comparison of methods for estimating stream channel gradient using GIS. USDA Forest Service, Rocky Mountain Research Station Boise Aquatic Sciences Lab."
5,Valley Gradient,VALGRAD,REAL,RME_IGO_valleybottom_gradient,Change in elevation along Centerline length,200,400,1400,(π‘‘π‘œπ‘ π‘œπ‘“ π‘ π‘’π‘”π‘šπ‘’π‘›π‘‘ 𝑒𝑙𝑒𝑣. βˆ’π‘π‘œπ‘‘π‘‘π‘œπ‘š π‘œπ‘“ π‘ π‘’π‘”π‘šπ‘’π‘›π‘‘ 𝑒𝑙𝑒𝑣.)/(π‘£π‘Žπ‘™π‘™π‘’π‘¦ π‘π‘œπ‘‘π‘‘π‘œπ‘š π‘π‘’π‘›π‘‘π‘’π‘Ÿπ‘™π‘–π‘›π‘’),1,1,
6,Relative Flow Length,RELFLWLNGTH,REAL,RME_IGO_rel_flow_length,,200,400,1400,sum of all blue line lenth/valley length,1,1,
7,Valley Azimuth,VALAZMTH,REAL,RME_IGO_valley_azimuth,,200,400,1400,,,0,
8,Confluence Count,CONF,INTEGER,RME_DGO_confluences,,200,400,1400,,2,1,
9,Diffluence Count,DIFF,INTEGER,RME_DGO_diffluences,,200,400,1400,,2,1,
10,Tributary Junction Count,TRIBS,REAL,RME_IGO_trib_per_km,,400,400,1400,,2,1,

I don't understand why some missing values are correctly being set to NULL while others are not. Probably an easy fix to check for either empty string or None.

Note that I believe the code that imports the CSV is used by several of our models. Definitely BRAT as well.

r-k-g commented 2 months ago

Fixed with commit https://github.com/Riverscapes/riverscapes-tools/commit/b94c53ee41032625fc10830cd071b51d2d8033cb.

With the schema fix, there are no NULL values in this column. This is expected behaviour because that column is present in the CSV, just with no content. NULL would be indicative of a missing column or other issue.