The core BASIN-3D application that uses a generalized data synthesis model that applies across a variety of earth science observation types (hydrology, geochemistry, climate etc.). https://basin3d.readthedocs.io/
Is your feature request related to a problem? Please describe.
basin3d doesn't support a common missing value. If a datasource's missing value is a numeric, it gets passed thru as if it is a result value.
For consideration:
have a basin3d missing value vocabulary
map missing value from data source to basin3d
catch and translate the datasource missing value
potentially: filter by missing value
Examples from USGS. Note, need to figure out what happens with non-numeric values like NaN, NA.
Using django-basin3d: ~/measurement_tvp_timeseries/?monitoring_features=USGS-09147500&observed_property_variables=RDC&start_date=2023-02-01&end_date=2023-02-10
Return: USGS missing value -999999 is converted to -28316.818683153
'''
"statistic": "MEAN",
"result_points": [
["2023-02-01T00:00:00.000",-28316.818683153],
["2023-02-02T00:00:00.000",-28316.818683153],
["2023-02-03T00:00:00.000",-28316.818683153],
["2023-02-04T00:00:00.000",-28316.818683153],
["2023-02-05T00:00:00.000",-28316.818683153],
["2023-02-06T00:00:00.000",1.5149513145],
["2023-02-07T00:00:00.000",1.5800800625999998],
["2023-02-08T00:00:00.000",-28316.818683153],
["2023-02-09T00:00:00.000",1.614060279],
["2023-02-10T00:00:00.000",-28316.818683153]
],
'''
"""
{"name":"ns1:timeSeriesResponseType","declaredType":"org.cuahsi.waterml.TimeSeriesResponseType","scope":"javax.xml.bind.JAXBElement$GlobalScope","value":{"queryInfo":{"queryURL":"http://waterservices.usgs.gov/nwis/dv/parameterCd=00060&startDT=2023-02-01&endDT=2023-02-10&sites=09147500&format=json&siteType=ST&siteStatus=all","criteria":{"locationParam":"[ALL:09147500]","variableParam":"[00060]","timeParam":{"beginDateTime":"2023-02-01T00:00:00.000","endDateTime":"2023-02-10T00:00:00.000"},"parameter":[]},"note":[{"value":"[ALL:09147500]","title":"filter:sites"},{"value":"[ST]","title":"filter:siteType"},{"value":"[mode=RANGE, modifiedSince=null] interval={INTERVAL[2023-02-01T00:00:00.000-05:00/2023-02-10T00:00:00.000-05:00]}","title":"filter:timeRange"},{"value":"methodIds=[ALL]","title":"filter:methodId"},{"value":"2023-04-21T15:38:38.319Z","title":"requestDT"},{"value":"95cb98e0-e05a-11ed-9320-2cea7f58f5ca","title":"requestId"},{"value":"Provisional data are subject to revision. Go to http://waterdata.usgs.gov/nwis/help/?provisional for more information.","title":"disclaimer"},{"value":"vaas01","title":"server"}]},"timeSeries":[{"sourceInfo":{"siteName":"UNCOMPAHGRE RIVER AT COLONA, CO","siteCode":[{"value":"09147500","network":"NWIS","agencyCode":"USGS"}],"timeZoneInfo":{"defaultTimeZone":{"zoneOffset":"-07:00","zoneAbbreviation":"MST"},"daylightSavingsTimeZone":{"zoneOffset":"-06:00","zoneAbbreviation":"MDT"},"siteUsesDaylightSavingsTime":true},"geoLocation":{"geogLocation":{"srs":"EPSG:4326","latitude":38.33143056,"longitude":-107.779225},"localSiteXY":[]},"note":[],"siteType":[],"siteProperty":[{"value":"ST","name":"siteTypeCd"},{"value":"14020006","name":"hucCd"},{"value":"08","name":"stateCd"},{"value":"08091","name":"countyCd"}]},"variable":{"variableCode":[{"value":"00060","network":"NWIS","vocabulary":"NWIS:UnitValues","variableID":45807197,"default":true}],"variableName":"Streamflow, ft³/s","variableDescription":"Discharge, cubic feet per second","valueType":"Derived Value","unit":{"unitCode":"ft3/s"},"options":{"option":[{"value":"Mean","name":"Statistic","optionCode":"00003"}]},"note":[],"noDataValue":-999999.0,"variableProperty":[],"oid":"45807197"},"values":[{"value":[{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-01T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-02T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-03T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-04T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-05T00:00:00.000"},{"value":"53.5","qualifiers":["P"],"dateTime":"2023-02-06T00:00:00.000"},{"value":"55.8","qualifiers":["P"],"dateTime":"2023-02-07T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-08T00:00:00.000"},{"value":"57.0","qualifiers":["P"],"dateTime":"2023-02-09T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-10T00:00:00.000"}],"qualifier":[{"qualifierCode":"Ice","qualifierDescription":"Value is affected by ice at the measurement site.","qualifierID":0,"network":"NWIS","vocabulary":"uv_rmk_cd"},{"qualifierCode":"P","qualifierDescription":"Provisional data subject to revision.","qualifierID":1,"network":"NWIS","vocabulary":"uv_rmk_cd"}],"qualityControlLevel":[],"method":[{"methodDescription":"","methodID":18989}],"source":[],"offset":[],"sample":[],"censorCode":[]}],"name":"USGS:09147500:00060:00003"}]},"nil":false,"globalScope":true,"typeSubstituted":false}
"""
Is your feature request related to a problem? Please describe. basin3d doesn't support a common missing value. If a datasource's missing value is a numeric, it gets passed thru as if it is a result value.
For consideration:
Examples from USGS. Note, need to figure out what happens with non-numeric values like NaN, NA.
Using django-basin3d: ~/measurement_tvp_timeseries/?monitoring_features=USGS-09147500&observed_property_variables=RDC&start_date=2023-02-01&end_date=2023-02-10
Return: USGS missing value -999999 is converted to -28316.818683153 ''' "statistic": "MEAN", "result_points": [ ["2023-02-01T00:00:00.000",-28316.818683153], ["2023-02-02T00:00:00.000",-28316.818683153], ["2023-02-03T00:00:00.000",-28316.818683153], ["2023-02-04T00:00:00.000",-28316.818683153], ["2023-02-05T00:00:00.000",-28316.818683153], ["2023-02-06T00:00:00.000",1.5149513145], ["2023-02-07T00:00:00.000",1.5800800625999998], ["2023-02-08T00:00:00.000",-28316.818683153], ["2023-02-09T00:00:00.000",1.614060279], ["2023-02-10T00:00:00.000",-28316.818683153] ], '''
Corresponding USGS direct query: https://waterservices.usgs.gov/nwis/dv/?parameterCd=00060&startDT=2023-02-01&endDT=2023-02-10&sites=09147500&format=json&siteType=ST&siteStatus=all
""" {"name":"ns1:timeSeriesResponseType","declaredType":"org.cuahsi.waterml.TimeSeriesResponseType","scope":"javax.xml.bind.JAXBElement$GlobalScope","value":{"queryInfo":{"queryURL":"http://waterservices.usgs.gov/nwis/dv/parameterCd=00060&startDT=2023-02-01&endDT=2023-02-10&sites=09147500&format=json&siteType=ST&siteStatus=all","criteria":{"locationParam":"[ALL:09147500]","variableParam":"[00060]","timeParam":{"beginDateTime":"2023-02-01T00:00:00.000","endDateTime":"2023-02-10T00:00:00.000"},"parameter":[]},"note":[{"value":"[ALL:09147500]","title":"filter:sites"},{"value":"[ST]","title":"filter:siteType"},{"value":"[mode=RANGE, modifiedSince=null] interval={INTERVAL[2023-02-01T00:00:00.000-05:00/2023-02-10T00:00:00.000-05:00]}","title":"filter:timeRange"},{"value":"methodIds=[ALL]","title":"filter:methodId"},{"value":"2023-04-21T15:38:38.319Z","title":"requestDT"},{"value":"95cb98e0-e05a-11ed-9320-2cea7f58f5ca","title":"requestId"},{"value":"Provisional data are subject to revision. Go to http://waterdata.usgs.gov/nwis/help/?provisional for more information.","title":"disclaimer"},{"value":"vaas01","title":"server"}]},"timeSeries":[{"sourceInfo":{"siteName":"UNCOMPAHGRE RIVER AT COLONA, CO","siteCode":[{"value":"09147500","network":"NWIS","agencyCode":"USGS"}],"timeZoneInfo":{"defaultTimeZone":{"zoneOffset":"-07:00","zoneAbbreviation":"MST"},"daylightSavingsTimeZone":{"zoneOffset":"-06:00","zoneAbbreviation":"MDT"},"siteUsesDaylightSavingsTime":true},"geoLocation":{"geogLocation":{"srs":"EPSG:4326","latitude":38.33143056,"longitude":-107.779225},"localSiteXY":[]},"note":[],"siteType":[],"siteProperty":[{"value":"ST","name":"siteTypeCd"},{"value":"14020006","name":"hucCd"},{"value":"08","name":"stateCd"},{"value":"08091","name":"countyCd"}]},"variable":{"variableCode":[{"value":"00060","network":"NWIS","vocabulary":"NWIS:UnitValues","variableID":45807197,"default":true}],"variableName":"Streamflow, ft³/s","variableDescription":"Discharge, cubic feet per second","valueType":"Derived Value","unit":{"unitCode":"ft3/s"},"options":{"option":[{"value":"Mean","name":"Statistic","optionCode":"00003"}]},"note":[],"noDataValue":-999999.0,"variableProperty":[],"oid":"45807197"},"values":[{"value":[{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-01T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-02T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-03T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-04T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-05T00:00:00.000"},{"value":"53.5","qualifiers":["P"],"dateTime":"2023-02-06T00:00:00.000"},{"value":"55.8","qualifiers":["P"],"dateTime":"2023-02-07T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-08T00:00:00.000"},{"value":"57.0","qualifiers":["P"],"dateTime":"2023-02-09T00:00:00.000"},{"value":"-999999","qualifiers":["P","Ice"],"dateTime":"2023-02-10T00:00:00.000"}],"qualifier":[{"qualifierCode":"Ice","qualifierDescription":"Value is affected by ice at the measurement site.","qualifierID":0,"network":"NWIS","vocabulary":"uv_rmk_cd"},{"qualifierCode":"P","qualifierDescription":"Provisional data subject to revision.","qualifierID":1,"network":"NWIS","vocabulary":"uv_rmk_cd"}],"qualityControlLevel":[],"method":[{"methodDescription":"","methodID":18989}],"source":[],"offset":[],"sample":[],"censorCode":[]}],"name":"USGS:09147500:00060:00003"}]},"nil":false,"globalScope":true,"typeSubstituted":false} """