Difficulting writing missing timeseries values with CDA

[msweier]

Does anyone know how to write a missing value with CDA? Seems like the jython DBAPI values of -340282346638528859811704183484516925440 and -901 don't write to missing with CDA.

I tried writing the missing qualifier as 4997 which did write the correct qualifier, but did not clear the value. Deleting values by time frame not as convenient as just writing a missing.

[MikeNeilson]

if you right click is there a "set missing option" in the DVE? That should use the java Double -Infinity that indicates a missing value.

[msweier]

What's DVE? I guess I want to know if there is a set of characters or values that can be written via CDA to clear the value in the database.

[MikeNeilson]

"Data validation Editor", e.g. what you put in a screen shot of. Sorry, thought that initialism was more common throughout the community.

[MikeNeilson]

So basically right click on the row, should be a set missing that clears it, and try again. I suspect it may still not work but it would be something to go on.

[msweier]

Got it...should be able to see the behavior below.

same result trying to write with the missing qualifier after doing the above w/ the gui

[MikeNeilson]

yeah, and in that case it's just setting the flag not totally clearing out the value. I'm running a test in CDA now, and well, some how we forgot to account for null values.

I have now idea how given how much i've used "null" missing values, but there it is.

[MikeNeilson]

Okay, I found how how to let CDA process null values; we might have had a similar issue in OpenDCS but it uses "Double.NEGATIVE_INFINITY" in the java code as the actual database call expects double vs Double (the first one can't be null) so if the input value is null I set to that special value.

Having some annoyances buttoning down the change and test so likely next week, but will definitely be in for the final release.

[MikeNeilson]

... might have to check the negative infinity on the retrieve. sorry, more work to do, but start of testing is in place.

[msweier]

I did get it to work with Double.NEGATIVE_INFINITY but "-Inifinity" is returned instead of null which can be saved to the database.

Thanks @MikeNeilson for the breadcrumb in https://github.com/USACE/a2w/issues/6. Writing the value as "-Infinity" works, but then it causes issues with cwms-python because we are mixing string and numeric values. In some ways setting missing as"-Infinity" is more intuitive than some numeric code, but then some catches would have to be built into cwms-python and maybe opendcs so type errors aren't triggered. Do you know if "-Infinity" causes OpenDCS CP errors? Sounds like you have a plan on what the null code should be though and it's not "-Infinity".

Tagging @Enovotny so he can track too.

[MikeNeilson]

I'm planning to just get "null" working. If for whatever reason that doesn't work, M or Missing would be the next best. But I don't see any major issue getting plain old null to work.

[msweier]

@MikeNeilson don't strangle me, but this does work in the current CDA version if you set the units to metric. Since I am using English units, the unit change occurred and then the null value was not written. I should've checked this sooner.

[MikeNeilson]

Fair question, but the null/-Float.MAX_VALUE check is performed before any unit conversions are performed. Your code should also leave null (None) or -Float.MAX_VALUE values as they are.

[msweier]

Fair question, but the null/-Float.MAX_VALUE check is performed before any unit conversions are performed. Your code should also leave null (None) or -Float.MAX_VALUE values as they are.

This occurs in the fix you did today, right? The current, production version of CDA will accept -Float.MAX_VALUE if you stay in SI.

[MikeNeilson]

Oh, well then. odd. will have to investigate more. Thanks for looking into that.

In theory with the new version (which hasn't been pushed anywhere yet) it shouldn't be an issue.

[msweier]

Ok, so it's not a unit thing. You have to set the value to -Float.MAX_VALUE (-340282346638528859811704183484516925440) and the quality code to missing (4997) for it to work in the current version. I think that is what you were getting it with setting the flag in the GUI. I'm not sure why I didn't catch it with the GUI test.

[MikeNeilson]

Oh yeah. I was looking at the code in the database. it may automatically add the missing code if the data is null. Otherwise you have to set the flag manually.

There's not really a good way to report that to the user as there's several conditions where it's not an error; at least to the database.

[msweier]

Yeah would be nice if it just would know if you feed it a missing value or a missing qualifier for it write it as missing because I can't think of a case where you would want one without the other.

[MikeNeilson]

We could do that at the API level, but given the database itself supports a setting I want to talk to Perryman about it first.

I agree with you, if it's not at the user level of choice, there should only be the "one way" to save and deal with missing.

[MikeNeilson]

Oh FYI the actually "missing value" quality is 5. or more correctly that that lower 3 bits all have to be set. I think 4997 just happens to meet that definition while encoding other information.

[msweier]

Oh FYI the actually "missing value" quality is 5. or more correctly that that lower 3 bits all have to be set. I think 4997 just happens to meet that definition while encoding other information.

Copy that. value = -340282346638528859811704183484516925440 and quality code = 5 for missing