Observation not being treated as fainter-than and being classified as Bad (validation type)

[dbenn]

@SaraAAVSO reported this via email:

I just found something a little strange in a plot of R CrB from JD 2454000 to 2455000. You will see one visual observation that is obviously much brighter than the mean at JD 2454629.64. In the database, this observation has a magnitude of <7.9 (with the fainterthan) but for some reason, VStar is not treating it as such.

I also noticed that VStar classifies it as Validation: Bad...I don't see anything wrong with it in the database. ... Here are the results of a search of that part of the star's data using the API: https://www.aavso.org/vsx/index.php?view=api.object&data&ident=%22R%20CrB%22&fromjd=2454629&tojd=2454630

[dbenn]

I replied in email that:

I’ve been debugging this.

There’s a couple of things going on here.

The web service call used by VStar returns CSV format not XML element or XML attribute based, on the grounds that CSV was faster. We moved towards this over time.

Here’s the CSV URL for the current case (add &csv):

https://www.aavso.org/vsx/index.php?view=api.object&data&ident=%22R%20CrB%22&fromjd=2454629&tojd=2454630&csv

It starts to lose the plot in this line (the JD you mentioned):

2454629.643,7.9,,Vis.,PPL,"B","79W",,"b chart AAVSO",""b" scale chart. clear, moon up. 70mm binoculars.",,,V,,,R CRB,AAVSO,,,,,1272635022,1,Visual,,"Plante, Philip",US

There’s some inconsistent/incorrect quoting going on:

"b chart AAVSO",""b" scale chart. clear, moon up. 70mm binoculars.”

Note the 2 quotes: ""b" scale chart ...

So the web service in CSV mode needs to quote all string fields and escape quotes but it is not. We had this problem with download format CSV files quite awhile ago as I recall then it was fixed.

The other issue is that the mag is returned as 7.9 in the CSV with 1 (true) in the fainter-than column but this is not read correctly for the same reason that the val (validation type) is not: incorrect quoting of CSV columns throws the CSV reader code out.

Note that if you select Minimal Fields in the AID load dialog, the correct ob info is loaded (fainter than and validation flag).

In the XML element format we have an element for fainter than, validation type etc and the XML attribute web service response has similar info:

1272635022 2454629.643 true 7.9 Visual Vis. PPL B 79W b chart AAVSO "b" scale chart. clear, moon up. 70mm binoculars. V R CRB AAVSO Plante, Philip US

I switched back to my old XML reader code to test that it reads correctly. It does, just more slowly.

So, my conclusion is that this is an error originating in the web service which is not quoting fields. Maybe I need to add another parameter to the URL to request a quoted CSV col response or maybe the web service has a bug.

[dbenn]

An additional URL parameter like “&quoted-csv” could be added that returns a quoted response, maintaining backward compatibility, then later “&csv" could be deprecated and removed (or just left).

[dbenn]

In correspondence with @SaraAAVSO, the test server with corrected quoting fixes the error at JD 2454629.64 (no longer visible at the cross hairs in the attached plot).

I note however that there is mag 7.4 observation that perhaps should be marked as discrepant. It's apparently not a fainter-than:

2454391.33,7.4,,Vis.,DHJ,,,,"41.03","B10x50",,,U,,,R CRB,,STD,,,BAAVSS,1284163544,0,Visual,,"Davies, H",

[dbenn]

I'll leave this issue open just to keep track of the problem until the bug fix is put into production @SaraAAVSO.

[dbenn]

@SaraAAVSO reported that this was fixed in the VSX API and I can confirm this.