Geospatial assertions not being caught in DQ filters

nickdos commented 3 years ago

https://biocache.ala.org.au/occurrences/search?q=country%3A%22Australia%22&qualityProfile=ALA#tab_mapView

Shows lots of records on the map that I would expect to have been removed by the ALA DQ profile. See:

Selecting a record from the bunch over Japan, shows it has "Latitude is negated" warning (PRESUMED_NEGATED_LATITUDE), but this has not excluded it from the "ALA general" DQ profile.

I'm wondering about the following:

previously the ALA corrected such errors in the processed version of decimalLatitude but this is no longer happening. So checking that was an intended change and not a regression bug?
I'm thinking the warning above should be included in the ALA general profile section called "Exclude records with additional spatial quality issues", which does not list this assertion (PRESUMED_NEGATED_LATITUDE).

nickdos commented 3 years ago

Other assertions appearing on records that are being shown:

STATE_COORDINATE_MISMATCH
PRESUMED_NEGATED_LATITUDE
PRESUMED_NEGATED_LONGITUDE

nickdos commented 3 years ago

After discussing this, we've decided to try and include the assertions in the ALA general profile, as suggested above.

nickdos commented 3 years ago

Using the biocache-test, I've added the 2 PRESUMED_NEGATED_L... assertions to the ALA general profile ("Exclude records with additional spatial quality issues" section) and it looks to have worked:

nickdos commented 3 years ago

Fixed in prod

AtlasOfLivingAustralia / la-pipelines

Geospatial assertions not being caught in DQ filters #510