Issues and questions with assertions

RobinaSanderson commented 3 years ago

The review of the wiki pages documenting assertions identified some problems with the assertions as currently implemented, some things that would improve the assertions and some questions about assertions. I've listed these all in this issue as an epic. The review is documented at: https://docs.google.com/spreadsheets/d/1-DC_8rEmW0dLYDqmfIdyC8TD4smB3tIUFEmKVnHDdfg/edit#gid=820855861

Sub-issues or tasks could be generated as a decision is taken on what should be done and how to work on the assertions.

Bugs or not implemented The following were identified as bugs or not implemented in the pipelines code.

[ ] UNCERTAINTY_NOT_SPECIFIED - Need to check how this assertion is being calculated... I get no records for this query: https://biocache.ala.org.au/occurrences/search?q=assertions%3A%22UNCERTAINTY_NOT_SPECIFIED%22&disableAllQualityFilters=true, but given that over 45m records don't have an uncertainty value (using this query: https://biocache.ala.org.au/occurrences/search?q=-coordinate_uncertainty%3A*), I expect something is wrong.
[ ] COORDINATE_UNCERTAINTY_METERS_INVALID - of the 49m records with this assertion, 45.4m records simply have no value (which possibly should be another assertion). Of the remainder, 383k records have a value greater than 5km and only about 1200 have an uncertainty value greater than 10km, the rest seem to have a realistic value (eg: 1.2m of these records have a value of 100m). The current code sets the valid range up to 5000km so something is going wrong.
[ ] GEOREFERENCED_DATE_UNLIKELY - The implementation is looks like it is for GEOREFERENCED_DATE_INVALID.
[ ] GEOREFERENCED_DATE_INVALID - Not implemented. Although, the implementation of GEOREFERENCED_DATE_UNLIKELY in the code seems to be more like checking for invalid date
[ ] AMBIGUOUS_INSTITUTION - Not implemented
[ ] AMBIGUOUS_COLLECTION - Not implemented
[ ] RECORDED_DATE_UNLIKELY - This is incorrectly flagged for some records. Looks more like RECORDED_DATE_INVALID https://biocache.ala.org.au/occurrences/1ec5d6e4-8d4d-4cfe-bd39-818616f303bf, https://biocache.ala.org.au/occurrences/b508e682-07e6-4a68-9a08-1ffe37a7eaf7
[ ] MODIFIED_DATE_INVALID - Not implemented
[ ] IDENTIFIED_DATE_INVALID - Not implemented
[ ] UNRECOGNISED_INSTITUTION_CODE - Not implemented
[ ] INTERPRETATION_ERROR - This appears not to be implemented, and seems to be pretty generic. Do we want this implemented? If so, what information should be given in the wiki page about when it occurs and how to fix it?
[ ] TAXON_MATCH_AGGREGATE - Not implemented. Should this be removed?

Improvements These are all improvements as the assertions currently do what is described but could work better.

[ ] MISSING_GEOREFERENCE_DATE - Note that nearly all ALA records have this assertion, so I question the value of it. In fact given the number of records without this field, it might be more value to flag when it is included because it might indicate a manual calculation of coords.
[ ] COORDINATE_REPROJECTED - Currently we only raise this issue if the reprojected value doesn't match the original value provided. We should raise assertion if reprojected regardless of whether the value changes. Should also use verbatimSRS in calculation - need to check whether this is being used.
[ ] INDIVIDUAL_COUNT_INVALID - If the number is given as a float rather than an integer, this issue is raised. It would be better just to interpret the float where it is an integer followed by ".0"
[ ] ID_PRE_OCCURRENCE - Assertion is being raised where the date identified is to the year only and the date identified is day month year. Maybe it is using 01/01/Year when it should use 31/12/year.
[ ] TYPE_STATUS_INVALID - This is currently implemented using a vocabulary, TDWG allows free text in this field, so values that are valid by the standard will be considered invalid in our implementation.

Questions These were raised as questions in the subject matter review, and will need Dev investigation or a policy decision.

[ ] ELEVATION_NON_NUMERIC - Is this a matter of having the wrong data type for minumumElevationInMetres and/or maximumElevationInMetres, or can the non-numeric values come from verbatimElevation?
[ ] DEPTH_NON_NUMERIC - Same as above
[ ] COORDINATES_CENTRE_OF_STATEPROVINCE - How is this determined? And what is the list or shapefile and version (or other source) for determining the centre point value?
[ ] COORDINATES_CENTRE_OF_COUNTRY - As above
[ ] COUNTRY_COORDINATE_MISMATCH - What is the shapefile(s) and versions used to place coordinates? Add this to the wiki page.
[ ] MISSING_GEOREFERENCESOURCES - Note that 97% of ALA records have this assertion, is it useful?
[ ] MISSING_GEOREFERENCEVERIFICATIONSTATUS - Note the wiki description refers to the TDWG page for this field and recommends a controlled vocabulary even though all the records I sampled had a value in dwc:georeferenceVerificationStatus that didn't match any of the suggested vocab - is the vocab useful and do we need to map the existing terms to the vocab as processed values?
[ ] OCCURRENCE_STATUS_INFERRED_FROM_BASIS_OF_RECORD - The level of issue for this should be "comment" not "warning". The level is not listed on the spreadsheet and should be checked.
[ ] ID_PRE_OCCURRENCE - Assertion is being raised where the date identified is to the year only and the date identified is day month year. Maybe it is using 01/01/Year when it should use 31/12/year?
[ ] GEOREFERENCE_POST_OCCURRENCE - How should this work? Records fail the test even if the eventDate and the georeferencedDate are ostensibly the same. I think that the test is to indicate whether the georeference was done when the specimen was collected or the observation was made assuming it is more accurate then, but that does not make sense if you know how people collect (e.g. take a GPS reading in the car park and then walk a five kilometer loop) and also not because collectors never supply georeferenced date and we always assume that when the georeference is supplied by the collector the georeferenced date is the same as the collecting date.