Identify lower quality records...

[M-Nicholls]

... so that the data set that is trusted

Identify and filter incomplete records:

• no value supplied in core elements - eventDate, scientificName, decimalLatitude, decimaLongitude

https://github.com/AtlasOfLivingAustralia/DataQuality/issues/249

Identify and filter invalid records:

Values are

• unreadable due to data type, length, format (this is different to doesn’t match a vocabulary)
• out of range values e.g. coordinates

Identify and filter potentially incorrect records:

Automated identification of incorrect records:

• spatial outlier detection

  • expert distribution
  • ALA generated convex hull https://github.com/AtlasOfLivingAustralia/DataQuality/issues/251

• environmental outlier detection https://github.com/AtlasOfLivingAustralia/DataQuality/issues/250

• taxonomic outlier detection (scientific names not in the names lists) https://github.com/AtlasOfLivingAustralia/DataQuality/issues/252

Manually identify and filter incorrect records

• user flagged incorrect record
  • indivdually
  • in bulk https://github.com/AtlasOfLivingAustralia/DataQuality/issues/253

Identify and filter duplicate records

• Duplicate detection

Identify and filter less authoritative records

• Lacking supporting and contextual fields – validation status, georeferenced date and method etc
• Internally inconsistent – e.g. wrong state for coordinates, coordinates are too accurate
• Non standard values – e.g. units, vocabulary matching
• Records with default values – centre of Aus, centre of state/province, 1st of month, year, century.

Identify not fit for purpose records

• Precision requirements – coordinate precision and uncertainty, taxonomic rank, temporal precision
• Remove record types not suitable e.g. cultivated, eDNA, fossil, absence, pre-1700

[M-Nicholls]

For each filter needed establish:

• what needs to be filtered,
• is there an assertion or field(s) already,
• if so does the assertion work correctly
• is the field processed correctly - e.g. date parsing issues, update field processing
• if not design and implement the assertion or field(s) and add to the data pre-filters

wherever an assertion is used to filter, check that that the assertion is operating correctly e.g. https://biocache.ala.org.au/occurrences/search?q=assertions:RECORDED_DATE_INVALID

many of the records appear to have valid dates: https://biocache.ala.org.au/occurrences/4eaa0bd1-5bb0-4e40-9452-31e8afb0a040