search

AtlasOfLivingAustralia / data-quality-filter-service

0 stars 2 forks source link

Data Quality filter contentTypes filtering problem #15

Closed kylie-m closed 3 days ago

kylie-m commented 6 months ago

In an all data query (https://biocache.ala.org.au/occurrences/search?q=) with the DQ filters on, there are an expected 1.1 million- ish records excluded based on record type (ie basis of record).

When you click on the filter to see the excluded records, the query doesn't return the 1.1 million-ish records, instead only 279 records, and this seems to be related to how the query is parsed. The FOSSIL_SPECIMEN make it in, but not the "Environmental DNA" contentType

https://biocache.ala.org.au/occurrence/search?q=&qualityProfile=ALA&disableAllQualityFilters=true&fq=basisOfRecord%3A%22FOSSIL_SPECIMEN%22+%28%2BbasisOfRecord%3AMATERIAL_SAMPLE+%2BcontentTypes%3AEnvironmental+DNA%29#tab_mapView

adam-collins commented 5 months ago

The moment we entered anything remotely complex into a filter value the chance that this would work was reduced. Given the very wide variety of inputs possible, this is the most robust solution:

  1. Added a field to QualityCategory for the inverseFilter. This permits overriding the value that is automatically generated and sometimes incorrect.
  2. Updated the UI to put in a warning when the inverseFilter may be required.

Image

In test, please test.

kylie-m commented 5 months ago

Thanks Adam!

@peggynewman I gave this a try on test and it's looking ok to me. Since you initially raised it, could you also give it a try and confirm you're happy with the changes?