Errors displayed with "Data licensed for all uses" filter profile

M-Nicholls commented 4 years ago

error when viewing excluded records for reusable records: https://biocache-dq-test.ala.org.au/occurrences/search?q=*%3A*&qualityProfile=re-usable#tab_mapView

error when interacting (e.g. adding user filters) with the open licensed data profile: Error from server at http://aws-sc7b.ala:8983/solr/biocache: org.apache.solr.search.SyntaxError: Cannot parse 'license:CC-BY-NC\ license:CC-BY-NC\ 4.0\ (Int)\ license:CC-BY-NC\ 3.0\ (Aus)\ license:Creative\ Commons\ -\ license\ at\ record\ level\ license:CC-BY-NC-SA\ 4.0\ (Int)\ license:CC-BY-NC-ND\ 4.0\ (Int)\ license:other\ license:CC-BY-SA\ 4.0\ (Int)\ license:CC-BY-NC\ 3.0\ (Au)\ license:CC\ BY-NC-Int\ license:CC-BY-NC-Int\ license:Custom\ license:CC-BY-NC\ 2.5\ (Int)\ license:CC-BY-NC-Aus\ license:CC-BY-ND\ 4.0\ (Int)': Encountered " ")" ") "" at line 1, column 474. Was expecting one of: ... ... ... "+" ... "-" ... ... "(" ... "*" ... "^" ... ... ... ... ... ... ... "[" ... "{" ... ... "filter(" ... ...

M-Nicholls commented 4 years ago

If difficult to fix we can hide the profile until it's fixed

alexhuang091 commented 4 years ago

The root cause of this issue is when fq=-license:"CC-BY-NC" AND -license:"CC-BY-ND 4.0 (Int)" and we pass it into the parser, the clauses discard the "". So the constructed inversefq=license:CC-BY-NC license:CC-BY-ND 4.0 (Int) is incorrect, it should have "" around the query.

We need to escape the " before doing the parse.

alexhuang091 commented 4 years ago

This is fixed.

Incorrect inversed fq license:CC-BY-NC license:CC-BY-NC 4.0 (Int) license:CC-BY-NC 3.0 (Aus) license:Creative Commons - license at record level license:CC-BY-NC-SA 4.0 (Int) license:CC-BY-NC-ND 4.0 (Int) license:other license:CC-BY-SA 4.0 (Int) license:CC-BY-NC 3.0 (Au) license:CC BY-NC-Int license:CC-BY-NC-Int license:Custom license:CC-BY-NC 2.5 (Int) license:CC-BY-NC-Aus license:CC-BY-ND 4.0 (Int)

is now changed to license:"CC-BY-NC" license:"CC-BY-NC 4.0 (Int)" license:"CC-BY-NC 3.0 (Aus)" license:"Creative Commons - license at record level" license:"CC-BY-NC-SA 4.0 (Int)" license:"CC-BY-NC-ND 4.0 (Int)" license:"other" license:"CC-BY-SA 4.0 (Int)" license:"CC-BY-NC 3.0 (Au)" license:"CC BY-NC-Int" license:"CC-BY-NC-Int" license:"Custom" license:"CC-BY-NC 2.5 (Int)" license:"CC-BY-NC-Aus" license:"CC-BY-ND 4.0 (Int)"

But since biocache-service needs update to handle this https://github.com/AtlasOfLivingAustralia/biocache-service/issues/502 so it all depends on biocache-service now. I locally reverted the buggy fix I did to biocache-service, it can handle this new fq correctly so nothing to worry about.

Also I tested 'view excluded records' with 'CSDM' and 'Default' profile, the new change doesn't break anything.

M-Nicholls commented 4 years ago

alexhuang091 commented 4 years ago

the inverse of the open license groups is not producing an error any more but there are no records returned: https://biocache-dq-test.ala.org.au/occurrences/search?q=*%3A*&qualityProfile=re-usable&disableAllQualityFilters=true&fq=license%3A%22CC-BY-NC%22+license%3A%22CC-BY-NC+4.0+%28Int%29%22+license%3A%22CC-BY-NC+3.0+%28Aus%29%22+license%3A%22Creative+Commons+-+license+at+record+level%22+license%3A%22CC-BY-NC-SA+4.0+%28Int%29%22+license%3A%22CC-BY-NC-ND+4.0+%28Int%29%22+license%3A%22other%22+license%3A%22CC-BY-SA+4.0+%28Int%29%22+license%3A%22CC-BY-NC+3.0+%28Au%29%22+license%3A%22CC+BY-NC-Int%22+license%3A%22CC-BY-NC-Int%22+license%3A%22Custom%22+license%3A%22CC-BY-NC+2.5+%28Int%29%22+license%3A%22CC-BY-NC-Aus%22+license%3A%22CC-BY-ND+4.0+%28Int%29%22

Hi Miles, see my comment above. Once https://github.com/AtlasOfLivingAustralia/biocache-service/issues/502 is fixed it will work correctly.

M-Nicholls commented 4 years ago

This is working now - there is no error and the correct number of records is returned but the fq terms are not listed in the "User selected filters:" space

e.g. inverse for no license: https://biocache-dq-test.ala.org.au/occurrences/search?q=*%3A*&qualityProfile=re-usable&disableAllQualityFilters=true&fq=-license%3A%5B*+TO+*%5D#tab_mapView

alexhuang091 commented 4 years ago

This is working now - there is no error and the correct number of records is returned but the fq terms are not listed in the "User selected filters:" space

Yes I also saw this sometimes. Maybe a new bug. I'll create a new issue on Biocache-service.

alexhuang091 commented 4 years ago

https://github.com/AtlasOfLivingAustralia/biocache-service/pull/535 is created to fix it.

nickdos commented 4 years ago

@alexhuang091 can you pls QA this - assuming its deployed to biocache-dq-test?

nickdos commented 3 years ago

Done in release 2.6.0.

AtlasOfLivingAustralia / DataQuality

Errors displayed with "Data licensed for all uses" filter profile #130