QID creation process needs error checking to avoid sending invalid queries to biocache-service

ansell commented 4 years ago

Describe the bug When QID creation fails (as it does), spatial-hub/spatial-service need to be checking the result and not sending invalid queries to biocache-service afterwards.

To Reproduce Emulate/mock a failed QID response from biocache-service and check that everything fails gracefully without sending any queries out to biocache-service.

Expected behavior Biocache-service should not have to handle invalid syntax queries from our own spatial-hub/spatial-service product.

Actual behavior

2020-08-21 10:54:28,553 [http-nio-127.0.0.1-8080-exec-7281] ERROR org.apache.solr.client.solrj.impl.CloudSolrClient  (CloudSolrClient.java:1167) - Request to collection biocache failed due to (400) org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error from server at http://aws-sc3b.ala:8983/solr/biocache: undefined field qid, retry? 0
2020-08-21 10:54:28,554 [http-nio-127.0.0.1-8080-exec-7281] ERROR au.org.ala.biocache.dao.SearchDAOImpl  (SearchDAOImpl.java:4063) - query failed: q=qid:<\!doctype+html><html+lang%3D"en"><head><title>HTTP+Status+400+–+Bad+Request</title><style+type%3D"text/css">body+{font-family:Tahoma,Arial,sans\-serif&fl=occurrence_date,country,geospatial_kosher,year,point-0.01,data_resource,taxonomic_kosher,lat_long,record_number,species_guid,state_conservation,occurrence_year,genus_guid,taxon_concept_lsid,occurrence_remarks,point-0.1,rank,data_provider,user_assertions,id,state,assertions,common_name,raw_common_name,order,longitude,basis_of_record,collection_uid,image_url,location_remarks,point-1,sensitive,point-0.0001,identification_verification_status,country_code,license,collectors,month,genus,occurrence_id,ibra,lft,species_group,subspecies,occurrence_details,collection_name,collection_code,rank_id,raw_taxon_name,occurrence_date_end_dt,data_resource_uid,data_hub,latitude,data_provider_uid,type_status,institution_name,collector,imcra,institution_uid,subspecies_guid,rights,taxon_name,class,rgt,grid_ref,catalogue_number,aust_conservation,data_hub_uid,names_and_lsid,point-0.001,photographer_s,coordinate_uncertainty,all_image_url,kingdom,institution_code,raw_type_status,places,multimedia,phylum,species,raw_basis_of_record,family&rows=0&fq=species_group:Bacteria&facet.missing=true&start=0&sort=score+asc&_stateVer_=biocache-2020-08-20-06-34:98 : Error from server at http://aws-sc3b.ala:8983/solr/biocache: undefined field qid

==> /var/log/nginx/access.log <==
54.206.97.92 - - [21/Aug/2020:10:54:28 +1000] "GET /ws/webportal/wms/reflect?&service=WMS&version=1.1.0&request=GetMap&format=image%2Fpng&layers=ALA%3Aoccurrences&transparent=true&opacity=0.6&CQL_FILTER=qid%3A%3C%21doctype+html%3E%3Chtml+lang%3D%22en%22%3E%3Chead%3E%3Ctitle%3EHTTP+Status+400+%E2%80%93+Bad+Request%3C%2Ftitle%3E%3Cstyle+type%3D%22text%2Fcss%22%3Ebody+%7Bfont-family%3ATahoma%2CArial%2Csans-serif%3B%7D+h1%2C+h2%2C+h3%2C+b+%7Bcolor%3Awhite%3Bbackground-color%3A%23525D76%3B%7D+h1+%7Bfont-size%3A22px%3B%7D+h2+%7Bfont-size%3A16px%3B%7D+h3+%7Bfont-size%3A14px%3B%7D+p+%7Bfont-size%3A12px%3B%7D+a+%7Bcolor%3Ablack%3B%7D+.line+%7Bheight%3A1px%3Bbackground-color%3A%23525D76%3Bborder%3Anone%3B%7D%3C%2Fstyle%3E%3C%2Fhead%3E%3Cbody%3E%3Ch1%3EHTTP+Status+400+%E2%80%93+Bad+Request%3C%2Fh1%3E%3Chr+class%3D%22line%22+%2F%3E%3Cp%3E%3Cb%3EType%3C%2Fb%3E+Status+Report%3C%2Fp%3E%3Cp%3E%3Cb%3EMessage%3C%2Fb%3E+Unable+to+generate+QID+for+query%3C%2Fp%3E%3Cp%3E%3Cb%3EDescription%3C%2Fb%3E+The+server+cannot+or+will+not+process+the+request+due+to+something+that+is+perceived+to+be+a+client+error+%28e.g.%2C+malformed+request+syntax%2C+invalid+request+message+framing%2C+or+deceptive+request+routing%29.%3C%2Fp%3E%3Chr+class%3D%22line%22+%2F%3E%3Ch3%3EApache+Tomcat%2F9.0.31+%28Ubuntu%29%3C%2Fh3%3E%3C%2Fbody%3E%3C%2Fhtml%3E&fq=species_group%3ABacteria&ENV=color%253Aff%253Bname%253Acircle%253Bsize%253A9%253Bopacity%253A1&SRS=EPSG%3A3857&DPI=200&BBOX=1.5944684424791636E7%2C-2527609.959887326%2C1.6046693038281048E7%2C-2425601.334236327&WIDTH=256&HEIGHT=256&TRANSPARENT=true HTTP/1.1" 400 135 "-" "Java/1.8.0_265" "54.206.97.92" request_time=0.257 upstream_response_time=0.008 upstream_connect_time=0.000 upstream_header_time=0.008 upstream_cache_status=-

Additional context The previous issue was closed, but it was attached to a specific release, so not reopening it. See its history in #259

qifeng-bai commented 3 years ago

https://github.com/AtlasOfLivingAustralia/spatial-hub/pull/358

adam-collins commented 3 years ago

Test failed.

qifeng-bai commented 3 years ago

Another pull request created by @adam-collins https://github.com/AtlasOfLivingAustralia/spatial-hub/pull/369

nickdos commented 3 years ago

Can you code review that PR @qifeng-bai? This issue is assigned to me but I'm not sure what I'm expected to do - maybe you should take it back?

qifeng-bai commented 3 years ago

@nickdos I have done. The previous change was made by me, so I asked you to review. And Adam made the last change, I have reviewed and approved

AtlasOfLivingAustralia / spatial-hub

QID creation process needs error checking to avoid sending invalid queries to biocache-service #340