Searching for something with ( breaks it

odscjames commented 5 years ago

Open Network Traffic Debugger

Open https://standard.open-contracting.org/latest/en/search/?q=json(&check_keywords=yes&area=default#

Watch the request to the server 500

odscjames commented 5 years ago

Standard Search: Searching for something with ( breaks it

kindly commented 5 years ago

We currently use text query in elasticsearch

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

This parses the search term and errors if there are problems with it. A single open ( will cause an issue as elastic-search can not parse this.

We should probably just catch the error and supply feedback to the user as to why the search failed.

odscjames commented 5 years ago

@kindly Can you check that the simple search feature in elastic does actually support multi-term search? Ta

robredpath commented 5 years ago

We should probably just catch the error and supply feedback to the user as to why the search failed.

I'm not sure. If I'm searching docs, I don't think I expect to be able to use any sort of advanced search operators

jpmckinney commented 4 years ago

Copying comment by @robredpath in Trello:

Great! So we think there's a choice to be made here between:

Using a simple search feature in elastic that treats special chars as normal characters that can be searched on, or
Keeping the current featureful search that we use, and exposing UI to allow users to understand the advanced search operators. This UI would require translation, of course.

@kindly is going to check that the simple search feature in elastic does actually support multi-term search

jpmckinney commented 4 years ago

FYI, simple_query_string is fine and has the desired behavior: https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html

jpmckinney commented 4 years ago

Noting that queries with / also break, which are quite common given the character appears in JSON paths.

$ zgrep -Eo 'query_string":{"query":"[^"]+"' /var/log/elasticsearch/*
elasticsearch-2018-08-14-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-14-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-14-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-14-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-14-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/procurementMethod"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/moo"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/moo"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/moo"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/moo"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"tender/moo"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-08-15-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-10-24-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-10-24-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-10-24-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-10-24-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-10-24-1.log.gz:query_string":{"query":"\"
elasticsearch-2018-12-06-1.log.gz:query_string":{"query":"contract/suppliers"
elasticsearch-2018-12-06-1.log.gz:query_string":{"query":"contract/suppliers"
elasticsearch-2018-12-06-1.log.gz:query_string":{"query":"contract/suppliers"
elasticsearch-2018-12-06-1.log.gz:query_string":{"query":"contract/suppliers"
elasticsearch-2018-12-06-1.log.gz:query_string":{"query":"contract/suppliers"
elasticsearch-2018-12-17-1.log.gz:query_string":{"query":"AAAA-MM-DDThh:mm:ssZ"
elasticsearch-2018-12-17-1.log.gz:query_string":{"query":"AAAA-MM-DDThh:mm:ssZ"
elasticsearch-2018-12-17-1.log.gz:query_string":{"query":"AAAA-MM-DDThh:mm:ssZ"
elasticsearch-2018-12-17-1.log.gz:query_string":{"query":"AAAA-MM-DDThh:mm:ssZ"
elasticsearch-2018-12-17-1.log.gz:query_string":{"query":"AAAA-MM-DDThh:mm:ssZ"
elasticsearch-2019-02-06-1.log.gz:query_string":{"query":"application/pdf"
elasticsearch-2019-02-06-1.log.gz:query_string":{"query":"application/pdf"
elasticsearch-2019-02-06-1.log.gz:query_string":{"query":"application/pdf"
elasticsearch-2019-02-06-1.log.gz:query_string":{"query":"application/pdf"
elasticsearch-2019-02-06-1.log.gz:query_string":{"query":"application/pdf"
elasticsearch-2019-02-19-1.log.gz:query_string":{"query":"https://openopps.com/tenders/architectural-services-for-the-construction-of-disc-iii-new-offices-and-laboratories-at-harwell-campus-didcot/ocds-0c46vo-0001-d0f20c14-afc2-4a87-a649-9e4dcdac7ec6?format=json"
elasticsearch-2019-02-19-1.log.gz:query_string":{"query":"https://openopps.com/tenders/architectural-services-for-the-construction-of-disc-iii-new-offices-and-laboratories-at-harwell-campus-didcot/ocds-0c46vo-0001-d0f20c14-afc2-4a87-a649-9e4dcdac7ec6?format=json"
elasticsearch-2019-02-19-1.log.gz:query_string":{"query":"https://openopps.com/tenders/architectural-services-for-the-construction-of-disc-iii-new-offices-and-laboratories-at-harwell-campus-didcot/ocds-0c46vo-0001-d0f20c14-afc2-4a87-a649-9e4dcdac7ec6?format=json"
elasticsearch-2019-02-19-1.log.gz:query_string":{"query":"https://openopps.com/tenders/architectural-services-for-the-construction-of-disc-iii-new-offices-and-laboratories-at-harwell-campus-didcot/ocds-0c46vo-0001-d0f20c14-afc2-4a87-a649-9e4dcdac7ec6?format=json"
elasticsearch-2019-02-19-1.log.gz:query_string":{"query":"https://openopps.com/tenders/architectural-services-for-the-construction-of-disc-iii-new-offices-and-laboratories-at-harwell-campus-didcot/ocds-0c46vo-0001-d0f20c14-afc2-4a87-a649-9e4dcdac7ec6?format=json"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-02-1.log.gz:query_string":{"query":"tender/item"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TEST L)IVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TEST L)IVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TEST L)IVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TEST L)IVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TEST L)IVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TES (TIVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TES (TIVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TES (TIVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TES (TIVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"TES (TIVE"
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-08-07-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-09-05-1.log.gz:query_string":{"query":"json("
elasticsearch-2019-10-30-1.log.gz:query_string":{"query":"validator}"
elasticsearch-2019-10-30-1.log.gz:query_string":{"query":"validator}"
elasticsearch-2019-10-30-1.log.gz:query_string":{"query":"validator}"
elasticsearch-2019-10-30-1.log.gz:query_string":{"query":"validator}"
elasticsearch-2019-10-30-1.log.gz:query_string":{"query":"validator}"
elasticsearch-2019-11-15-1.log.gz:query_string":{"query":"UN/CEFACT"
elasticsearch-2019-11-15-1.log.gz:query_string":{"query":"UN/CEFACT"
elasticsearch-2019-11-15-1.log.gz:query_string":{"query":"UN/CEFACT"
elasticsearch-2019-11-15-1.log.gz:query_string":{"query":"UN/CEFACT"
elasticsearch-2019-11-15-1.log.gz:query_string":{"query":"UN/CEFACT"
elasticsearch-2019-11-29-1.log.gz:query_string":{"query":"UN/LOCODE"
elasticsearch-2019-11-29-1.log.gz:query_string":{"query":"UN/LOCODE"
elasticsearch-2019-11-29-1.log.gz:query_string":{"query":"UN/LOCODE"
elasticsearch-2019-11-29-1.log.gz:query_string":{"query":"UN/LOCODE"
elasticsearch-2019-11-29-1.log.gz:query_string":{"query":"UN/LOCODE"
elasticsearch-2019-12-17-1.log.gz:query_string":{"query":"tender/mainProcurementCategory"
elasticsearch-2019-12-17-1.log.gz:query_string":{"query":"tender/mainProcurementCategory"
elasticsearch-2019-12-17-1.log.gz:query_string":{"query":"tender/mainProcurementCategory"
elasticsearch-2019-12-17-1.log.gz:query_string":{"query":"tender/mainProcurementCategory"
elasticsearch-2019-12-17-1.log.gz:query_string":{"query":"tender/mainProcurementCategory"
elasticsearch-2019-12-27-1.log.gz:query_string":{"query":"[SCHEMA VERSION"
elasticsearch-2019-12-27-1.log.gz:query_string":{"query":"[SCHEMA VERSION"
elasticsearch-2019-12-27-1.log.gz:query_string":{"query":"[SCHEMA VERSION"
elasticsearch-2019-12-27-1.log.gz:query_string":{"query":"[SCHEMA VERSION"
elasticsearch-2019-12-27-1.log.gz:query_string":{"query":"[SCHEMA VERSION"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-02-28-1.log.gz:query_string":{"query":"contratacion cerrada}"
elasticsearch-2020-03-09-1.log.gz:query_string":{"query":"/VZRequest/VZ/CisloEvidencni"
elasticsearch-2020-03-09-1.log.gz:query_string":{"query":"/VZRequest/VZ/CisloEvidencni"
elasticsearch-2020-03-09-1.log.gz:query_string":{"query":"/VZRequest/VZ/CisloEvidencni"
elasticsearch-2020-03-09-1.log.gz:query_string":{"query":"/VZRequest/VZ/CisloEvidencni"
elasticsearch-2020-03-09-1.log.gz:query_string":{"query":"/VZRequest/VZ/CisloEvidencni"
elasticsearch-2020-04-08-1.log.gz:query_string":{"query":"procurmentOfficer\"
elasticsearch-2020-04-08-1.log.gz:query_string":{"query":"procurmentOfficer\"
elasticsearch-2020-04-08-1.log.gz:query_string":{"query":"procurmentOfficer\"
elasticsearch-2020-04-08-1.log.gz:query_string":{"query":"procurmentOfficer\"
elasticsearch-2020-04-08-1.log.gz:query_string":{"query":"procurmentOfficer\"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-06-22-1.log.gz:query_string":{"query":"contracts/items/classification/scheme"
elasticsearch-2020-07-02-1.log.gz:query_string":{"query":"OBJECT_DESCR/JUSTIFICATION"
elasticsearch-2020-07-02-1.log.gz:query_string":{"query":"OBJECT_DESCR/JUSTIFICATION"
elasticsearch-2020-07-02-1.log.gz:query_string":{"query":"OBJECT_DESCR/JUSTIFICATION"
elasticsearch-2020-07-02-1.log.gz:query_string":{"query":"OBJECT_DESCR/JUSTIFICATION"
elasticsearch-2020-07-02-1.log.gz:query_string":{"query":"OBJECT_DESCR/JUSTIFICATION"
elasticsearch-2020-07-23-1.log.gz:query_string":{"query":"use cases\\"
elasticsearch-2020-07-23-1.log.gz:query_string":{"query":"use cases\\"
elasticsearch-2020-07-23-1.log.gz:query_string":{"query":"use cases\\"
elasticsearch-2020-07-23-1.log.gz:query_string":{"query":"use cases\\"
elasticsearch-2020-07-23-1.log.gz:query_string":{"query":"use cases\\"
elasticsearch-2020-08-03-1.log.gz:query_string":{"query":"http://www.perucontrata.com.pe/"
elasticsearch-2020-08-03-1.log.gz:query_string":{"query":"http://www.perucontrata.com.pe/"
elasticsearch-2020-08-03-1.log.gz:query_string":{"query":"http://www.perucontrata.com.pe/"
elasticsearch-2020-08-03-1.log.gz:query_string":{"query":"http://www.perucontrata.com.pe/"
elasticsearch-2020-08-03-1.log.gz:query_string":{"query":"http://www.perucontrata.com.pe/"
elasticsearch-2020-08-10-1.log.gz:query_string":{"query":"tender/additionalProcurementCategories"
elasticsearch-2020-08-10-1.log.gz:query_string":{"query":"tender/additionalProcurementCategories"
elasticsearch-2020-08-10-1.log.gz:query_string":{"query":"tender/additionalProcurementCategories"
elasticsearch-2020-08-10-1.log.gz:query_string":{"query":"tender/additionalProcurementCategories"
elasticsearch-2020-08-10-1.log.gz:query_string":{"query":"tender/additionalProcurementCategories"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-05-1.log.gz:query_string":{"query":"tender/awardCriteria"
elasticsearch-2020-10-20-1.log.gz:query_string":{"query":"Buyer / Vendor Unique Hash"
elasticsearch-2020-10-20-1.log.gz:query_string":{"query":"Buyer / Vendor Unique Hash"
elasticsearch-2020-10-20-1.log.gz:query_string":{"query":"Buyer / Vendor Unique Hash"
elasticsearch-2020-10-20-1.log.gz:query_string":{"query":"Buyer / Vendor Unique Hash"
elasticsearch-2020-10-20-1.log.gz:query_string":{"query":"Buyer / Vendor Unique Hash"

OpenDataServices / standard-search

Searching for something with ( breaks it #12