SPIKE: Elastic Search Special Character Usage

As a DAWSON project member, I need to understand the relationship between 'special characters' (. - , ' ; : () * & ) and how Elastic Search parses these characters in a query and a content context so that I can effectively design and code search.

Timebox: 4 hours

Some reference to help:

https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-simple-query-string-query.html#supported-flags

Pre-Conditions

Acceptance Criteria

Team has an understanding of how ES interprets special characters as inputs into search query Team has an understanding of how ES interprets special characters in content index Team has an understanding of how to manipulate and work with special characters in ES coding/configuration. Document what we learn - start in story then markdown file in documents directory

Mobile Design/Considerations

IRS API Considerations

Do these changes impact the IRS API?

Security Considerations

[ ] Does this work make you nervous about privacy or security?
[ ] Does this work make major changes to the system?
[ ] Does this work implement new authentication or security controls?
[ ] Does this work create new methods of authentication, modify existing security controls, or explicitly implement any security or privacy features?

Notes

Tasks

Definition of Done (Updated 4-14-21)

Product Owner

[ ] Acceptance criteria have been met and validated on the Court's migration environment

[ ] Business test scenarios to meet all acceptance criteria have been written
[ ] Usability has been validated
[ ] Wiki has been updated (if applicable)
[ ] Story has been tested on a mobile device (for external users only)
[ ] Add scenario to testing document, if applicable (https://docs.google.com/spreadsheets/d/1FUHKC_YrT-PosaWD5gRVmsDzI1HS_U-8CyMIb-qX9EA/edit?usp=sharing)

Engineering

[ ] Automated test scripts have been written
[ ] Field level and page level validation errors (front-end and server-side) integrated and functioning
[ ] Verify that language for docket record for internal users and external users is identical
[ ] New screens have been added to pa11y scripts
[ ] All new functionality verified to work with keyboard and macOS voiceover https://www.apple.com/voiceover/info/guide/_1124.html
[ ] READMEs, other appropriate docs, JSDocs and swagger/APIs fully updated
[ ] UI should be touch optimized and responsive for external only (functions on supported mobile devices and optimized for screen sizes as required)
[ ] Interactors should validate entities before calling persistence methods
[ ] Code refactored for clarity and to remove any known technical debt
[ ] Acceptance criteria for the story has been met
[ ] Deployed to any Flexion environment for UX Review
[ ] Deployed to the Court's migration environment

Using the whitespace tokenizer in a custom analyzer would return results for search keywords containing special characters, as opposed to ignore special characters (like it is currently built to do).
When searching a field that is NOT analyzed, reserved characters can be escaped with a backslash, or if an exact match on the keyword is required, the keyword can be wrapped in double-quotes. (For more information on when analyzers are used vs not, this link is useful) Example:

In this case, the documentContents field is analyzed using a custom analyzer, whereas the docketNumberWithSuffix field is not.

Depending on the business need, we may need a way to use certain reserved characters as they are interpreted by ES (eg -,!). If it is something the court wants/needs, there are tools to do so (eg. specify in query or analyzer)

Reserved chars in ES simple query string:

flexion / ef-cms