Spike: Explore ways to make search (adv doc in particular) more performant

[JessicaMarine]

As a development team, so that search performs well with many users, we need to investigate solutions that allow users to receive results quickly and not get logged out or timed out, and reduce impact to other critical systems that depend on elastic search.

Pre-Conditions:

• Engineers have reviewed the notes below and the previous documentation in https://github.com/ustaxcourt/ef-cms/issues/872
• Review findings from SPIKE #8072

Acceptance Criteria:

(1) Investigate other alternatives -- they can't negatively impact other part of the system;

• Investigate and document the pros and cons (including cost) for solutions that make search more performant
• Proposed solutions must prevent search from timing out or logging users out
• Proposed solutions don't negatively impact critical systems
• Utilize notes (below) in this spike and https://github.com/ustaxcourt/ef-cms/issues/872 when considering solutions
• TBD - include target performance standards for search (How many concurrent searches, etc)

(2) Decide on and document approach; and

• Document potential impacts of proposed solutions on re-index/migration
• Document the outcome of the new solutions, including diagrams that show the usage for each data persistence product (Dynamo, ES, S3)

(3) Make sure stories in backlog are revised to implement that approach. (Jessica to do)

• User stories in the backlog are revised (approx 10) or created based on outcome of spike

  ---

Notes

• During pre-launch load testing, Court found that concurrent searches would spike elastic search CPU. If the load spikes to 100, the rest of the app starts to flail, because so many things are using ES for indexing.
• One core problem is that the lookup on caseCaption and isSealed on the Case when searching upon the Docket Entry index makes ES work too hard (at least for Advanced Document Search).
• Another core problem, (if we abandoned the Parent/Child Join) is when a Case receives an update, we would have to update ALL of the docket entries with the caseCaption or isSealed
  • this possibly could create a backlog of items that need to be processed by the Streams Interactor
  • this also makes it difficult during blue/green migrations. The Docket entries would have to exist in ES in order to receive the update... unless when you index a docket entry, it also looks up the record for the Case
• Perhaps the elephant in the room is the documentContents and where / when / how you index that information into ES. This is information that doesn't change often. Do we need to index this upon every blue/green deploy? Can it be decoupled from the docket entry index? It takes a long time and is expensive.
• Makes elastic search inoperable. Anything that relies upon ES, would take longer or be slowed down or now respond with valid information. This is why we pulled search out of MVP.
• Should we be allowing public functionality to impact internal user functionality?
• Prevent public from impacting internal users
• Internal users should also not be negatively impacted when performing concurrent searches

Tasks

• Collect additional statistics on performance of parent/child with 4 and 5 nodes

Proposed Solution

Step 1) Add a rate limiter in front of all ElasticSearch queries to limit the number of requests that reach the ElasticSearch cluster. The rate and cluster size could be adjusted to demand based on needs of the users. Instead of throwing a generic error when the searches fail, throw a specific error that informs the user that search is currently unavailable. Set replicas to 2 and shards to 4 for the docketEntry index for performance boosts over the current configuration.

Step 2) Gather metrics from production search usage to identify common searches and statistics.

Step 3) Additional performance tuning

• Adding nodes to the cluster: The easiest way to boost search performance
• Splitting clusters
• Removing parent / child join: 3X search performance increase, would be considerable engineering effort
• Query syntax optimizations
• cache results for common queries in different persistence layer (redis, dynamo, other?)

[matthopson]

Services / Features By ES Index

efcms-case-deadline

• Case Deadlines

efcms-case

• Case Advanced Search
• Case Public Search
• Blocked Cases Report
• Case Inventory Report
• Get Case By User ID
• Get First Single Case
• Get Ready For Trial Cases
• Get Case Deadlines (report)

efcms-docket-entry

• Advanced Document Search (USES JOIN ON CASE)
• Pending Items (USES JOIN ON CASE)
• Reconciliation Report

efcms-message

• Messages (user / section, inbox / completed)

efcms-user-case

• Indexed Cases for User (open / closed cases, verify user pending email)

efcms-user

• Practitioner By Name

efcms-work-item

• Document QC (user / section)

[sutt0n]

Specific tasks via https://github.com/flexion/ef-cms/issues/8072#issuecomment-853359853:

• Utilize filter ElasticSearch context (#8326 related)
• Migration for isOrder and isOpinion flags on DocketEntry entity and remove eventCode.S terms filter
• Increase filter index cache percentage (default is 10%)

cc: @codyseibert @kkoskelin @matthopson @mmarcotte

[ericsorenson]

Here's what GitHub does!

[ericsorenson]

Re-indexing Experiment:

Blue/green deploy in ustaxcourt's develop environment including all order and opinion document indexes. Once with 0 replicas, once without modifying replicas.

Success: Takes less than 6 hours, all documents are indexed.

[matthopson]

Roughly documenting needed stories:

Rate Limiter

• Limit the rate at which users can make requests against any endpoints using the efcms-docket-entry index
• Public facing endpoints would have a lower rate than internally executed endpoints

Search Interstitial Messaging (maybe multiple stories)

• As a user, I should see an indication that my search request is “processing” after starting a search
• Maybe differentiate messaging between internal / external (or role) and type of search

Error Handling

• As a user, I should see a more descriptive error when requests being made to any endpoints using the efcms-docket-entry index are impacted by service latency or unavailability without being logged out.
• Maybe differentiate messaging between internal / external (or role) and type of search

Setup a means to gather metrics from prod

• Number of requests and request time (possibly from Kibana?)
• ES cluster health
  • CPU Utilization
  • Search rate
  • Memory pressure

[JessicaMarine]

@matthopson Here are the stories that I queued up for ORDER search. Can you please review them with the Search Party and see if this is covers everything:

7372: Concurrent Order searches

7374: Prevent User from being logged out during Order search

7368: Prevent User from being timed out during Order search

7369: Exact match on Order search

7383: Partial matches on Order search

7756: Rate-limiting [Note: This is 1 user abusing the system not the auto-rate limiting to keep ES healthy]

8423: Gather search metrics from prod

Happy to revise/edit as necessary.