Deploy OpenSearch in EZID production environment

[jsjiang]

Deploy OpenSearch in EZID production environment.

Check list:

• [x] Verify that all OpenSearch queries are sanitized by the OpenSearch client library/configuration
• [x] Check that there are sufficient OpenSearch resources (enough storage capacity for index on EZID PRD)
• [x] Create the EZID Prd index in OpenSearch
• [x] Define/Understand EZID search/OpenSearch SLA
• [x] Develop backup plan when OpenSearch becomes unavailable
  • [x] Confer with Ashley about our ability to recover from or otherwise handle a loss in search functionality when using OpenSearch
• [x] Implement OpenSearch health checks
• [x] Setup Nagios alerts
• [x] Investigate whether we have comparable metrics in Librato for OpenSearch
• [x] Search functionality tests
• [x] UI Tests - search + reports + download + basic functional tests
• [x] Test dual DB-index write failure

Production deployment schedule:

• Create prod index the week of June 17 or 24
• Prd deployment: 1st or 2nd week of July

Production deployment plan:

• create the initial EZID prd index on OpenSearch (1st index)
• the day before prod deployment, update the index (2nd index: 1st index + updates)
• deploy the code
• update the index (3rd index: 2nd + updates)

Note: the proc-search-indexer background job will take care of the on going updating the OpenSearch index.

[adambuttrick]

OpenSearch index recovery described in https://github.com/CDLUC3/ezid-docs-internal/blob/main/docs/ezid_search_opensearch.md.

[jsjiang]

Production deployment plan

• date/time: 8/29, 5am

• procedure

• [x] create the initial EZID prd index on OpenSearch (1st index)

• [x] the day before prod deployment, update the index (2nd index: 1st index + updates)

• [x] verify SSM paramentes for EZID Prd

• [x] verify resources on EC2 and RDS

• [x] deploy the code (tag: v3.2.19)

• [x] update the index (3rd index: 2nd + updates) (this will do just the past couple of days: python -u manage.py opensearch-update --updated_since=2024-08-27T00:00:00Z

• [x] Stop the ezid service (not the background jobs) on both ezid-prd01 and ezid-prd02

• [x] run the get-rds-opensearch-counts.py script to get the identifier counts from RDS and OpenSearch

• [x] restart the ezid service on both ezid-prd01 and ezid-prd02

• [x] run the ezid-ops-scripts.git/ezidverify_ezid_after_patching.py script on the ezid instances to verify services

• [x] run the ezid-ops-scripts.git/ezid_ui_tests.py script from your laptop to test UI

• [x] testing keyword search

• [x] testing search after login

• [x] testing download all report

• [x] testing Download report in CSV format

[jsjiang]

Getting record counts:

ezid@uc3-ezidui-prd02:05:42:18:~/ezid$ python manage.py get-rds-opensearch-counts
Getting OpenSearch identifier count ...
OpenSearch identifier count: 34763431 (Execution time: 0.70 seconds)
Getting RDS Search Identifier count ...
Search Identifier count: 34760687 (Execution time: 26.43 seconds)
Getting RDS Identifier count ...
Total identifiers: 34761537 (Execution time: 21.52 seconds)
Identifiers without anonymous owned: 34760698 (Execution time: 25.07 seconds)
Anonymous owned identifiers: 839
WARN: OpenSearch and Search Identifier counts did not match.
INFO: It is recommended to run this script while the EZID service is stopped.

[jsjiang]

Note: Tested the Download report in CSV format using the eschol_harvester account. The downloaded report does not match what shows on the UI. This is not OpenSearch related. Create a ticket to investigate.

Ticket created: #718

[adambuttrick]

Released with https://github.com/CDLUC3/ezid/releases/tag/v3.2.19.