Test High Availability EZID Stage servers

[jsjiang]

Test High Availability EZID Stage servers:

• [x] Both servers uc3-ezidui-stg01.cdlib.org and uc3-ezidui-stg02.cdlib.org are accessible
• [x] https://ezid-stg.cdlib.org/ is going to the new servers
• [x] Load is distributed on both servers
• [x] No mySQL database related issues (no dead locks, no performance issue)
• [x] S3/Batch download tested
• [x] Out going emails tested (have a consistent from and reply to email address)
• [x] Passed functional tests (verify_ezid_after_patching.py)
• [x] Passed performance tests (using Locust)

[jsjiang]

Load test results:

1. mint with 10 or 15 users on single server vs
2. mint with 5, 10, 15, 20 users against 2 servers behind a load balancer (date: 20230327)

[jsjiang]

Load tests results (3/27/2024):

• total request per second: increased from 9 to 16
• average response time: decreased a little bit
• graphs show a short period performance decrease before starting to gain higher performance

[jsjiang]

More tests:

• mint with 5, 10, 15 users against 2 servers behind a load balancer (date: 20230328)

[jsjiang]

Load tests results (date 3/28/2024):

• total request per second: 15-16 (no changes compared to tests on 3/27)
• average response time: no noticeable changes
• no short period performance decrease

[jsjiang]

Sender's email address in email notifications (Batch download, link checker report):

• current: sent from ezid@uc3-ezidui02x2-prd.cdlib.org, ezid@uc3-ezidui02x2-stg.cdlib.org

Subject: Your EZID batch download link
From: ezid@uc3-ezidui02x2-prd.cdlib.org <ezid@uc3-ezidui02x2-prd.cdlib.org>
To: sid.scd@adm.univ-lyon1.fr <sid.scd@adm.univ-lyon1.fr>

Thank you for using EZID to easily create and manage your identifiers. The batch download you requested is available at:

https://ezid.cdlib.org/s3_download/xa6p4FUUYuvTGedn.zip

The download will be deleted in 1 week.

Best,
EZID Team

This is an automated email. Please do not reply.

• new/with load balancer: sent from ezid@uc3-ezidui-stg01.cdlib.org or ezid@uc3-ezidui-stg02.cdlib.org

  
  Subject: Your EZID batch download link
  From: ezid@uc3-ezidui-stg01.cdlib.org <ezid@uc3-ezidui-stg01.cdlib.org>
  To: jing.jiang@ucop.edu

Thank you for using EZID to easily create and manage your identifiers. The batch download you requested is available at:

https://ezid-stg.cdlib.org/s3_download/CCVXwtDxnIVydv3s.csv.gz

The download will be deleted in 1 week.

Best, EZID Team

This is an automated email. Please do not reply.

[jsjiang]

OpenSearch index filebeat-uc3-ezid-stg* (source file: /ezid/var/log/ezid/ecs_json.log) distribute

• date range: Last 7 days March 25- April (mostly 3/28, 3/29 produced by Locust load tests)
• uc3-ezidui-stg01: 65896
  • stg01 AND 99999 AND createIdentifier: 1801
  • stg01 AND 99999 AND error: 29 (error code 503: service not available)
• uc3-ezidui-stg02: 66,862
  • stg02 AND 99999 AND createIdentifier: 1835
  • stg02 AND 99999 AND error: 0