Run archival/retrieval stress tests for LTS

[phwissmann]

Description Devise and implement a (stress) test scenarios to understand the following:

• How many failures occur among N retrievals of the same dataset over time
• How many failures occur among M archivals and retrieval of same datasets
• How does the system behave with parallel writes and reads?

Solution proposals

Repeated Retrieval of single dataset

Procedure:

1. Create a dataset

Setup
number of files	800
size per file	200 MB
dataset size	160 GB
target block size	50 GB

done: pid 11223344

3. Archive once on LTS test share

   • Manual: http://scopem-openem.ethz.ch/prefect-ui/flow-runs/flow-run/cf00689a-473f-44c3-aad0-66d8dc859fc3?tab=Logs

4. Run scheduled retrieval against LTS share|

   • Scheduled every 4h
   • will run from landing zone first, expect to run from tape after 8h+
   • Question: what is the retention time on the landing zone? Schedule needs adaptation

Results

• archival succeeded: 2h48min
• single retrieval (from landing zone, not tape) succeeded: 52min

Repeated Archival

Procedure:

1. Create 30 datasets

Setup
number of files	100
size per file	100 MB
dataset size	10 GB
target block size	50 GB

2. Archive datasets concurrently
   • Concurrency limit on workpool level (4)

Result

• no failures
• concurrency was not set in workpool
• landing zone bucket not cleaned up

Large dataset

Recommendation by Daniele: one 1-2TB dataset to see any issues

[phwissmann]

Additional info regarding landing zone handling etc:

The files of your test-share are written to tape 24 hours after last access_time, every day at 04:20am in the morning. At this point the "T-flag" is not set yet, as the file still has a copy on the landing_zone ( on disk ). The delete script, which removes the copy from the landing_zone runs at 13:45 every noon. After that the "T-flag" should be set. The time schedules for the copy and delete scripts are different for each share, and can also change, so they should not be hard-coded inside your code.

Please note, that the "T-flag" will be gone again, after copying a file back from tape, as it has 2 copies then, one on tape and one on disk ( landing_zone ). The file will be removed with the next Delete-Job ( Cleanup Job ), and the "T-flag" will show up again then.