[STORY] Configure USyd AIS k8s node to run "xnat-ingest" upload for TBP

[tclose]

Metadata

Epic: #271 Feature: #273

Description

• The ghcr.io/australian-imaging-service/xnat-ingest:latest image needs to be regularly pulled to the USyd AIS k8s cluster to pick up updates
• Configure the staging command, docker run ghcr.io/australian-imaging-service/xnat-ingest:latest upload, to run regularly:
  • Daily @ 10am
    • XNAT_INGEST_UPLOAD_STAGED=\<s3-bucket>
    • XNAT_INGEST_UPLOAD_HOST=https://xnat.sydney.edu.au
    • XNAT_INGEST_UPLOAD_USER=<role-account-user>
    • XNAT_INGEST_UPLOAD_PASS=<role-account-pass>
    • XNAT_INGEST_UPLOAD_ALWAYSINCLUDE=medimage/dicom-series
    • XNAT_INGEST_UPLOAD_STORE_CREDENTIALS=\<s3-bucket-access-key>,\<s3-bucket-access-secret>
    • XNAT_INGEST_UPLOAD_LOGFILE=\<somewhere-sensible>,INFO
    • XNAT_INGEST_UPLOAD_DELETE=0 (for now, will eventually be 1 when we are satisfied everything is working)
    • XNAT_INGEST_UPLOAD_TEMPDIR=\<somewhere-else-sensible> (temporary scratch space)
    • XNAT_INGEST_UPLOAD_REQUIRE_MANIFEST=1
    • XNAT_INGEST_UPLOAD_CLEANUP_OLDER_THAN=30

Acceptance Criteria

• [ ] 1. Data is pulled in from S3 bucket and uploaded to XNAT in the corresponding project/subject/session/scan/resource structure defined by the "directory" structure on S3

Blocked by

• [x] #278

[fxusyd]

Testing this within the K8s cluster on staging server now.

[fxusyd]

S3 bucket used by xnat-ingest upload is not accessible. Emailed RNSH to add it to whitelist.

[fxusyd]

Similar with #278

1. Fixed pod within K8s cluster couldn't make external connections due to proxy.
2. Waiting on new container image with script to bypass SSL verification due to self-signed certificate is used in RNSH.

[tclose]

Manual upload works, can be automated but waiting on improvements in XNAT-ingest workflow