Notify Failed error in hca-util sync

[aaclan-ebi]

Steps to replicate:

1. Create submission. Get the s3 location of the submission upload area
2. Sync files from hca-util upload area 57987572-88dd-4d32-a19c-72f70b3be902 to submission upload area

Actual Behaviour: Error message says "Transfer complete with error." Each file's error messages says "Transferred. Notify failed."

Expected Behaviour: There should be no errors. Successful transfer

Notes:

(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ hca-util select 57987572-88dd-4d32-a19c-72f70b3be902
Selected upload area is 57987572-88dd-4d32-a19c-72f70b3be902/
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ 
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ hca-util list
57987572-88dd-4d32-a19c-72f70b3be902/ ux  HumanLiverImmuneCells_GSE125188 
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R1.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R2.fastq.gz
57987572-88dd-4d32-a19c-72f70b3be902/sra_metadata_d43814f9-0258-4a30-b323-011659a626d0.csv
17 items
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ hca-util sync <submission-upload-area> --profile test-util-admin
Transferring...
17 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 663G/663G [05:49<00:00, 1.90GB/s]
16 files failed to transfer: 
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R1.fastq.gz Transferred. Notify failed.
Transfer complete with error.
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ hca-util sync s3://org-hca-data-archive-upload-prod/3f3baa51-6577-41ce-b71c-18cf8917e459/ --profile test-util-admin
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗ hca-util sync s3://org-hca-data-archive-upload-prod/3f3baa51-6577-41ce-b71c-18cf8917e459/ --profile test-util-admin
Transferring...
17 files: 100%|██████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 663G/663G [05:26<00:00, 2.03GB/s]
16 files failed to transfer: 
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448139/S1_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448140/S2_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448141/S3_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448142/S4_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448143/S5_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448145/S7_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R1.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448147/S9_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R2.fastq.gz Transferred. Notify failed.
57987572-88dd-4d32-a19c-72f70b3be902/SRR8448144/S6_R1.fastq.gz Transferred. Notify failed.
Transfer complete with error.
(venv_2) ➜  3125f4a0-dda1-40c6-ab69-d27f795f08e1 git:(prod) ✗  

@ami-day worked around this by uploading copies of files from different hca-util upload area

[amnonkhen]

xc

b