search

icgc-argo / ega-file-transfer

0 stars 0 forks source link

Transfer PACA-AU WXS from EGA to Collab #6

Closed hknahal closed 3 years ago

hknahal commented 3 years ago

EGA has staged whole exome data for 46 donors: https://github.com/icgc-argo/ega-file-transfer/blob/master/ega_operation/20210114/PACA-AU/to_stage_analysis.tsv. Will need to generate jobs and kick off download. Related to https://github.com/icgc-argo/ega-file-transfer/issues/5

hknahal commented 3 years ago

Began EGA-> Collab transfer of WXS data for 46 PACA-AU donors (90 jobs)

Details: Metadata from ICGC and EGA for WXS PACA-AU donors

Files to transfer from EGA -> Collab: https://github.com/icgc-argo/ega-file-transfer/blob/master/ega_operation/20210114/PACA-AU/to_stage_analysis.tsv

VM: 10.1.0.6 Queue ID: 3097917c-82e7-4d63-9829-de703d0f808b

hknahal commented 3 years ago

Jan 25: Split the jobs over 2 VMs to speed up transfer

VM: 10.1.0.6 (60 jobs) Queue ID: 3097917c-82e7-4d63-9829-de703d0f808b

VM: 10.0.0.13 (30 jobs) Queue ID: 5f392e52-5381-490e-af46-bf5ba2fdeb80

hknahal commented 3 years ago

EGA transfer complete on both VMs. 90 jobs total: 59 completed 1 failed on generate_bai task. Details on VM 10.1.0.6 at jthome/account.af1309f7-6dd0-4dce-95d7-7e66f989c1a3/node/workflow.6c0b551e-153c-4a18-a144-cd35bdafba7e/0.8.47/queue.3097917c-82e7-4d63-9829-de703d0f808b/executor.b66bbcf3-e8be-4224-bd28-46f8f8fecdeb/job.80b76748-64f4-427b-a8a7-c8ec76da5ec7/task.generate_bai