For data files transferred from EGA to Collab, add EGA repo to 'File copy' section of the portal file index

[junjun-zhang]

Here is one such file: https://dcc.icgc.org/repositories/files/FI743257. It is originated from EGA, we transferred to Collaboratory, but the file page only shows this file exist in Collab but not in EGA.

We need a way to let the portal repo indexer know addition copy of the file exists in EGA as well. This could be as easy as detecting whether dataBundleId starts with EGA, if so, there must be a copy of the file exist in EGA.

We may also need additional EGA specific information for the file copy, such as repoFileId, in this case, we need EGAFxxxxx ID to be populated, so will need a way to pass it to indexer.

[junjun-zhang]

@baminou can you please mirror all of the EGA file transfer git repositories hosted internally under http://142.1.177.124/jt-hub to the public github repo under https://github.com/icgc-dcc/?

There should be 5 or 6 of them.

[junjun-zhang]

The goal is to add a new fileCopy entry for files already transferred from EGA to Collaboratory. We first identify the SONG Analysis in Collab, this can be done by Analysis ID. The ID takes form of EGAZ000000000 or EGZR000000000. Then we will need two fields specific to EGA fileCopy entry: repoDataSetIds and repoFileId. Below we give detail how values for these fields can be found.

The git repos for ega transfer jobs have been mirrored from our internal server to GitHub:

• https://github.com/icgc-dcc/ega-file-transfer-to-collab-1-jtracker
• https://github.com/icgc-dcc/ega-file-transfer-to-collab-2-jtracker
• https://github.com/icgc-dcc/ega-file-transfer-to-collab-3-jtracker
• https://github.com/icgc-dcc/ega-file-transfer-to-collab-4-jtracker
• https://github.com/icgc-dcc/ega-file-transfer-to-collab-5-jtracker

The files contained needed information are:

1. Job JSON file with EGA Dataset ID, pattern for file path/name: ega-file-transfer-to-collab-*-jtracker/blob/master/ega-file-transfer-to-collab.*.jtracker/job_state.completed/job.*/job.*.json. Fields: 'bundle_id' (use as 'repoDataBundleId', same as SONG's Analysis ID), 'ega_dataset_id' (use as 'repoDataSetIds').
2. Uploaded data file to Collab, pattern for file path/name: ega-file-transfer-to-collab-*-jtracker/blob/master/ega-file-transfer-to-collab.*.jtracker/job_state.completed/job.*/task_state.completed/worker.*/task.upload.*/task.upload.{ega_file_id}.json. Use the value {ega_file_id} in the file name as 'repoFileId'

Example files:

• https://github.com/icgc-dcc/ega-file-transfer-to-collab-3-jtracker/blob/master/ega-file-transfer-to-collab.0.6.jtracker/job_state.completed/job.EGAZ00001019451.PBCA-DE.7EP32_tumor.EGAN00001209707/job.EGAZ00001019451.PBCA-DE.7EP32_tumor.EGAN00001209707.json
• https://github.com/icgc-dcc/ega-file-transfer-to-collab-3-jtracker/blob/master/ega-file-transfer-to-collab.0.6.jtracker/job_state.completed/job.EGAZ00001019451.PBCA-DE.7EP32_tumor.EGAN00001209707/task_state.completed/worker.1503517275.5f9ce66a.host.4b92f40d.10.10.0.24/task.upload.EGAF00000585961/task.upload.EGAF00000585961.json

[junjun-zhang]

Just to give two examples here:

• data file transferred to Collab from EGA but indexed for Collab only: https://dcc.icgc.org/api/v1/repository/files/FI743257. Need to add one extra fileCopy entry for EGA
• data file indexed with copied in both Collab and EGA: https://dcc.icgc.org/api/v1/repository/files/FI660. Note the EGA specific fields: repoDataBundleId, repoFileId, repoDataSetIds

[rosibaj]

EGA indexing to be investigated in future; do not work on this until we know more about those specs, OR until we have more EGA data to transfer to collaboratory.