Add Clinical Data to File Repository

[rosibaj]

For the set of Dono

• A Data release is composed of a set of data that has been verified to be releasable; this includes both Clinical and Genomic.
• Need a snapshot of data files and clinical data at the point of Data Release.
• From the snapshot, we need to generate the clinical export. Read from the snapshot and generate it into the clinical export TSVS. Assemble based on a Donor selection from the UI.

Tasks:

• [ ] Define the index & snapshot structure --- Needs to include clinical data. --- Needs to include file-tags to be added.

• [ ] Complete the swap over of the File-Service application for maestro.
  -- make sure file data is being correctly indexed. -- all files are currently in state RELEASED

• [ ] Extend Files-Service to reach out to clinical and include clinical data into the new index. --- program-x-re_1 index times n programs as a file repository index. --- on a daily bases, the ETL manager will reindex ------ files form the gateway that are NOT released for donors that are NOT YET EVER released ------ get the clinical data for those donors ------ build program-x-unreleased_1 --- Each program has a released index, and an unreleased index . The unleased index is

• [ ] To download clinical data, we need to do that as gateway-api code. It will be a zipped file export. This is likely supported by another application. Gateway could fetch and compile the TSVs. How does it know the structure of the TSVs from the zip file? <-- to be answered.

• [ ] Storing the data does not necessarily need to be the search index format; it can be a snapshot with the data dictionary version

The Outcome of this Epic is to

• [ ] Be able to download all released clinical data for a project from the File Repository
• [ ] The format of the files can mimic the format used to submit data to clinical.
• [ ] Be able to search by some clinical fields (like primary site) in the facet panel, display that in the table and create a new graph for primary site.

[akachru-github]

For adding clinical data being introduced into the entity page:

• Update the file centric mapping to include the Donor section. Both mapping for the index and the arranger mapping.
• File Manager
  • For individual file that is being updated we need to be able to fetch the data from clinical
  • batch files, we will need to batch them together, updating donor info for a group of files.
  • When clinical information changes / updated, communicate changes to file manager (requires design)
  • Need to confirm what clinical changes are possible. Updates are possible.
  • Can we update public indices?