Complete Data Release 2

[rosibaj]

• [x] Snapshot file-centric-1.0
• [x] Re-index all of song in staging
• [x] Get number issue in arranger solved.
• [x] complete QC and suppression of analysis if needed (For Release, QC steps/process prepared here: https://wiki.oicr.on.ca/display/icgcargotech/Data+Relase+QC)
• [x] finagle new index [ file-centric-2.0] --- reindex all of song -- remove MUTECT 2 files from release
• [x] document the release

[blabadi]

blocked by :

https://github.com/overture-stack/arranger/issues/615 and https://github.com/icgc-argo/roadmap/issues/558

[rosibaj]

The index looks good, but we have had a request to remove some documents for release:

Exclude No. 1 - 404 No Data variant timing metrics :

query ($filters: JSON){
  file {
    aggregations (filters:$filters){
      data_type{
        buckets{
          key
          doc_count
        }
      }
    }
  }
}

{
"filters":{"content":[{"content":{"field":"data_category","value":"__missing__"},"op":"in"}],"op":"and"}
}

Exclude No. 2 - Mutect2 Variant Calling Data that is being used for testing:

query ($filters: JSON){
  file {
    aggregations (filters:$filters){
      data_type{
        buckets{
          key
          doc_count
        }
      }
    }
  }
}

{
    "filters": {
        "content": [{
            "content": {
                "field": "analysis.workflow.workflow_name",
                "value": ["GATK Mutect2 Variant Calling"]
            },
            "op": "in"
        }],
        "op": "and"
    }
}

Exclude No. 3 - PACA-CA WXS Donors with only reads, not variant calls

query ($filters: JSON){
  file{
    aggregations (filters:$filters){
        data_type{
        buckets{
          key
          doc_count
        }
      }
    }
  }
}

{
    "filters": {
    "content": [{
        "content": {
            "field": "analysis.experiment.experimental_strategy",
            "value": "WXS"
        },
        "op": "in"
    }, {
        "content": {
            "field": "donors.donor_id",
            "value": ["DO35442","DO35098", "DO35330"
            ]
        },
        "op": "in"
    }],
    "op": "and"
}
}

Exclude No. 4 - 2 PTC-SA Donors with only bad reads that had issues when variant calling was run

query ($filters: JSON){
  file{
    aggregations (filters:$filters){
        data_type{
        buckets{
          key
          doc_count
        }
      }
    }
  }
}

{
    "filters": {"content":[{"content":{"field":"donors.donor_id","value":["DO231311","DO231493"]},"op":"in"}],"op":"and"}
}

[rosibaj]

Closing as this is done! https://docs.icgc-argo.org/docs/release-notes/data-releases#data-release-20