Create job to download filtered data using Yulia's scripts

[mokolodi1]

See "Outlier analysis" email from July 13th.

[e-t-k]

branch: f-exprAndVarFilters

Requires the following repos:

• patient-care
• job-runner
• external-tools
• primary collections
• (MedBook)

Tasks [not necessarily exhaustive haha]

Phase 1 : creating the sample group filter tool

• [x] external-tools : add filter scripts
• [x] primary collections : add job class for applyExprAndVarFilters
• job runner
  • [x] add external tools paths to my config
  • [x] create job framework for filtering
  • [x] export sample group to rectangular file a la outlier analysis
  • [x] run filters on exported file
  • [x] expression level filter
  • [x] variance filter
  • [x] re-import file as blobs2
• patient care
  • [x] display on sample group page -- job could be run / job running / filtered file exists
  • [x] helper : does filtered file exist
  • [x] helper : is job currently running
  • [x] verbiage on sample group page
  • [x] hook up button to run job to apply filter
  • [x] route to download blobs2 without needing associated job
  • [x] hook up button to download filtered
  • [x] download with per-samplegroup filename
• MedBook
  • [ ] add new external tools paths to MEDBOOK_SETTINGS
  • [ ] update submodule versions

Phase 2 : applying it to outlier analysis

• patient care
• [x] new toggle : use filtered sample group? (Default on)
• [x] if so, create filtered SG first (use prerequisite_job_id to ensure ordering)
• [x] when checking for previously run, make sure to also match whether filter was used or not
• [x] UI - show in previously run outlier analyses whether filter was used or not
• [x] UI - show in outlier analysis results page whether filter was used.
• job runner
  • [x] outlier analysis to check for filtered sample group (and check whether it was requested --how?) and use that instead of genomic expression export

[e-t-k]

Currently: TESTING

• [x] Make new data set with 10+ samples , 100+ features on each
  • [x] apply filters manually to source files using original scripts
• [x] new sample group : apply filters from sample group UI
  • [x] filters applied successfully? filtered data correct?
  • [x] filtered data downloads ok
  • [x] outlier analysis - without filters -- unfiltered data correctly used?
  • [x] same analysis again - does it find it instead of recalculate?
  • [x] same iqr, different test sample : does it use median/up/down files correctly?
  • [x] outlier analysis - WITH filters -- existing filters correctly used?
  • [x] same analysis again - does it find it instead of recalculate?
  • [x] same iqr, different test sample : does it use median/up/down files correctly?

---

• [x] new sample group: don't apply filters in sample group UI
  • [x] outlier analysis - WITH filters - filter successfully created ?
  • [x] same analysis again - does it find it instead of recalculate?
  • [x] same iqr, different test sample : does it use median/up/down files correctly?

---

code review

• [x] ETK read through all changes in this branch
  • [x] patient care changes
  • [x] jobrunner
  • [X] primary-collections
  • [X] external tools
• [x] remove XXX console comments
• [x] ETK take teo through changes
  • [x] demo of feature
  • [x] look at code
• merge into develop (all repos)
  • [x] external tools -> master
  • [x] primary collections -> master
  • [x] patient care -> develop
  • [x] update subrepo versions
  • [x] jobrunner -> develop
  • [x] update subrepo versions
• [x] main medbook -> develop - update subrepo versions
• [x] push to staging
  • [ ] above tests on staging using existing large data set to make sample group

Next step, once its made live:

updating sample groups / data sets on prod to match these changes

Right now, which data sets are 'raw' and unfiltered ? which had the filters 'baked in' before uploading to medbook? I don't know

[mokolodi1]

What do you mean by baked in before uploading to MedBook?

Teo

On Mon, Aug 1, 2016 at 1:06 PM, e-t-k notifications@github.com wrote:

Currently: TESTING

Next step, once its made live:

updating sample groups / data sets on prod to match these changes

Right now, which data sets are 'raw' and unfiltered ? which had the filters 'baked in' before uploading to medbook? I don't know

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/UCSC-MedBook/patient-care/issues/29#issuecomment-236691552, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRIv0BXoYo9iymh_gWjTom3edtQpsuCks5qblGsgaJpZM4JOFhG .

[e-t-k]

What do you mean by baked in

My recollection is that for the current "CKCC reference cohort" data set in the prod db, I was told that these filters had been applied manually before the samples were uploaded to medbook. (the sample group for the filtering would have been the entire data set.) Don't know if any other data sets are known to be like this. Do i recall correctly? since this data set was imported before I got here...

Re-applying filters to a sample group where the filter is already "baked in" - expression level filter would have no effect (since all relevant genes already removed) but variance filter would remove another 20% of genes.

[mokolodi1]

Ah, I see.

The CKCC Reference Cohort is the only one.

On Mon, Aug 1, 2016 at 1:32 PM, e-t-k notifications@github.com wrote:

What do you mean by baked in

My recollection is that for the current "CKCC reference cohort" data set in the prod db, I was told that these filters had been applied manually before the samples were uploaded to medbook. (the sample group for the filtering would have been the entire data set.) Don't know if any other data sets are known to be like this. Do i recall correctly? since this data set was imported before I got here...

Re-applying filters to a sample group where the filter is already "baked in" - expression level filter would have no effect (since all relevant genes already removed) but variance filter would remove another 20% of genes.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/UCSC-MedBook/patient-care/issues/29#issuecomment-236698585, or mute the thread https://github.com/notifications/unsubscribe-auth/ABRIv2r4ZW0xIxW1aHm2N8WkdhrxpxfZks5qblfrgaJpZM4JOFhG .

[e-t-k]

Issues found in testing (dev box):

• [x] sample group name in output file genes_filtered.tsv isn't urlencoded (eg "this/is/a/test")
• [x] minor outdated comments

Issues found in code review with Teo

• [x] reorder unfiltered above filtered samples in SG page
• [x] indent comments x2
• [x] wording "your filters are being applied
• [x] moved heading

Issues found in testing(staging) ... Jobs fail with, roughly, " undefined is not a function at server.js at 105". Suggests that JobClasses is not correctly being populated with my new JobClass. docker problem? looking into this... Update: docker-compose up wasn't building the new code. (This turns out to be the documented behavior when there's an existing image.). fixed.

[e-t-k]

-- When I'm running a job to generate filters, Teo, looking at the same sample group, can't see that that job is running, and the button to generate filters is available to him.

Fix : let the sample group subscribe to all jobs of that type associated with that sample group, regardless of owner.

[e-t-k]

This is live, & sample groups / data sets etc have been updated to new cohort .