clemente-lab / mmeds-meta

A database for storing and analyzing omics data
https://mmeds.org
2 stars 1 forks source link

Enhancement analysis testing #383

Closed mstapylton closed 2 years ago

mstapylton commented 2 years ago

Pull Request Template for MMEDS

Currently, I'm only testing one analysis, study setup. I know we want to test other demultiplexing options but it'd also be helpful to connect the setup of generating study files, analysis files (like creating the Qiime2_0 directory) with then running the study through github actions. Testing each study will add ~10mins runtime to test so we also might want to split up the github actions job into several that run concurrently as Adam suggested. I think it's best to handle all those things in a separate PR and have this PR focused on the initial changes needed for analysis testing.

What has changed

To automate running analyses, we needed to be able to run them through github actions which involved:

  1. automating installing all required packages, see tests.yml and also changed conda env files, changed module files.
  2. Created an example analysis by uploading test files to MMEDs, then modifying the study, analysis files to be run through github actions
  3. Analyses typically run asynchronously via the watcher that hands off the job to lsf submission. To run the analysis via github actions, there needed to be some modifications to allow the analysis to run synchronously. Mostly this is handled through the new function in utils.py, run_analysis() but also through some changes to the job files. However, the changes to the jobfiles are just to consolidate the qiime2, jupyter, latex commands so that they can be run from mmeds subprocess calls, which is maybe a good thing.

Next steps: Add more analyses, automate creation of a new uploaded study from a "user" upload

Checklist of pre-requisites

codecov[bot] commented 2 years ago

Codecov Report

Merging #383 (da85fc6) into master (c6e2e9c) will decrease coverage by 0.40%. The diff coverage is 4.25%.

@@            Coverage Diff             @@
##           master     #383      +/-   ##
==========================================
- Coverage   51.18%   50.78%   -0.41%     
==========================================
  Files          29       29              
  Lines        4964     4988      +24     
==========================================
- Hits         2541     2533       -8     
- Misses       2423     2455      +32     
Flag Coverage Δ
server 49.65% <4.25%> (-0.41%) :arrow_down:
unit 15.49% <4.25%> (-0.04%) :arrow_down:

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
mmeds/summary.py 0.00% <0.00%> (ø)
mmeds/util.py 43.65% <7.69%> (-0.68%) :arrow_down:
mmeds/config.py 83.79% <100.00%> (-3.41%) :arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update c6e2e9c...da85fc6. Read the comment docs.