Refactor data analysis and pipeline to workflow

henrikstranneheim commented 5 months ago

Description

Added

-

Changed

Refactor data analysis and pipeline to workflow

Fixed

-

How to prepare for test

[ ] ssh to hasta (depending on type of change)
[ ] activate stage: us
[ ] request trailblazer-stage on hasta: paxa

[ ] install on stage:

bash /home/proj/production/servers/resources/hasta.scilifelab.se/update-tool-stage.sh -e S_trailblazer -t trailblazer -b refactor-data_analysis-and-pipeline-to-workflow -a

[ ] ssh to clinical-db (depending on type of change)
[ ] install on stage: bash /home/proj/production/servers/resources/clinical-db.scilifelab.se/update-trailblazer-ui-stage.sh [THIS-BRANCH-NAME]

How to test

[ ] login to ...
[ ] do ...

Expected test outcome

[ ] check that ...
[ ] Take a screenshot and attach or copy/paste the output.

Review

[ ] tests executed by
[ ] "Merge and deploy" approved by Thanks for filling in who performed the code review and the test!

This version is a

[x] MAJOR - when you make incompatible API changes
[ ] MINOR - when you add functionality in a backwards compatible manner
[ ] PATCH - when you make backwards compatible bug fixes or documentation/instructions

Linked to: https://github.com/Clinical-Genomics/cg/pull/2932

henrikstranneheim commented 4 months ago

cg workflow mip-dna start justhusky 👍

cg workflow mip-dna start justhusky
[INFO] 2024/02/13 08:21:57 MIP_ANALYSE - Wrote: /home/proj/stage/rare-disease/cases/justhusky/analysis/justhusky_deliverables.yaml
Action running set for case justhusky
mip-dna run started!

Feb 13 08:21:15 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:15 +0100] [6] [DEBUG] POST /api/v1/get-latest-analysis
Feb 13 08:21:15 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:15 +0100] "POST /api/v1/get-latest-analysis HTTP/1.0" 200 564 "-" "python-requests/2.31.0"
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:57 +0100] [5] [DEBUG] POST /api/v1/get-latest-analysis
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:57 +0100] "POST /api/v1/get-latest-analysis HTTP/1.0" 200 564 "-" "python-requests/2.31.0"
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:21:57 +0100] [5] [DEBUG] POST /api/v1/add-pending-analysis
Feb 13 08:21:57 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:21:57 +0100] "POST /api/v1/add-pending-analysis HTTP/1.0" 201 719 "-" "python-requests/2.31.0"
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:22:07 +0100] [5] [DEBUG] OPTIONS /api/v1/analyses
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:22:07 +0100] "OPTIONS /api/v1/analyses?pipeline=&pageSize=50&page=1 HTTP/1.0" 204 0 "https://cigrid-stage.scilifelab.se/" "Mozil>
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: [2024-02-13 08:22:07 +0100] [6] [DEBUG] GET /api/v1/analyses
Feb 13 08:22:07 cg-vm1.scilifelab.se podman[2678621]: 10.0.2.100 - - [13/Feb/2024:08:22:07 +0100] "GET /api/v1/analyses?pipeline=&pageSize=50&page=1 HTTP/1.0" 200 28135 "https://cigrid-stage.scilifelab.se/" "Mozil>
lines 1-3

Trailblazer scan 👍

θ63° [henrik.stranneheim@hasta:~] [S_base] 130 $ trailblazer scan
Error updating analysis for: case - grandaphid : [Errno 13] Permission denied: '/home/proj/production/cancer/cases/grandaphid/analysis/slurm_jobids.yaml'
Error updating analysis for: case - goodbengal : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/goodbengal/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - helpedlab : [Errno 2] No such file or directory: '/home/proj/production/nipt/cases/helpedlab/output/sacct/submitted_jobs.yaml'
Error updating analysis for: case - newgopher : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9662_slurm_ids.yaml'
Error updating analysis for: case - onestag : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9966_slurm_ids.yaml'
Error updating analysis for: case - funnyaphid : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/funnyaphid/results/trailblazer_config.yaml'
Error updating analysis for: case - proudfish : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9337_slurm_ids.yaml'
Error updating analysis for: case - setcolt : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC9963_slurm_ids.yaml'
Error updating analysis for: case - handyshrimp : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/handyshrimp/results/trailblazer_config.yaml'
Error updating analysis for: case - divinetomcat : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC5119_slurm_ids.yaml'
Error updating analysis for: case - HW2V2DRX2 : [Errno 2] No such file or directory: '/home/proj/production/flowcells/novaseq/runs/230504_A00689_0805_AHW2V2DRX2/slurm_job_ids.yaml'
Error updating analysis for: case - savingstag : [Errno 2] No such file or directory: '/home/proj/production/mutant/cases/savingstag/results/trailblazer_config.yaml'
Error updating analysis for: case - topewe : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC12566_slurm_ids.yaml'
Error updating analysis for: case - fullgoat : [Errno 13] Permission denied: '/home/proj/production/microbial/results/reports/trailblazer/ACC12567_slurm_ids.yaml'
Status in Tower: coolkodiak - 282044 - WPjrjs23yxQwq
Error logging case - coolkodiak :  JSONDecodeError
Updated status correctplatypus - 287509: pending
Error updating analysis for: case - 22FYCNLT3 : [Errno 2] No such file or directory: '/home/proj/production/flow_cells/20231222_LH00188_0053_A22FYCNLT3/slurm_job_ids.yaml'
Updated status hotglider - 288340: failed
Updated status refinedboar - 288356: completed
Updated status gladbuzzard - 288357: completed
Updated status definitemoray - 288359: completed
Updated status absoluteglowworm - 288360: completed
Error updating analysis for: case - lastinghoneybee : [Errno 2] No such file or directory: '/home/proj/production/rare-disease/cases/lastinghoneybee/analysis/slurm_job_ids.yaml'
Updated status choicepuma - 288389: failed
Error updating analysis for: case - HYVYMCCXY : [Errno 2] No such file or directory: '/home/proj/production/flow_cells/190307_ST-E00266_0341_AHYVYMCCXY/slurm_job_ids.yaml'
Updated status setamoeba - 288397: cancelled
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HGYFNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180509_D00450_0598_BHGYFNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - HJCFFALXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/170517_ST-E00266_0210_BHJCFFALXX/slurm_job_ids.yaml'
Error updating analysis for: case - HLYWYDSXX : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/190927_A00689_0069_BHLYWYDSXX/slurm_job_ids.yaml'
Error updating analysis for: case - HM2LNBCX2 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/181005_D00410_0735_BHM2LNBCX2/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HK33MDRX3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/230912_A00187_1009_AHK33MDRX3/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - HL32LCCXY : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/180508_ST-E00269_0269_AHL32LCCXY/slurm_job_ids.yaml'
Error updating analysis for: case - 22F52TLT3 : [Errno 2] No such file or directory: '/home/proj/stage/flow_cells/20231108_LH00188_0028_B22F52TLT3/slurm_job_ids.yaml'
Updated status 22F52TLT3 - 288425: completed
Updated status HK33MDRX3 - 288426: completed
Updated status HM2LNBCX2 - 288427: completed
Status in Tower: cuddlyhen - 288428 - 3YsiGXRdGQtp6S
Status in Tower: tendergoose - 288431 - 4pfgqVdbxWSXhr
Updated status justhusky - 288432: running
Updating upload job 664516416
Updating upload job 664516417
Updating upload job 664516418
Updating upload job 664516419
Updating upload job 664516420
Updating upload job 664516421
Updating upload job 664516422
Updating upload job 664516423
Updating upload job 664516424
Updating upload job 664516425
Updating upload job 664516426
Updating upload job 664516427
Updating upload job 664516428
Updating upload job 664516429
Updating upload job 664516430
Updating upload job 664516431
Updating upload job 664516432
Updating upload job 664516433
Updating upload job 664516434
Updating upload job 664516435
Updating upload job 664516436
Updating upload job 664516437
Updating upload job 664516438
Updating upload job 664516439
Updating upload job 664516440
Updating upload job 664516441
Updating upload job 664516442
Updating upload job 664516443
Updating upload job 664516444
Updating upload job 664516445
Updating upload job 664516446
Updating upload job 664516447
All analyses updated!

henrikstranneheim commented 4 months ago

@moedarrah A bit surprised that cigrid still work when deploying and testing these PRs. Could you help me understand why it does not fail even though I changed the response?

moedarrah commented 4 months ago

@moedarrah A bit surprised that cigrid still work when deploying and testing these PRs. Could you help me understand why it does not fail even though I changed the response?

I believe the issue might be related to the cache; the Trailblaze endpoints aren't functioning for me on the stage.

henrikstranneheim commented 4 months ago

@moedarrah Right now cg-vm1 is down for maintenance.

islean commented 4 months ago

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

henrikstranneheim commented 4 months ago

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

moedarrah commented 4 months ago

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

Sure 👍

moedarrah commented 4 months ago

@henrikstranneheim Did you mean that the Pipeline column was populated? Or were you surprised that it rendered at all? My guess is that the column would simply be empty with this change, since no value called data_analysis is being exported.

Right, and it is empty. That makes sense. @moedarrah do you have time this week to make a PR to accommodate this?

Yes, we're using conditional rendering on the frontend to prevent UI breaks if the value is missing.

sonarcloud[bot] commented 4 months ago

Quality Gate passed

Issues
0 New issues

Measures
0 Security Hotspots
No data about Coverage
0.0% Duplication on New Code

See analysis details on SonarCloud

henrikstranneheim commented 4 months ago

@moedarrah Could you link the Cigrid PR here when you are done and I can test cg/TB and Cigrid on stage?

moedarrah commented 4 months ago

@moedarrah Could you link the Cigrid PR here when you are done and I can test cg/TB and Cigrid on stage?

The PR has been deployed to the stage: https://github.com/Clinical-Genomics/cigrid-ui/pull/509

henrikstranneheim commented 4 months ago

👍

islean commented 4 months ago

Works well with the new order views as well 💪

henrikstranneheim commented 4 months ago

Planning to M&D on Monday!

henrikstranneheim commented 4 months ago

Deployed in production:

Log deploy... done.
trailblazer, version 21.0.0

Clinical-Genomics / trailblazer