Clinical-Genomics / microSALT

Microbial Sequence Analysis and Loci-based Typing pipeline for use on NGS WGS data.
GNU General Public License v3.0
2 stars 3 forks source link

Set slurm QoS based on priority of case for microSALT #154

Closed moahaegglund closed 1 year ago

moahaegglund commented 2 years ago

Description

QoS is set to high for all jobs in microSALT, instead QoS should be set based on the priority of the case.

This can be closed when

QoS is set according to priority of case.

Mropat commented 2 years ago
    {
        "CG_ID_project": "ACC9306",
        "CG_ID_sample": "ACC9306A28",
        "Customer_ID": "cust056",
        "Customer_ID_project": 980548,
        "Customer_ID_sample": "22ET100024om",
        "application_tag": "MWRNXTR003",
        "date_arrival": "2022-01-26 00:00:00",
        "date_libprep": "2022-01-28 00:00:00",
        "date_sequencing": "2022-02-02 13:00:35",
        "method_libprep": "1764:16",
        "method_sequencing": "1830:7",
        "organism": "Neisseria spp.",
        "priority": "standard",
        "reference": "CP016017.1",
        "sequencing_qc_passed": true
    }

Config for crackeagle generated in cg with priority "stadard" as intended config.log found in crackeagle workdir:


[0|0|0] 1d [hiseq.clinical@hasta:~] [base] $ cat  /home/proj/production/microbial/results/ACC9306_2022.2.2_19.23.5/config.log 
ANALYSIS STARTED BY: B'Hiseq Clinical\N'
{
  "_comment":"Genologics temporary configuration file",
  "slurm_header":{
    "time":"12:00:00",
    "threads":"8",
    "qos":"high",
    "job_prefix":"MLST",
    "project":"production",
    "type":"core"
  },
  "regex":{
    "mail_recipient":"clinicaldelivery@scilifelab.se",
    "file_pattern":"\\w{8,12}_\\w{6,10}(?:-\\d+)*(?:\\w+)*_L\\d{1,3}_(?:R)*(\\d{1})(?:_\\d{1,3})*.fastq.gz",
    "_comment":"Organisms recognized enough to be considered stable",
    "verified_organisms":[
      "Enterococcus faecalis",
      "Enterococcus faecium",
      "Escherichia coli",
      "Klebsiella pneumoniae",
      "Staphylococcus aureus"
    ]
  },

QOS is high for entire case, seems like it is set high by default with microsalt

Seems to be populated from env file


[0|0|0] 1d [hiseq.clinical@hasta:~] [P_main] 4s $ echo $MICROSALT_CONFIG
/home/proj/production/servers/config/hasta.scilifelab.se/microSALT.json

Which seems to be the same envvar for stage and prod

[0|0|0] 1d [hiseq.clinical@hasta:~] [S_main] 3s $ echo $MICROSALT_CONFIG
/home/proj/stage/servers/config/hasta.scilifelab.se/microSALT-stage.json

Staging is low by default. Transferring this to microsalt

This part of code seems responsible https://github.com/Clinical-Genomics/microSALT/blob/2a4638f0748358e3301f064188935cf5bf7fe1d3/microSALT/utils/job_creator.py#L471