Create a 'big job' option in STAR

[angarb]

Problem

• Most of the STAR jobs run within 24 hours on Sumner.
• With the ongoing issues on Sumner, the large Fastq files (>200 million reads) are taking longer than 48 hours to run. (48 hours is our current limit)
• As the time/memory requested impacts our positioning in the Sumner queue, it would benefit us to have a standard time/memory request that encompasses most of our runs and a max time/memory request to accommodate our larger runs.

Solution

• [ ] For large jobs, have the Option to change the 'Mega' resource allocation to 72 hours. Note - I am not sure if it makes sense to change this to 72 or ~67 hours as the time cap on the NF pbs script is 72 hours. This could be a dynamic request (request time limit) or more of a boolean (with a star_max = true).
• [ ] make a max_job pbs script to #SBATCH -t 3-0:00:00?

Implementation

[PhilPalmer]

Hey @angarb You can find mega memory defined here How would you want to toggle this behaviour? I'm guessing a new parameter would make sense but what parameter would you want?

For example, some ideas could be to have:

• --big_job which would increase all of the resources i.e. time + memory + cpus
• --star_time + modify the existing star_memory parameter to be able to set dynamically the resources for star
• Any other ideas? @angarb or @sk-sahu?

[sk-sahu]

I check with this following approaches.

1. Having two label (48h and 72h) which dynamically change based an external param (like @angarb suggested star_max = true) This doesn't seems feasible (Tried every possible combination of hack method)
2. As suggested by @PhilPalmer -

   --star_time + modify the existing star_memory parameter to be able to set dynamically the resources for star

This also seems difficult because of selector priority

Any other suggestions/ideas welcome, including from @cgpu

[angarb]

@sk-sahu and @cgpu - I think I implemented a fix for this on my branch new_config-

In the Sumner config, I set a parameter mega_time that can be overwritten (by the main config). For example, in my test run, I set mega_time to 68.h in the NF_splicing_pipeline.config - and it worked!

I set the default to 20.h, because I think that should be sufficient for most jobs.

params  {
  mega_time = 20.h
  }

  withLabel: mega_memory {
    cpus = { check_max (24, 'cpus')}
    memory = { check_max( 120.GB * task.attempt, 'memory' ) }
    time = { check_max( params.mega_time * task.attempt, 'time' ) }
  }

[angarb]

However, @cgpu and @sk-sahu, definitely open to other ideas!

[angarb]

One sample took 2 days and 9 hours to run through STAR! Eeeep!

[sk-sahu]

Good to know you have a fix @angarb

As I already mentioned above tried different approach, no better solution. So having a different config file only make sense.

If your fix solves the problem can we close this issue?

[angarb]

Hi @sk-sahu, so I actually made a new parameter - mega_time described above.

Making a new config also ended up being a bit more complicated.

Should I push the new parameter to the master? The new parameter is within our Sumner Config.

[sk-sahu]

@angarb yes you can push it to master, but please try to do it though a PR. So that we can make sure everything is okay (just an extra pair of eyes better)

[sk-sahu]

Fixed https://github.com/TheJacksonLaboratory/splicing-pipelines-nf/pull/191 by @angarb