hyperqueue (>=0.17.0) and Nextflow interpret memory resource differently

[bguo068]

Bug report

Expected behavior and actual behavior

Expected behavior: hyperqueue and nextflow should have same interpretation of memory resources.

Actual behavior: When set memory 2.GB in Nextflow, the .command.run a work directory uses this:

#HQ --resource mem=2147483648

This is problematic as newer versions of hyperqueue interpret 2147483648 as 2147483648 Mb (2 PB) instead of 2147483648 Bytes(2GB). This results in the hyperqueue job waiting for the a big memory resource that would never be possible for most HPCs.

Steps to reproduce the problem

main.nf:

process A {
        executor 'hq'
        cpus 1
        memory 2.GB
        input: val(x)
        output: path("*.txt")
        script:
        """
        echo ${x} > ${x}.txt
        """
}

workflow{
        channel.fromList([1,2]) | A
}

Run the pipeline:

nextflow main.nf

Program output

Nextflow output:

 N E X T F L O W   ~  version 24.04.2

Launching `main.nf` [scruffy_cantor] DSL2 - revision: 091b7ae540

WARN: The support for HyperQueue is an experimental feature and it may change in a future release
executor >  hq (2)
[8e/ff2537] A (2) | 0 of 2                                  <--------------- Never finish

And it will stuck forever.

Example .command.run file:

#!/bin/bash
#HQ --name nf-A_(2)
# ....
#HQ --resource mem=2147483648               <------------------
#HQ --cpus 1
....

Environment

• Nextflow version: 24.04.2 build 5914
• Java version: 17.0.7 2023-04-18 LTS
• Operating system: Linux
• Bash version: GNU bash, version 4.2.46(2)-release (x86_64-redhat-linux-gnu)
• hyperqueue: v0.19.0 <-----------------------

Additional context

break change in hyperqueue about mem resource

[bguo068]

Workaround for now:

process A {
        executor 'hq'
        cpus 1
        memory 2048   //  <---------- Change 2.GB to 2048, to allow hq interpret it as 2048MB
        input: val(x) //    Caveats: if changing executor to sge/slurm, 
        output: path("*.txt") // nextflow will interpret it as 2048bytes
        script:
        """
        echo ${x} > ${x}.txt
        """
}

workflow{
        channel.fromList([1,2]) | A
}

[pditommaso]

Think it's related to this PR https://github.com/nextflow-io/nextflow/pull/4884

[bentsherman]

I suggest we merge that PR and update the docs to say that Hyperqueue >=0.17.0 is required in the next edge release

[bguo068]

I suggest we merge that PR and update the docs to say that Hyperqueue >=0.17.0 is required in the next edge release

That would be great!

[pditommaso]

Solved via b45f7c48