Possible bug with slurm and storage fs plugin for local keyword

[raphaelbetschart]

I have some issues understanding the local keyword when using the storage fs plugin. I am trying to create a rule which takes two files as input: One input file which should get copied to a compute node and another input file which should NOT get copied.

My Snakefile looks like this:

BAM                = "/storage/{sample}.bam"
CRAM_REMOTE = "/storage/{sample}.remote.cram"
CRAM_LOCAL = "/storage/{sample}.local.cram"

REFERENCE_GENOME = "/storage/reference/hg38.fa"

SAMPLES = ["S1", "S2"]

rule all:
        input:
                expand(CRAM_LOCAL, sample = SAMPLES),
                expand(CRAM_REMOTE, sample = SAMPLES)

rule Remote:
        input:
                BAM = BAM,
                REF = REF
        output:
                CRAM_LOCAL
        envmodules:
                "samtools/1.18"
        threads:
                16
        shell:
                """
                samtools view -@ {threads} -C -T {input.REF} -o {output} {input.BAM}
                """

use rule Remote as Local with:
        input:
                BAM = BAM,
                REF = local(REF)
        output:
                CRAM_REMOTE

In rule Remote I would like to copy both input files to the compute node directory, specified like this remote-job-local-storage-prefix: /scratch/$SLURM_JOB_ID".

In rule Local only the BAM input file should get copied to the compute node.

What happens now when I submit these 4 jobs is that only jobs spawned by rule Remote are getting submitted to SLURM, the two jobs from Local are run locally.

My profile is the following:

default-resources:
  cpus_per_task: 16
  mem_mb: 32000
executor: slurm
default-storage-provider: fs
shared-fs-usage:
  - persistence
  - software-deployment
  - sources
  - source-cache
local-storage-prefix: "/scratch/"
remote-job-local-storage-prefix: "/scratch/$SLURM_JOB_ID"

What I am doing wrong here?

Thanks for any hints.