Issue with polyfun.py and memory allocation

[mkoromina]

Hi Omer,

Thank you for all the hard work on Polyfun software!

I have a question about the polyfun step (polyfun.py script). I ran it using two different ways (i) implemented through workflows such as snakemake and nextflow, and (ii) through a normal job submission on our local cluster. When I followed approach (ii), the job ran all the way through using roughly 250GB of memory. However, when I opted for approach (i) and requested through the workflows 300GB of memory for the polyfun.py step, I received the following error message: (error message exported from the traceback)

  File "/home/jjs/conda/7688fdtg/lib/python3.8/site-packages/pandas/core/array_algos/take.py", line 158, in _take_nd_ndarray
    out = np.empty(out_shape, dtype=dtype)
numpy.core._exceptions.MemoryError: Unable to allocate 782. MiB for an array with shape (187, 548212) and data type float64

Do you know what could be going wrong perhaps in that instance? Many thanks!

[jdblischak]

If approach ii worked, then the problem with approach i is almost certainly a problem with the resource allocation with the pipeline tool.

You should be able to create a test rule that returns the amount of memory available to it. This will enable you to confirm you can adjust the memory allocated to each step on your cluster. Below is an example for Snakemake and Slurm. Even easier, you could add the equivalent of echo "$SLURM_MEM_PER_NODE" right before the call to polyfun.py. This will ensure you know exactly how much memory is being allocated to that step.

rule report_mem:
    output:
        "output/memory.txt",
    shell:
        """
        echo "$SLURM_MEM_PER_NODE" > {output}
        """

If you happen to be using Snakemake and Slurm, I recommend trying the simple Slurm profile I created, smk-simple-slurm. Note that as of now (2024-04-01 for future readers), it requires Snakemake 7 (we're working on Snakemake 8 support).

[mkoromina]

Thank you @jdblischak! I noticed that there were incompatibilities between the resources allocated using the cluster.yaml file from snakemake, compared to the resources allocated after setting up a snakemake slurm profile. I am closing this issue, as it reflects further cluster and memory issues (and I can follow this up in your repo if it's okay)!