search

NBISweden / Earth-Biogenome-Project-pilot

Assembly and Annotation workflows for analysing data in the Earth Biogenome Project pilot project.
https://www.earthbiogenome.org/
GNU General Public License v3.0
9 stars 8 forks source link

Large memory node needed for certain processes #82

Open n-equals-one opened 4 months ago

n-equals-one commented 4 months ago

I realized that most likely we need a big memory node for the FCS-GX process.

I am not sot sure how to ask for a specific type of node for a certain process. (e.g. "-C mem1TB")

mahesh-panchal commented 4 months ago

Like this in your local nextflow.config:

process {
    withName: 'HIFIASM' {
        cpus   = 20
        memory = 1024.GB
        time   = 5.d
        clusterOptions = { "-A $params.project -C mem1TB" }
    }
}

See https://github.com/NBISweden/BGE-Austropotamobius_torrentium-2024-assembly/tree/main/analyses/01_assembly-workflow_initial-run_rackham

mahesh-panchal commented 3 months ago

Doing some testing: Have implemented ramdisk. Must use large memory node if ramdisk is used, otherwise ends up:

  2024/03/07 16:01:05 NOTICE: Config file "/home/mahesh/.rclone.conf" not found - using defaults
  2024/03/07 16:01:05 ERROR : all.gxi: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:05 ERROR : all.gxs: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : Attempt 1/3 failed with 2 errors and: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : all.gxs: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : all.gxi: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : Attempt 2/3 failed with 2 errors and: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : all.gxi: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : all.gxs: Failed to copy: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 ERROR : Attempt 3/3 failed with 2 errors and: preallocate: file too big for remaining disk space
  2024/03/07 16:01:06 Failed to copy with 2 errors: last error was: preallocate: file too big for remaining disk space

without ramdisk running time is very long, but doesn't use up much memory, or cores. Estelle's run showed:

from execution_traces:
name: FCSGX_RUNGX (hifiasm-decontaminated-default-hap0)
duration: 5d 10h 31m 49s
realtime: 5d 6h 45m 35s
%cpu: 13.2%
peak_rss: 12.9 GB
peak_vmem: 467 GB
rchar: 3.8 GB
wchar: 3.7 GB

i.e. very poor memory and cpu utilization, but took 5.5 days.