search

frank0434 / Master

repo for the master project - water stress for APSIMX lucerne model
Creative Commons Zero v1.0 Universal
1 stars 1 forks source link

Job submission is clumbsome ATM #14

Closed frank0434 closed 3 years ago

frank0434 commented 3 years ago

Need to have an executable .sh which contains all submission requirements.

frank0434 commented 3 years ago

Job running too slow on one core.

frank0434 commented 3 years ago

Modified the build_apsimx to only output configuration files without creating apsimx files. The HPC setup seems not to support openlava yet.

https://mschubert.github.io/clustermq/articles/userguide.html#configuration

frank0434 commented 3 years ago

Suggestions:

  1. nextflow is probably an option.
  2. LSF is the platform behind openlava
frank0434 commented 3 years ago

A combination of R and bash code tofind --> concatenate commands --> split --> submit

frank0434 commented 3 years ago

unfamiliar error message

libgcc_s.so.1 must be installed for pthread_cancel to work

frank0434 commented 3 years ago

find and chop

[Frank😨]$ find Data/ProcessedData/ -maxdepth 1 -name "*.txt" | split -l 2 -d - "batch"

split into length 2 vectors and labelled by numbers with prefix as batch

frank0434 commented 3 years ago

bsub can't process split output asub can't do -e / -o flags

frank0434 commented 3 years ago

use R to find all configuration files and generate a list of jobs to run

$bsub -e Logs/CreateSH_$$.err -o Logs/CreateSH_$$.out Rscript generateInstructionFile.R -j createSH
frank0434 commented 3 years ago

Debugging the core.* files

https://support.nesi.org.nz/hc/en-gb/articles/360001008136#h_cf410d73-e14d-4abf-897a-374c965aa9dc

frank0434 commented 3 years ago

this is not going to go anywhere. Restructure the configuration file and document in a new issue.

frank0434 commented 3 years ago

https://github.com/frank0434/Master/issues/16#issue-752878269

frank0434 commented 3 years ago

FLAW DESIGN CLOSE FOR NOW