search

SOCR / CBDA

Compressive Big Data Analytics (CBDA)
GNU Lesser General Public License v3.0
14 stars 3 forks source link

Make the submission of multiple experiments withing a single bash script #2

Open simeonem opened 8 years ago

simeonem commented 8 years ago

Submit multiple experiments at the same time with a single bash script. Right now I am submitting single R script with the nohup Rscript "script.R" & option (batch and background mode). I would like to be able to give a list of specs and then loop through them with a script that submits them all independently.

simeonem commented 7 years ago

I am working right now to generate a workflow that can work on almost "any" batch system (not just on Cranium). The reasons are many....My goal now is to have the workflow run on OSG-Condor, where I was able to get unlimited CPU/hrs of compoutation. OSG (Open Science Grid) is perfect for the CBDA-SL jobs, since each job is short-burst (5-60 minutes), whether we run a simple binomial with the SuperLearner function, or a Multinomial with a CV.SuperLearner function. The LONI pipeline server (i.e., Cranium at usc.edu) is great, but limited to 3000 jobs to be submitted simultaneously. By using makeflow or Pegasus, we are working to recast the current LONI pipeline workflow into a more general platform that can potentially run on Condor (OSG), PBS (most medium size servers), slurm (XSEDE). If this step works (it will !!), we can potentially submit ~1 million jobs (in 10K batches) almost at the same time. Any suggestion is welcome....;-)