Closed 6zhc closed 4 years ago
Hey, What hardware are you running this on? I am maybe wrong, but nextflow is maybe launching processes on your computer? The following situation doesn't seem to conflict with what you are saying: If nextflow is running on your local computer which has, (lets say 8 cpus), he will be able to submit 8 processes on your local machine, and maybe this is too much for the computer which explode your RAM and then start swapping.... If this is the situation, you could try and limit the number of submitted processes to 1.
I don't think so. The log shows there is just three process working and having worked for alongtime. Createrecord should be 9 process, and other six has been finished.
Yes, but only the creation of the training record is long and memory intensive (the others are for validation and test and are relatively small) Is there any swap going on or what hardware are you running on? I believe your issue is only a nextflow/hardware configuration
I run it on the sever with 4CPU each has 12 processors, 4 GTX1080 GPU and 256G memory. I think hardware is good enough. I have no idea how to change the config, for nextflow is totally new for me. I just use your configure. it might cause by it. Could you tell me how to set it?
ok, indeed you hardware is good enough.
This is one local machine and with such specs it shouldn't pause.
Could you add to your nextflow.config the following lines:
profiles {
local {
process.executor = 'local'
executor.queueSize = 1
}
}
This should force nextflow to only submit one process at a time. Hopefully this should work. If you were running on a cluster (like a SGE cluster, or PBS, or slurm)
You can set process.executor = 'sge'
to the right scheduler.
If this fails, I would connect to my server, change directory to one of the hanging processes (like /home2/zhc/DRFNS/work/4a/265bb911fd52b33cb3f49a2f6e5b22 for instance), and run the command: bash .command.sh This should run, show you python errors and other specific errors in code.
I clone your code and alreay build the environment for it. After I run the commond
nextflow run realdataset.nf --epoch 80 -c nextflow.config -resume
, It suck in the step of createTFrecord and the mem usage increase fast. After almost 4 hours, it still not get to the next step and memory usage come to more than 100G. I wonder whether it go wrong? I use the dataset download bydownload_data.sh
Here is the output in the .nextflow.log