search

ojwoodford / batch_job

Parallelize MATLAB for loops across workers, without the Parallel Computing Toolbox
MIT License
18 stars 6 forks source link

excluding master MATLAB from number crunching #13

Closed spotlightgit closed 4 years ago

spotlightgit commented 4 years ago

Hey Oliver,

in my use case I have a couple of machines which I want to use for parallelisation and my own machine should be part of this cluster. To access different machines you are using the SSL connection and the number of workers on this machine is also specified in this synatax: {'machine_xy',numWorkers} Unfortunately I am not able to use this syntax: {'own_machine',numWorkers} for using my local machine for number crunching. It seems like (at least Windows) ist not able to have client and server on the same machine for this use case (using certificates for authorization to avoid password prompt). Therefore I am using this syntax: {'',numWorkers} to tell your toolbox to include the local machine for number crunching. Part of your implementation is, that you decrease the number of generated workers with 1 and use the actual already running Master-MATLAB instance for number crunching too. In my use case I run a huge complex simulation model which throws a lot of warnings which could not be deactivated. Therfore the command line of my Master MATLAB is flooded with unneccessary information and I am not able get orientation from the command line output of my master script, which controls the parallelisation. I don't know if there is much benefit to use the master MATLAB for number crunching or not. I think it is helpfull for debugging at the beginning (understanding your toolbox), but it has disadvanteges if everything is up and running. My suggestion is that you implement an option, if the master MATLAB should be used for number crunching or not. Than the user is able to decide what is most beneficial for him. What do you think?

ojwoodford commented 4 years ago

You could use the '-async' mode, and wait until the job finishes (using '-progress') before retrieving the output.

ojwoodford commented 4 years ago

Actually, look at the '-timeout' option: "If non-zero, the current MATLAB instance is not used to run any iterations". So you could just use a large timeout value.