work on same project with different registry

[smilesun]

For instance, I am doing a very large project, since there are maybe 40 algorithms and 4 problems and I cant wait until I finish all the code to submit my jobs. So I do it in small piece and once I finish a block of my code, I copy the folder to a new folder(actually I have several branches and each branch has a folder) and submit the new added algorithms.

1. Not sure if this is a good way to continuously work on a project ? and how to merge these different folders? Since I am always afraid that once I rerun the program, already existed results will be overwritten.

2. I have to copy a folder, rename it and load the result to submit the new algorithm job or just resubmit those expired jobs, but since i CHANGED the folder name, I got

   Warning messages:  In loadRegistry("bt_reg_classif2/") :
   The absolute path of the file.dir has changed (before 'xxx', now 'xxx'). Enabling read-only mode for the registry.
   The work.dir '' does not exist, jobs might fail to run on this system.

   Is it a good habit just to hack it with reg$writable = T ?

[mllg]

This has now completely changed. You automatically enter a read-only mode if you change the registry in the background. My suggestions for running stuff on the same system:

1. Keep one session up for read-write ops: submit, sync, waitForJobs, ...
2. Open 2nd session with loadRegistry(..., writeable = FALSE) to monitor, inspect logs, reduce partial results, ...

If you need to parallelize on multiple systems, I'd do as follows:

1. Create jobs locally
2. Thoroughly test stuff
3. Copy file dir to system1 and start subset of jobs
4. Copy file dir to system2 and start subset of jobs not startet on system1
5. Copy back file dirs and use batchtools:::mergeRegistries() locally to merge registries (https://github.com/mllg/batchtools/blob/master/R/mergeRegistries.R). This function is not well tested yet, you might need to fix smaller stuff.

If you add new jobs or redefine jobs after 2, registries will diverge.