search

edenduthie / pals

Web application implementing the protocol for the analysis of land surface models
1 stars 3 forks source link

R: Analyses triggered by MO upload all run on one processor #52

Open gabsun opened 11 years ago

gabsun commented 11 years ago

But this doesn't happen when all analyses are re-run, apparently - please check with Eden. Yoichi suspects batch call from Java means a single R instance.

edenduthie commented 11 years ago

I think that the multi-threading was removed from this at some time in the past because it was causing problems.

yoichi01 commented 11 years ago

On 13/11/2012, at 4:11 PM, Eden Duthie notifications@github.com<mailto:notifications@github.com> wrote:

I think that the multi-threading was removed from this at some time in the past because it was causing problems.

Thanks for that.

I haven't yet looked at the code and the issue.

Although I don't have to yet (because I am not deploying on tempest), I have swapped Java to Oracle 1.6.0-20.

Now TestNG reports more problems. (I think that I had just 3 errors yesterday).

Although I have not looked through them all, the firs error is about ImageIO.

I think that I sort of remember that I had to install something extra for this. (I could be wrong). Do you know why?

Thanks. Yoichi

yoichi01 commented 11 years ago

Apparently, R runs on only one core as a default. In order to spread the calculation to multicore, it requires a specific package before v. 2.14. Quote: From R Version 2.14.0 add-on, packages are not necessarily required due to the inclusion of the "parallel" package as a recommended package shipped with R. "parallel" includes functionality from the "multicore" and "snow" packages, largely unchanged. More ref: CRAN Task View: High-Performance and Parallel Computing with R (http://cran.r-project.org/web/views/HighPerformanceComputing.html) "multipcore" is an implicit parallelization package. It uses CPU and OS ability to use multicore. It does not require R syntax to be changed. Other packages, e.g. Revolution and Rmpi (R interface to Open MPI library) do require R syntax change to implement explicit parallelization in the logic. On tornado, the latest R is 2.12.2. "multicore", "snow" and "snowfall" have been installed by Martin but not "parallel". On tempest, the latest R is 2.12.2. None of the multicore or parallel package has been installed. Probably the observation of the problem was made on tempest? The solution may be simply to load the appropriate packages. Kai, however, is using Rmpi package, which may have solved the issue with explicite paralleization. On the PalsDevVM, R 2.15 is installed (with "parallel") and it would be impossible to reproduce the problem. On the other hand, reproduing the problem on tempest is also fruitless. Solution may be to request Kai to load R with "multicore" on tempest to match the environment of tornado. (I am guessing that "snow" and "snowfall" are not used.)

yoichi01 commented 11 years ago

My theory of not having multicore (R module) on tempest does not seem right. As Gab suggested I might try to upload another MO file on Tempest and Tornado. Also, as Eden suggested, it might have been because multi-threading was switched off at one time.

yoichi01 commented 11 years ago

Tested on Tornado and all R processes propagate to 8 cores. It seems 2 cores are reserved to system and other processes, but Martin said that he has not set it at all.