Closed mpadge closed 2 years ago
@jeroen swapping system
for sys::exec_wait()
improves things enough to make parallel computation feasible. It still fails because of nodes suffering "error reading from connection" that causes a whole-system dump. My guess there is an error attempting to read from the files produced during the exec_wait()
calls, for which there is no signal sent from the process to actually wait for. The whole thing is nevertheless chunked with intermediate saves, and using sys
makes it stable enough to run in parallel, and just restart when system dumps. system
would never even make it through a single chunk.
Great confirmation for the stability of sys
over system
- thanks!
Update: Nope, looking at the results, the parallel job still doesn't work - the failed nodes dump entire arrays of junk values which end up being most of them. The process itself is more stable, but the results end up just as useless. I'll use sys
regardless, but the analyses will have to remain single-threaded.
Following on from ropensci/roweb3#236, and this comment from @jeroen.