Open mtmorgan opened 5 years ago
Thanks for reporting. I guess I need to implement something like cf$startCluster()
and cf$stopCluster()
and call it internally in submitJobs()
. This has the drawback that submitJobs()
would have to wait for all jobs to finish, and thus asynchronicity is lost.
Just out of curiosity, where did this come up? Is this a problem while running R CMD check or for real world applications?
It is related to #221 and to checks in https://github.com/BiocParallel, both of which have consequence in real-world applications (I think). The connections are still open because the finalizer hasn't run. When it does run, the order in which the finalizer runs is not deterministic (https://stat.ethz.ch/pipermail/r-devel/2011-July/061612.html; it's added to a linked list of SEXP; the order of elements in the linked list depends on what other objects are added to / removed from the linked list; periodically, the finalizer runs at a time when symbols referenced by the finalizer (e.g., the socket connection used in serialize()
to send the "DONE" signal to the worker) have already been cleaned up; this signals an error).
For my use case I would be happy to be able to enforce synchronicity by calling stopCluster()
directly (I don't think the 'user' has access to the Socket instance directly?)
Some
makeCluster*
operations have side effects, e.g., opening connectionsThere is no way to 'undo' (e.g.,
destroyCluster(cl)
) these side-effects, and they are not destroyed by, e.g.,removeRegistry()
. I realize that there is a finalizer, sooften works, but actually finalizers are not run in a deterministic order so that the this is not robust.