Open wlandau opened 4 years ago
This could affect cloud computing strategies such as https://gist.github.com/DavisVaughan/865d95cf0101c24df27b37f4047dd2e5. On the bright side, though, future.callr
seems unaffected.
It's a hack, but you can avoid the shutdown of the cluster by using:
plan(sequential, .cleanup = FALSE)
Thanks, Henrik! I just committed https://github.com/wlandau/targets/commit/1ede0fb0d4c85fb0d7371fec5c0f9928475ef91e, which uses .cleanup
to preserve clusters until the very end of the pipeline. Seems to work. But what exactly makes .cleanup = FALSE
a hack?
Only that it exists in the first place for internal purposes and I don't want to commit to long term support for it at this time - despite it's been the for a long time already
Will there eventually be a different officially supported way to preserve clusters to allow more heterogeneity among workers?
This issue is related to https://github.com/HenrikBengtsson/future/issues/263#issuecomment-570395858 and https://github.com/wlandau/targets/issues/198#issuecomment-712433554, and it interferes with the ability of
targets
to create heterogeneous transient workers (cc @mattwarkentin).When I stick to a multisession plan before running any futures, things seem to work fine.
Created on 2020-10-19 by the reprex package (v0.3.0)
However, when I try to change to a sequential plan before calling
resolved()
, the connection gets corrupted.Created on 2020-10-19 by the reprex package (v0.3.0)
Same thing if I try to restore the original multisession plan.
Created on 2020-10-19 by the reprex package (v0.3.0)