search

HenrikBengtsson / future.batchtools

:rocket: R package future.batchtools: A Future API for Parallel and Distributed Processing using batchtools
https://future.batchtools.futureverse.org
84 stars 9 forks source link

Setting export compression #53

Closed benmarchi closed 4 years ago

benmarchi commented 4 years ago

I was wondering if there was any update on incorporating user defined registry compression in future.batchools (mllg/batchtools#245). It seems like batchtools wants the compression type to be specified in an external configuration file, which is passed as an argument during registry creation. Is it possible for future.batchtools to readily interface with this type of setup or is additional development needed on batchtools?

HenrikBengtsson commented 4 years ago

I've got little time to dive into that, but I'm open for proposals what that could look like and what needs to go on under the hood. It must be designed independently of the Future API itself.

benmarchi commented 4 years ago

My suggestion would be to allow the user to specify the compression type ("gzip" (default), "xz", FALSE, TRUE (equivalent to "gzip"), etc. as arguments in plan. That way the user can change the default compression behavior using something like:

mybatchtools <- tweak(batchtools_slurm, compress = FALSE)
plan(mybatchtools)

My best guess is that future.batchtools will have to point each future registry to a configuration file with the user specified compression type. Here is a code section from the new test in batchtools linked to registry compression. The first three lines seem to be where the compression specification is happening. If future.batchtools can create and link a configuration file based on the user's plan, it seems like batchtools should be passing the option to all its functions.

An alternative would be to modify the registry object after it has been created by batchtools with the user's compression method. I don't know how exactly this would work because it would need future.batchtools to interact with the future during its evaluation.

HenrikBengtsson commented 4 years ago

Thanks for the layout and for bumping this one. Support for controlling the batchtools 'compress' parameters has been added to the develop branch (commit 9419112b8d64ec4a937b6983d3d0e5f641035833)

plan(batchtools_local, registry = list(compress = FALSE))
HenrikBengtsson commented 4 years ago

FYI, future.batchtools 0.9.0 which implements this is now on CRAN