Open alexander-held opened 1 year ago
This will require a different approach to handle remote workers without access to a shared filesystem. To handle this, histograms would have to be passed back and saved at the head node. The current API does not surface the call to utils._name_and_save
, which is quite deep inside internal APIs.
Currently route.apply_to_templates
does generic function execution. This might have to be changed to support handling return values and passing those back to the higher level, e.g. templates.build
for saving. The downside of this is that saving would only happen (in the non-parallel version) after all templates are constructed (and passed back as histograms).
to-do / think about:
This updates the
templates.build
,templates.collect
, andtemplates.postprocess
implementations to accept a list of templates to produce via a newtemplate_list
keyword argument. The full list can be obtained viaroute.required_templates
. By default, all templates will still be processed.This allows users to parallelize the template histogram production / collection, partially addressing #401. An example of this would be the following:
compared to the non-parallelized version, which is unchanged:
breaking change:
route.apply_to_all_templates
toroute.apply_to_templates
and changed signature: now takes a list of tasks, which can be generated byroute.required_templates