Open nbren12 opened 6 years ago
To make this work, I will probably need a way to specify a unique name for dask arrays based on the target. Then, for each target I can create a multiple dispatched function for generating a unique name.
If I make this operate on the level of keys rather than collections, it should work better.
The main issue here is that
targeted
evaluates during graph construction and could access what kind of dask object it is passed. However, if it is passed aDelayed
object, it has no idea of knowing what the output type is.On the other hand,
writer
is executed on task execution, so it should now what the output type is. Unfortunately,reader
will not know what the output type is since it will only have access to thetarget
object. The best solution then is probably to create a Reader/Writer object which knows how to write a given type of data to a variety of luigi targets. Then, I can dispatchtargeted
based on the following rulestargeted(dask_collection, target)
is routed to a default argument for the given dask collection.targeted(delayed, target, reader_writer)
. Delayed objects need to explicitly specify a reader_writer.