lzamparo / embedding

Learning semantic embeddings for TF binding preferences directly from sequence
Other
0 stars 0 forks source link

Re-write data processing code to use `dask.distributed` for serving data to models #10

Open lzamparo opened 7 years ago

lzamparo commented 7 years ago

This is a bit pie-in-the-sky, but eventually I'd like to not have the burden of micro-managing data processing, instead off-loading scheduling and load-balancing to dask.

This would involve a totally new dataset_reader (or maybe just parallel functionality within?), which would function something like this:

lzamparo commented 6 years ago

Working on this in collaboration with Dask guy. Doesn't seem like Dask distributed is really ready for prime-time; there's no PBS template that just works, nevermind LSF.