datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.
https://dataflows.org
MIT License
194 stars 39 forks source link

Ignore resource stream #103

Open micimize opened 5 years ago

micimize commented 5 years ago

Is it possible to add a resource without streaming it's contents?

For context, I made a partitioning dataflow a while back, but would like to optimize it so that I'm writing directly to a csv, and not re-reading the results downstream. The partition target is an endpoint, but still part of the resulting package.

The alternative to a mechanism for ignoring a resource stream would be to remove it and add it back after the data flow, which just feels inelegant