[proposal] package-set level operations

datahq / dataflows

DataFlows is a simple, intuitive lightweight framework for building data processing flows in python.

https://dataflows.org

MIT License

194 stars 39 forks source link

[proposal] package-set level operations #62

Open micimize opened 5 years ago

micimize commented 5 years ago

Right now a dataflows and datapackage-pipelines can only do operations within one package at a time, but I think adding another layer to the api for handling a stream of multiple packages would make sense. This would be useful for:

splitting complex packages for multiple consumers
streaming multiple, self contained packages (say, bundles of user-wise data)

micimize commented 5 years ago

related to the package(packages[]) concept

akariv commented 5 years ago

While it's not necessarily the full feature you're describing, it si possible to do some of that functionality already, e.g.:

load resources from various sources into the stream
dump_to_path multiple times along the flow to create multiple packages
replace package metadata (if needed)

It's a cool idea, though, so I'll add it to the roadmap :)