search

lisad / phaser

library for batch-oriented complex data integration pipelines
MIT License
3 stars 1 forks source link

Adds helper features for manipulating outputs and sources #103

Closed jeffkole closed 2 months ago

jeffkole commented 2 months ago

With this change, all steps can declare extra_sources and extra_outputs that will be delivered to the step function as named parameters.

Sources and outputs can be in two forms:

  1. records -- a list of dictionaries
  2. mappings -- a dictionary

There are means to load and save these data structures. For the mappings, the mechanism is primitive and would benefit from a configurable serialization mechanism. That is an exercise for another day.

Addresses #93

jeffkole commented 2 months ago

There is more in this PR than I would normally like, but it just managed to get big as I got all the necessary functionality in place. I'm happy to walk through it live if that would be helpful.