Closed lazamar closed 1 week ago
Fixes #13
Here is an example connecting to two sources, one file and one Postgres table. The example is projecting data into a destination file.
The Postgres data has serial_value and partitioning_value. The file data has no partitioning or serial info implemented.
serial_value
partitioning_value
$ ./utils.sh run run --data-path tmp/data --config tmp/config.yml Up to date configuration loaded src: first_source | connected src: first_source | ingested. { "one": "a", "two": 1 } src: first_source | ingested. { "one": "b", "two": 2 } src: first_source | ingested. { "one": "c", "two": 3 } src: first_source | ingested. { "one": "d", "two": 4 } src: pg_source | connected src: pg_source | ingested. serial_value: 1, partitioning_value: 1 src: pg_source | ingested. serial_value: 2, partitioning_value: 1 src: pg_source | ingested. serial_value: 3, partitioning_value: 1 src: pg_source | ingested. serial_value: 4, partitioning_value: 1 src: pg_source | ingested. serial_value: 5, partitioning_value: 2 src: pg_source | ingested. serial_value: 6, partitioning_value: 2 src: pg_source | ingested. serial_value: 7, partitioning_value: 3 src: pg_source | ingested. serial_value: 8, partitioning_value: 2 src: pg_source | ingested. serial_value: 9, partitioning_value: 1 src: pg_source | ingested. serial_value: 10, partitioning_value: 2 src: pg_source | ingested. serial_value: 11, partitioning_value: 3 dst: file_destination | src: pg_source | sent. id: 1 aggregate_id: 1 dst: file_destination | src: pg_source | sent. id: 5 aggregate_id: 2 dst: file_destination | src: pg_source | sent. id: 7 aggregate_id: 3 dst: file_destination | src: pg_source | sent. id: 2 aggregate_id: 1 dst: file_destination | src: pg_source | sent. id: 6 aggregate_id: 2 dst: file_destination | src: pg_source | sent. id: 11 aggregate_id: 3 dst: file_destination | src: pg_source | sent. id: 3 aggregate_id: 1 dst: file_destination | src: pg_source | sent. id: 8 aggregate_id: 2 dst: file_destination | src: pg_source | sent. id: 4 aggregate_id: 1 dst: file_destination | src: pg_source | sent. id: 10 aggregate_id: 2 dst: file_destination | src: pg_source | sent. id: 9 aggregate_id: 1
Note that I did not add the descriptions for source and destination as that is redundant here since the source and destination IDs are not random hashes but something descriptive chosen by the user already.
Fixes #13
Here is an example connecting to two sources, one file and one Postgres table. The example is projecting data into a destination file.
The Postgres data has
serial_valueandpartitioning_value. The file data has no partitioning or serial info implemented.Note that I did not add the descriptions for source and destination as that is redundant here since the source and destination IDs are not random hashes but something descriptive chosen by the user already.