I've kept the commits fairly tidy so this is easier to follow. Basically:
Add the mapper: meltano add mapper meltano-map-transformer
For testing: meltano add loader target-jsonl
Add the mappings declaration for add-timestamps manually in yaml.
Test by sending the output to target-jsonl: meltan run tap-spreadsheets-anywhere add-timestamps target-jsonl
When it's working properly, add everywhere after tap-spreadsheets-anywhere is referenced in meltano run, Meltano job definitions, Makefile, CI jobs, etc.
plugins:
mappers:
- name: meltano-map-transformer
# ...
mappings:
- name: add-timestamps
# This adds timestamp columns to all of the named streams
# Debug by running:
# % meltano run --full-refresh tap-spreadsheets-anywhere add-timestamps target-jsonl
config:
stream_maps:
sample_stream_a:
_sdc_extracted_at: datetime.datetime.utcnow().isoformat()
sample_stream_b:
_sdc_extracted_at: datetime.datetime.utcnow().isoformat()
# ...
I've kept the commits fairly tidy so this is easier to follow. Basically:
meltano add mapper meltano-map-transformer
meltano add loader target-jsonl
mappings
declaration foradd-timestamps
manually in yaml.target-jsonl
:meltan run tap-spreadsheets-anywhere add-timestamps target-jsonl
tap-spreadsheets-anywhere
is referenced inmeltano run
, Meltano job definitions, Makefile, CI jobs, etc.