aaronsteers / meltano-demo-in-a-box

Adapted from the fantastic "MDS in a box" project, covering Monte Carlo simulation of the NBA season, leveraging meltano, dbt, duckdb and superset
3 stars 1 forks source link

feat: add inline mapper to append datetime columns #6

Open aaronsteers opened 1 year ago

aaronsteers commented 1 year ago

I've kept the commits fairly tidy so this is easier to follow. Basically:

  1. Add the mapper: meltano add mapper meltano-map-transformer
  2. For testing: meltano add loader target-jsonl
  3. Add the mappings declaration for add-timestamps manually in yaml.
  4. Test by sending the output to target-jsonl: meltan run tap-spreadsheets-anywhere add-timestamps target-jsonl
  5. When it's working properly, add everywhere after tap-spreadsheets-anywhere is referenced in meltano run, Meltano job definitions, Makefile, CI jobs, etc.
plugins:
  mappers:
  - name: meltano-map-transformer
    # ...
    mappings:
    - name: add-timestamps
      # This adds timestamp columns to all of the named streams
      # Debug by running:
      # % meltano run --full-refresh tap-spreadsheets-anywhere add-timestamps target-jsonl
      config:
        stream_maps:
          sample_stream_a:
            _sdc_extracted_at: datetime.datetime.utcnow().isoformat()
          sample_stream_b:
            _sdc_extracted_at: datetime.datetime.utcnow().isoformat()
          # ...