transferwise / pipelinewise-target-postgres

Singer.io Target for PostgreSQL - PipelineWise compatible
https://transferwise.github.io/pipelinewise/
Other
21 stars 39 forks source link

Stream not mapping to correct schema #87

Open Vaslo opened 2 years ago

Vaslo commented 2 years ago

Hello,

I felt like I have this setup correctly, but I cannot get the schema_mapping object to push the data toward the alternative schema. Basically most of my data goes to the 'jaffle_shop' schema, but I do have one table that should go to the stripe schema. I name my schema ids in the tap, but it keeps forcing the one for the 'stripe' schema into a new table in the 'jaffle_shop'. I followed the readme and looked at the notes in the code and thought I set this up correctly, but no luck. Any thoughts as to why this doesn't work?

extractors:
  - name: tap-spreadsheets-anywhere
    variant: ets
    pip_url: git+https://github.com/ets/tap-spreadsheets-anywhere.git
    config:
      tables:
        - format: detect
          name: customers
          path: file:///home/gvas/data_engineering/meltano_projects/jaffle_shop/extract
          pattern: jaffle_shop_customers*.csv
          start_date: "2017-05-01T00:00:00Z"
          key_properties: [id]
        - format: detect
          name: orders
          path: file:///home/gvas/data_engineering/meltano_projects/jaffle_shop/extract
          pattern: jaffle_shop_orders*.csv
          start_date: "2017-05-01T00:00:00Z"
          key_properties: [id]
        - format: csv
          name: payment
          path: file:///home/gvas/data_engineering/meltano_projects/jaffle_shop/extract
          pattern: ^stripe
          start_date: '2017-05-01T00:00:00Z'
          key_properties: [id]
  loaders:
  - name: target-postgres
    variant: transferwise
    pip_url: pipelinewise-target-postgres
    config:
      user: postgres
      dbname: dbt
      default_target_schema: jaffle_shop
      schema_mapping:
        payment:
          target_schema: stripe

Thanks in advance.