compose / transporter

Sync data between persistence engines, like ETL only not stodgy
https://github.com/compose/transporter/issues/523
BSD 3-Clause "New" or "Revised" License
1.45k stars 213 forks source link

Postgres sink update error where table name exists in multiple schemas #497

Closed matt-best-elateral closed 1 year ago

matt-best-elateral commented 4 years ago

Directions

Error write error, All primary keys were not accounted for when importing into Postgres sink where the table name exists in more than one schema.

Bug report

Relevant pipeline.js:

System info:

Reproducible Steps:

  1. Create two tables with the same name, in different schemas:
CREATE TABLE public.example_table
(
    public_example_table_id bigint NOT NULL,
    CONSTRAINT example_table_pkey PRIMARY KEY (public_example_table_id)
)

CREATE SCHEMA other
    AUTHORIZATION postgres;

CREATE TABLE other.example_table
(
    other_example_table_id bigint NOT NULL,
    CONSTRAINT example_table_pkey PRIMARY KEY (other_example_table_id)
)
  1. Run the script used by the Postgres writer, as found at https://github.com/compose/transporter/blob/9e154e76b7d2977d9ac7756660779b512cace87f/adaptor/postgres/writer.go#L178-L187
        SELECT
            column_name
        FROM information_schema.table_constraints constraints
            INNER JOIN information_schema.constraint_column_usage column_map
                ON column_map.constraint_name = constraints.constraint_name
        WHERE constraints.constraint_type = 'PRIMARY KEY'
            AND constraints.table_schema = 'other'
            AND constraints.table_name = 'example_table'
/*
column_name
----------------
public_example_table_id
other_example_table_id
*/
  1. Note that the primary keys from both tables are returned, not just the primary key from the requested schema.
  2. Run transporter, writing to the Postgres sink. The error write error, All primary keys were not accounted for is returned and the etl fails.

What did you expect to happened?

Only the primary keys for the destination table and schema are used.

What actually happened?

The import failed as transporter erroneously identified a primary key as missing from the message data.

matt-best-elateral commented 4 years ago

PR raised: https://github.com/compose/transporter/pull/498