toluaina / pgsync

Postgres to Elasticsearch/OpenSearch sync
https://pgsync.com
MIT License
1.11k stars 174 forks source link

BUGFIX: Fixed the through table issue on insert #413

Closed Shankar-khati closed 1 year ago

Shankar-khati commented 1 year ago

Fixed:

142

264

321

Insert is not working with through tables:

example schema

    {
        "database": "book",
        "index": "book",
        "mapping": {
            "authors": {
                "type": "nested"
            }
        },
        "nodes": {
            "table": "book",
            "columns": [
                "id",
                "isbn",
                "title",
                "description",
                "tags"
            ],
            "transform": {
                "rename": {
                    "id": "book_id",
                    "isbn": "book_isbn",
                    "title": "book_title"
                }
            },
            "children": [
                {
                    "table": "author",
                    "columns": [
                        "id", "name", "date_of_birth"
                    ],
                    "label": "authors",
                    "relationship": {
                        "type": "one_to_many",
                        "variant": "object",
                        "through_tables": [
                            "book_author"
                        ]
                    }
                }
            ]
        }
    }
]

PROBLEM:

in _get_foreign_keys function for though tables we are returning data like { 'pulblic.authors': ['id'], 'public.book': ['isbn'], 'public.book_authors': ['author_id', 'book_isbn'] // this is the node we get }

Filers were not generating correctly since in above output I can't decide like public.book_authors.author_id should map to pulblic.authors.id and public.book_authors.book_isbn should map to public.book.isbn.

FIX

Use get_foreign_keys instead of _get_foreign_keys and get the foreign keys one by one,

  1. get foreign keys in between book_authors and authors table and update filters.
  2. get foreign keys in between book_authors and book table and update filters.

That way we are able to perform all crud operation in book_authors, book and authors table

TEST CASES- PASSED

@toluaina PTAL

toluaina commented 1 year ago

Thanks for putting this together and sorry for the delay. I found a limitation with this approach had to spend some timedoing more analysis.

I addressed this in this commit ec966b91477957dc98f394db951fba0fbfcbc491 on the main branch

Shankar-khati commented 1 year ago

Thanks @toluaina. it's fixed in main branch.