search

move-coop / parsons

A python library of connectors for the progressive community.
Other
255 stars 125 forks source link

New Feature: DBSync can upsert or append updated rows #1029

Closed austinweisgrau closed 1 month ago

austinweisgrau commented 3 months ago

The DBSync method can already append new rows from a source db into a target db based on an incremental primary key.

This new method generalizes that behavior in a new feature which allows for appending or upserting rows that have an updated_at_column value in the source db greater than the value in the target db.

austinweisgrau commented 2 months ago

Hm linting

shaunagm commented 1 month ago

This looks merge-able, so I'm going to go ahead and do so!

austinweisgrau commented 1 month ago

Ah shoot I'm sorry I should have raised this earlier

I was testing this method and found some edge cases that make this functionality fail that I found too tricky to resolve and stopped working on it. I should have closed this PR or made a note of it

I don't remember off hand what those edge cases were.

IMO this should probably be reverted - I would consider the new behavior insufficiently tested to be ready for merge

Sorry again for not noting that here

On Thu, May 23, 2024, 1:10 PM Shauna @.***> wrote:

Merged #1029 https://github.com/move-coop/parsons/pull/1029 into main.

— Reply to this email directly, view it on GitHub https://github.com/move-coop/parsons/pull/1029#event-12914702490, or unsubscribe https://github.com/notifications/unsubscribe-auth/AO74QHVUQKNQPQDHA7SV36DZDZEM5AVCNFSM6AAAAABFUJBO7OVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJSHEYTINZQGI2DSMA . You are receiving this because you authored the thread.Message ID: @.***>

shaunagm commented 1 month ago

Gotcha - I will revert. Not a big deal, feel free to open a new PR if you figure out a way around the edge cases.