search

singer-io / tap-harvest-forecast

A Singer.io tap for extracting data from the Harvest Forecast API
GNU Affero General Public License v3.0
7 stars 10 forks source link

assignments table contains deleted data #9

Open cdilga opened 5 years ago

cdilga commented 5 years ago

Current Behavior: When a assignments are deleted and both the people assigned and projects the people are assigned to are still active, then the Harvest Forecast API no longer returns the records, though the singer target will continue to hold the out of date assignments.

Investigation: It was found that the hard deletes were not being handled by the singer tap by doing a full table refresh, and instead were being incrementally updated on the assignments.updated_at field.

The metadata returned by the tap when run in discovery mode indicates that it doesn't support a FULL_TABLE refresh method, which can be verified by running tap-harvest-forecast --config config.json --discover > catalog.json to inspect the supported methods.

Expected behaviour: Singer.io taps will handle hard deletes and not cause stale data in a singer tap or data warehouse endpoint

Proposed solution: Force the tap to use a FULL_TABLE replication method in the metadata of the stream so that services like Stitch can update correctly

cdilga commented 5 years ago

Further investigation and trialing found that setting the metadata has no effect, as this just advertises the available functionality to other tools.

The ACTIVE_VERSION message is required when processing streams, in addition to version numbers

kevincal commented 4 years ago

I can confirm this is still an issue. Only full table refresh removes the deleted rows. bump.

JDW818 commented 1 year ago

Did everyone give up on this? Running into this same issue 3 years later.

cdilga commented 1 year ago

Yep, we ended up just doing full table loads

On Thu, Apr 20, 2023 at 3:54 AM Jess Williams @.***> wrote:

Did everyone give up on this? Running into this same issue 3 years later.

— Reply to this email directly, view it on GitHub https://github.com/singer-io/tap-harvest-forecast/issues/9#issuecomment-1515141448, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA7DCJ4UE7PBDDP7CAK4UE3XCARGVANCNFSM4IZGNIIA . You are receiving this because you authored the thread.Message ID: @.***>