target-redshift to SDK - Githubissues

MeltanoLabs / Singer-Most-Wanted

4 stars 0 forks source link

target-redshift to SDK #18

Open aaronsteers opened 3 years ago

aaronsteers commented 3 years ago

Possible inspiration / starting point: https://github.com/transferwise/pipelinewise-target-redshift

techtangents commented 1 year ago

At my work, we have some huge tables in postgres that we want to ingest into Redshift. There are a few around 5 million rows, a couple are around 50 million, and one is about 5 billion.

The batch messages would speed this up dramatically. At this stage, ingesting several million rows at a time is error-prone for us, so the initial ingestions of tables like this are troublesome. The 5 billion row table is infeasible right now.

pnadolny13 commented 1 year ago

Theres been a couple updates in slack related to this:

@menzenski was looking for example SQL targets to get a sense of the difficulty level to rewrite it https://meltano.slack.com/archives/C01RKUVUG4S/p1675462154066899
JP (slack user) requesting info on how to move forward with the transferwise variant that is not maintained well https://meltano.slack.com/archives/C013EKWA2Q1/p1675861468111389

menzenski commented 1 year ago

Yeah, I have been challenged by the lack of maintainer response on the transferwise variant also. I've had an open PR there for a while and there's been no activity in something like 18 months it seems.

pnadolny13 commented 1 year ago

More updates in slack https://meltano.slack.com/archives/C013Z450LCD/p1684388483962929.

A few new forks and a user looking to take over maintenance of the default target.