search

MeltanoLabs / tap-snowflake

Other
4 stars 9 forks source link

working mvp #2

Closed kgpayne closed 1 year ago

kgpayne commented 2 years ago

Closes #1

kgpayne commented 1 year ago

OK, this is getting close 😅 I have stream property selection working for both RECORD and BATCH and also STATE working for RECORD. State and batch need to be worked on a bit. Right now INCREMENTAL is not supported with the BATCH message type (issue sdk#976) in the SDK, as we don't initialise any existing replication keys on that code path, nor do we provide a mechanism for users to report the bookmark associated with their batch 🤔

kgpayne commented 1 year ago

@pnadolny13 once this merges, it would be good to start using it in squared 😅 Also, do we have a sanitised Snowflake user/database/warehouse that we can use for testing? I have been using internal creds for development, but it would be good to create some tests that utilise an example Snowflake instance that is 'safe' to be used in the MeltanoLabs context. A read-only user with access to example data should be more than sufficient 🙂

pnadolny13 commented 1 year ago

@pnadolny13 once this merges, it would be good to start using it in squared 😅

@kgpayne sounds good I created https://github.com/meltano/squared/issues/425 to track migrating to this variant.

Also, do we have a sanitised Snowflake user/database/warehouse that we can use for testing? I have been using internal creds for development, but it would be good to create some tests that utilise an example Snowflake instance that is 'safe' to be used in the MeltanoLabs context. A read-only user with access to example data should be more than sufficient 🙂

We have sample data but we dont have a read only user for that right now but I can create one. I created an issue for tracking it.