search

estuary / connectors

Connectors for capturing data from external data sources
Other
48 stars 13 forks source link

estuary-cdk: investigate support for discovered incremental bindings with no primary key for airbyte shim #1486

Open williamhbaker opened 5 months ago

williamhbaker commented 5 months ago

Sometimes bindings may need to be discovered without a primary key specified, and this is currently not allowed by the estuary-cdk for airbyte-shim'd connectors, see here

An example of where this would be useful is source-google-ads, which allows users to specify custom queries. These custom query results do not have an obvious primary key, but they will have a discovered schema.

A possible user interaction would be to use the Estuary UI to select a key for the discovered collections after discovery has run. It may also be possible to assign some synthetic key to these collections, although that would be less useful.

jshearer commented 4 months ago

A possible user interaction would be to use the Estuary UI to select a key for the discovered collections after discovery has run

Does this functionality exist today? As in, if a collection is discovered without a primary key, will the UI direct the user to select one? If so, updating the CDK to allow discovery of resources without primary keys ought to be straightforward.

williamhbaker commented 4 months ago

A possible user interaction would be to use the Estuary UI to select a key for the discovered collections after discovery has run

Does this functionality exist today? As in, if a collection is discovered without a primary key, will the UI direct the user to select one? If so, updating the CDK to allow discovery of resources without primary keys ought to be straightforward.

It does, and it's not an awful process, although it could be better. But you can do it! See https://estuaryworkspace.slack.com/archives/C03QBN83GQ4/p1711046584144639