search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.81k stars 4.05k forks source link

When a new stream is added, access should be checked #2503

Open roshan opened 3 years ago

roshan commented 3 years ago

Tell us about the problem you're trying to solve

I have multiple databases I'm syncing from. Occasionally, I would like to add a new table from one of these databases to the sync. However, since the database is controlled by someone else, the user for the connection may not have access yet to the table. Airbyte will let me set up a sync to this table and then fail on sync attempt. Because of the prompt to reset the data when you add a new source stream in, this can mean that the destination is data-less while things are fixed.

Describe the solution you’d like

Just like initial connection errors, I'd like immediate feedback on whether this sync will work.

Describe the alternative you’ve considered or used

I don't have one. This makes permitting access to my Airbyte instance to non-engineers mildly risky (they can make a natural error without knowing).

Additional context

Add any other context or screenshots about the feature request here.

┆Issue is synchronized with this Asana task by Unito

sherifnada commented 3 years ago

@roshan thanks for reporting the problem -- completely agreed with your assessment here. I just want to understand how to reproduce. Is the following what you did:

  1. Have N tables in your source database
  2. Setup a sync from your database using N-k tables
  3. Sync runs fine
  4. Decide you want to sync another table, so you add it to the sync schema from the settings page
  5. Your DB user doesn't have permissions on the new table, but connection is not checked, so sync fails
roshan commented 3 years ago

My apologies for not producing reprex. Those are accurate instructions.

evantahler commented 2 years ago

Noting that these checks are probably specific to each connector - tagging the issue for the connector team(s)

cgardens commented 2 years ago

@grishick do we do this for dbs? I know we added some more stuff to check in dbs, but can't remember if this was one.

sherifnada commented 1 year ago

added to ops backlog as part of permission checking work. Please bounce back if this is incorrect categorization

alafanechere commented 2 months ago

This sounds like an AvailabilityStrategy to define at the source level. Given the age of this issue this might have been already implemented. Will assign to @evantahler's team.