airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
16.42k stars 4.18k forks source link

Streams Permissions Research #15217

Closed misteryeo closed 2 years ago

misteryeo commented 2 years ago

We recently introduced an additional step in our connector certification process which ensures that we're aware of the granular permissions required for a specific stream supported by a connector.

Please retroactively include this information in the research docs for the connectors we are certifying this quarter and our existing GA connectors:

ycherniaiev commented 2 years ago

TikTok: added granular permission research. There are 4 main roles that can access TikTok Ads account and seems like noone has restrictions on retrieving info on ads, campigns, reports etc.

misteryeo commented 2 years ago

@YuliiaNahoha Looks like @ycherniaiev has started work on this. Should this be added to the sprint and status updated to In Progress?

cc: @YowanR

ycherniaiev commented 2 years ago

Braintree oAith is in closed Beta now so closing with no updates

YowanR commented 2 years ago

Braintree oAith is in closed Beta now so closing with no updates

@ycherniaiev Could you clarify what that means? Are API keys available for Braintree as an alternative to OAuth or are we blocked here?

ycherniaiev commented 2 years ago

Braintree oAith is in closed Beta now so closing with no updates

@ycherniaiev Could you clarify what that means? Are API keys available for Braintree as an alternative to OAuth or are we blocked here?

@YowanR no, I think we are not blocked, we are blocked on oauth implementation.

igrankova commented 2 years ago

@YowanR we finished updating researches. Please review this issue and close if there's nothing to add. Thank you.

misteryeo commented 2 years ago

@YowanR My suggestion would be to have issues created for each individual connector where we've identified that there is a need for specific permissions so that the engineering team can go back improve the UX on this.

@sherifnada can you remind us what needs to be done here? As I recall it was:

  1. Update the error message so that it's actionable in fixing up the permissions error for a specific stream and;
  2. Update the connector so the rest of the sync continues even if a specific stream fails
igrankova commented 2 years ago

@YowanR can you please clarify this task for me?

ycherniaiev commented 2 years ago

@misteryeo This issue from unblockers prospective has two dimensions: API key( it rarely has permissions issues, but sometimes like in Stripe, admin can configure keys of different lists of permissions for different users - maybe user should be notified about this).Second: oauth that itself has two parts: app's(here like in Facebook or Instagram we should request access to certain streams separately- here we can help and create issues that either already resolved or will be resolved by unblockers) and user's ( there some roles available in certain connectors like TikTok: one like admin has after aftorization full access, other like standard user has not full access to data - these issues should be said to our user). All above we can do and create issues. Regarding error messages-maybe it is better to consult with devs?CC @igrankova

YowanR commented 2 years ago

Thanks @igrankova and @ycherniaiev. I agree with what @ycherniaiev said. As for the error message, I also agree that this should be left to the devs. For now, just making sure that we have the issues created and have some way to know which connectors are impacted by different permission for different streams should be enough. As we go deeper into getting per-stream up and running, we can then actually go and make changes to the connectors as need-be.

igrankova commented 2 years ago

Templates: Source [Source Name] enhancements according to MLP

igrankova commented 2 years ago

See the task for epics/issue creating here