airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.51k stars 4k forks source link

File Source: add Box storage provider #4926

Open sriharivishnu10 opened 3 years ago

sriharivishnu10 commented 3 years ago

Tell us about the problem you're trying to solve

Currently, the file source connector supports file providers such as GCS and S3.

Box is also a cloud storage service where users can store/share files (similar to Google Drive)

Screen Shot 2021-07-22 at 4 42 40 PM

Describe the solution you’d like

Allow users to download/sync box files from a given Box file URL.

Describe the alternatives you’ve considered or used

Are you willing to submit a PR?

Yes (in progress)

Phlair commented 3 years ago

This could use the new abstract files source (after this issue is completed)

grubberr commented 2 years ago

This provider requires authorization!!! Even to access public shared-links via API we need to authorize.

User Authentication Oauth2.0 - this type of authorization has one huge limitation. Access Token - works only 1 hour , Refresh Token can be used only once. Because we cannot re-save refresh-token connector will work only 2 hours.

our spec can have:

client_id client_secret refresh_token access_token

but sync will work only short period of time

@sherifnada @misteryeo @YowanR any idea how to be for this provider?

YowanR commented 2 years ago

@grubberr This work is an enhancement and we've decided not to implement it at this time and focus on other higher priority items. @lazebnyi as FYI

To answer your question, since this involves refresh tokens, the team will be working on a solution for this as soon as possible. Tracked here: #3990