search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.55k stars 4.01k forks source link

allow uploading files to the UI #4084

Open jrhizor opened 3 years ago

jrhizor commented 3 years ago

It's not intuitive to copy entire config files into a text box in the UI. This is mostly a concern since BigQuery, one of our more popular connectors, uses copied file contents. Since we should expect many potential users to attempt setting up BigQuery, we should make this experience as easy as possible for them.

I searched for a duplicate for this issue since we've talked about this many times but I didn't find it.

akoshterek commented 3 years ago

@jrhizor is it related somehow to https://github.com/airbytehq/airbyte/issues/3657 ? there is a waiting PR

jrhizor commented 3 years ago

It's related but not exactly the same as https://github.com/airbytehq/airbyte/issues/3657.

While #3657 may help in some deployment scenarios, all file uploads for configs should use something like https://developer.mozilla.org/en-US/docs/Web/HTML/Element/input/file instead of copy-pasting contents. This is more of a general UX problem which still exists if users don't deploy Airbyte to something that has service users permissioned for BQ.

marcosmarxm commented 3 years ago

This can be useful using ssh and ssl credentials too.

cgardens commented 3 years ago

Cool. @jrhizor how should we signal to the UI that a field can handle file uploads? Sounds like we will need to specify something in the spec.

jrhizor commented 3 years ago

I'm imagining something similar to airbyte_secret or multiline where it's just a flag in the spec for the specific fields. "file_upload": true maybe?