search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.87k stars 4.07k forks source link

Airbyte API should allow to set connector parameters #10001

Open awoehrl opened 2 years ago

awoehrl commented 2 years ago

Tell us about the problem you're trying to solve

When running connectors via the API, it should be possible to change parameters. A common use case would be backfilling with an external orchestrator like Prefect or Airflow. The start_date could be set by the orchestrator for example and the orchestrator would loop through all neccessary days.

See discussion on Slack: https://airbytehq.slack.com/archives/C021JANJ6TY/p1643203718010400

Describe the solution you’d like

Have all customizable connector parameters available via the api. These parameters when set from an api run request should overwrite the values set in the connector.

Describe the alternative you’ve considered or used

Add the backfilling logic to the connector itself, which should work, but is not very viable when using orchestrator tools.

evantahler commented 2 years ago

cc @marcosmarxm - Perhaps theres' a way to do this now with our update api, assuming the source in use supports a start_date-like field.

m-brea commented 1 year ago

Any updates here? thanks!

asheone commented 1 year ago

Hey @airbyteio, is there any update on this issue?

evantahler commented 1 year ago

Is the API to update a source or destination's configuration insufficient for this? Does 'parameter' in your description mean the source' configuration, or are you referring to the connections state, and you would like to update its cursor (for an incremental sync) and replay some data you previously read?