search

airbytehq / airbyte

The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses. Both self-hosted and Cloud-hosted.
https://airbyte.com
Other
15.48k stars 3.99k forks source link

Support True Multicast: Single Source to Multiple Destinations in a Single Connection #18129

Open GabrielGLevine opened 1 year ago

GabrielGLevine commented 1 year ago

Tell us about the problem you're trying to solve

Currently, if I need to sync the same data from a source to multiple destinations, I have to define multiple Connections. For example, I might need to sync Salesforce data to BigQuery for analysis and Postgres for application access. I can reuse the Source configuration but each Connection entails a separate request to the Source API for the same data. This results in higher load on my Airbyte cluster and an unnecessary depletion of my API quota leading to higher costs.

Describe the solution you’d like

The Airbyte protocol already supports the standard exchange of information between a Source and Destination and Sources and Destinations are run independent of one another. Provide the ability to specify multiple destinations for a single Connection and run them with a single Source instance that passes the same data to both (or N) Destinations.

Describe the alternative you’ve considered or used

Create a separate Connection for every Source/Destination combination.

Additional context

N/A

Are you willing to submit a PR?

Don't think I can.

sh4sh commented 1 year ago

@GabrielGLevine thanks for the suggestion, I've added it to the team's backlog