search

data-integrations / http

Sink plugin to send the messages from the pipeline to an external http endpoint
http://docs.cask.co/cdap
Other
1 stars 29 forks source link

Question: Using HTTP plugin to fetch multiple REST endpoints and sink data to BigQuery Multi table sink #143

Closed radzish closed 1 year ago

radzish commented 1 year ago

I would like to use Cloud Data Fusion to load data from multiple REST endpoints and store it in BigQuery in multiple tables (per endpoint).

I made it working using HTTP plugin as source and BigQuery sink. However I have to define pipeline for each endpoint, which is overkill, I suppose.

I noticed that Data Fusion has BigQuery Multi table sink, so I was expecting to connect multiple HTTP sources to it so that BigQuery create tables per each endpoint and load data into them. However when I run pipeline I am having error "Two different input schema were set for the stage BigQuery Multi Table". Apparently every endpoint has different schema.

Questions are: Is BigQuery Multi table sink appropriate to solve my problem? If yes, how should I configure it correctly to make it working?. If not, are there any other ways to do it other than defining pipeline per endpoint?

radzish commented 1 year ago

I realized that the only way to achieve what I need is to create own multi source plugin.