Open jim-barlow opened 3 years ago
@jimbeepbeep amazing! are you willing to transfer your code to a connector? I can help you set up and how to do that if you like
Sure, I do have some time reserved for learning how to build connectors this week (I'm on the CDK beta), do you have any other docs you could point me towards or should that be ok to start there?
Tell us about the new integration you’d like to have
OK this is a slightly strange one, however I often seem to need to snapshot table data from public URLs and stream the content as JSON into BigQuery.
Describe the context around this new integration
Typically I need to take daily snapshots for a historical record of data which is not available via any API.
Describe the alternative you are considering or using
What are you considering doing if you don’t have this integration through Airbyte?
I have tested this code in a notebook and it works fine, and I could spin it up into a scheduled Cloud Function (triggered by PubSub with the config in the payload), however I am keen on trying to use Airbyte as my single point of data ingestion so I thought this might be an interesting lightweight connector which could enable Airbyte to be used as a (limited) very simple web table scraper:
Note that this depends on the following function:
┆Issue is synchronized with this Asana task by Unito