search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
94 stars 68 forks source link

feat: support for webhook collection #1620

Open tayloramurphy opened 1 year ago

tayloramurphy commented 1 year ago

Feature scope

Taps (catalog, state, stream maps, etc.)

Description

I'm not sure how this would exactly work, but there's a lot of data sources that can send webhooks for data updates.

Particularly for Cloud it could be interesting to be able to configure a tap to accept webhooks and have it as a long-running end point. But in terms of the SDK we could build some nice helpers with that.

I see this as something to think about for the longer term (late 2023 / early 2024)

stale[bot] commented 1 year ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

tayloramurphy commented 1 year ago

Still very relevant!

stale[bot] commented 4 weeks ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.

tayloramurphy commented 3 weeks ago

@edgarrmondragon thoughts on this?

edgarrmondragon commented 1 week ago

@edgarrmondragon thoughts on this?

I like the idea of having a helper class for processing webhooks, roughly

class WebhookStream(Stream):
    def process_payload(self, payload):
       ...

the thing I'm not sure of is where would we output these messages to? Maybe a Singer queue/stream on Kinesis or similar, and a target on the other end reading messages?

Sounds like an interesting hackday project :D