openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Optionally send events asynchronously back to the user #19

Closed mlandauer closed 5 years ago

mlandauer commented 5 years ago

These events should include things like:

On the start of the run the user provides a callback url with optional headers that can be used for authentication

mlandauer commented 5 years ago

For the callback url and header information we have two options. In both cases the url and headers are provided when the scraper is started but then either:

The first option is better from a security point of view because in the second setup the scraper author could send fake events to the clay server which could specify arbitrary urls. It's much better to trust the clay user (who creates the run) where to send the callbacks.