search

openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Add ability to stop scraper #3

Closed mlandauer closed 5 years ago

mlandauer commented 5 years ago

We can currently delete a scraper but that will get rid of its state. We want something where we can stop the run of the scraper so it will exit, then allow us to get the output, for instance, and then allow us to delete the scraper

mlandauer commented 5 years ago

Thinking about this a little more it's not entirely obvious to me that we should want to be able to stop the scraper and get access to its state, like the scraper output. If we don't need access to the state then we can simply delete the run (via the API) as per normal and everything will be stopped and cleaned up.

Let's take a look at what morph does currently...

mlandauer commented 5 years ago

With morph currently, when the user stops a scraper it asks docker to kill the container. Then when it stops, morph gets the exit code and the output. It doesn't do anything different for when the scraper succeeds versus it failing.

mlandauer commented 5 years ago

My instinct here is to follow the simplest path which is to not to grab state if you explicitly kill the scraper before it's finished. In that case we can simply use the normal API call. So, closing this.