search

openaustralia / yinyo

A wonderfully simple API driven service to reliably execute many long running scrapers in a super scaleable way
https://yinyo.io
Apache License 2.0
6 stars 1 forks source link

Add API request to find out whether the scraper run has finished. #79

Closed mlandauer closed 4 years ago

mlandauer commented 4 years ago

This is to support people who don't want to use the callbacks or listen to the events stream. They might just decide to check back after a while. Not recommended for general use but still a useful thing to be able to do

Maybe we want to generalise this a little to return the stage that the scraper is at?

mlandauer commented 4 years ago

An alternative approach would be to generalise "/exit-status" to work even when the scraper isn't finished. It would also need to return whether the scraper has run to completion. Maybe call it "status" instead?

mlandauer commented 4 years ago

Fixed in 983dbf20ba5ff0c51a1b1261eec49f4fd06ae5f0