search

scraperwiki / code-scraper-in-browser-tool

Just like on ScraperWiki Classic; now a part of QuickCode.
https://quickcode.io
Other
38 stars 8 forks source link

un stopped by ScraperWiki, taking more than 1 hour Finished run: 2014-09-27 16:20:23+00:00 Exit code: 124 #137

Closed noelmas closed 7 years ago

noelmas commented 9 years ago

ScraperWiki appears to be stopping my scraper that I coded in the browser because it's taking more than 1 hour.

Is this meant to happen?

frabcus commented 9 years ago

Yes!

On Sun, Sep 28, 2014 at 12:58:03PM -0700, noelmas wrote:

ScraperWiki appears to be stopping my scraper that I coded in the browser because it's taking more than 1 hour.

Is this meant to happen?

Reply to this email directly or view it on GitHub: https://github.com/scraperwiki/code-scraper-in-browser-tool/issues/137

noelmas commented 9 years ago

If I want to create a scraper that will run for more than 1 hour. Can I not do it in the code in browser tool?

frabcus commented 9 years ago

That's correct.

We recommend you write long running scrapers so they are re-entrant. i.e. They can carry on from where they left off.

You probably need to do this anyway - there are lots of reasons long running scrapers can crash.

Francis

On Mon, Sep 29, 2014 at 02:09:11AM -0700, noelmas wrote:

If I want to create a scraper that will run for more than 1 hour. Can I not do it in the code in browser tool?

Reply to this email directly or view it on GitHub: https://github.com/scraperwiki/code-scraper-in-browser-tool/issues/137#issuecomment-57134518

noelmas commented 9 years ago

Thanks Francis. Makes sense, I will do that!