openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
461 stars 74 forks source link

Scraper error status codes are confusing #1109

Open equivalentideas opened 7 years ago

equivalentideas commented 7 years ago

Scrapers sometimes error before they run (during the compile?). When this happens you get told that there's an error, and what the error code is, e.g. "Last run failed 22 minutes ago with status code 128." I've seen status codes 128 and 255. There's no clue to the person what has gone wrong or what they can do about it.

screen shot 2017-01-23 at 12 09 18 pm

A number of people have reported these errors, and being confused by the status code. See

https://github.com/openaustralia/morph/issues/1086 :

https://morph.io/tmtmtmtm/argentina-diputados-wikidata keeps failing "with status code 255". It's not obvious at all what's going on here,

https://github.com/openaustralia/morph/issues/422 :

This scraper I wrote (and try to maintain!) has flagged an error, status code 255 ... I'm not sure what that means.

https://help.morph.io/t/scrapers-failing-with-status-code-128-and-255/314/3

I've got a variety of scrapers failing - mostly with status 128 and a few with status 255. What do these error codes mean?

@henare seems to know what status code 255 is:

As far as I know status code 255 only happens on a failed compile or when you try to stop a scraper and the container has disappeared so we definitely shouldn't be seeing this.

Firstly, we should fix whatever is causing these errors, so people’s scrapers just run.

If the root cause of the errors is here to stay:

I don't know what the solution is, but these codes seem to be confusing/frustrating people (including me) in there current incarnation.