Open wvengen opened 1 year ago
Things to look at (not an exhaustive list):
Here is the scrapyd code for the API, having several error responses. https://github.com/scrapy/scrapyd/blob/master/scrapyd/webservice.py
Note that if you're working on this with a team, it can make sense to split this issue up in multiple smaller ones.
Did some tests on the original scrapyd, but error handling is not very well worked out:
listspiders
on a nonexistant project returns status 200 with content
{"node_name": "x", "status": "error", "message": "Scrapy 2.9.0 - no active project\n\nUnknown command: list\n\nUse \"scrapy\" to see available commands\n"}
I think we can follow that. We don't need to have the exact error messages as scrapyd (they can be quite cryptic).
Errors related to invalid query and form parameters have been (mostly?) fixed. You can still schedule a project with an non-existant version, which means it will be stuck in the pending state. You can still schedule a project with an non-existant spider, which means it will error immediately and be finished.
A next step could be to handle exceptions and return a JSON-based error message, instead of http 500 server error (this may break clients). Ideally, this only happens for the API endpoints.
The main config-related error (where the daemon doesn't crash before starting) is when a project repository is missing in the configuration. That would be something to improve, either on load or when it is being used.
A small number of error-cases is handled (i.e. an error API response is returned). Add a default error handler that returns what scrapyd would return on an error. And handle more cases with helpful error messages.