search

mozilla / spade

Automated scraping markup+CSS from a list of relevant URLs, using a variety of user-agent strings. Provides reporting on usage of CSS properties and apparent user-agent sniffing.
22 stars 9 forks source link

Setup email notifications in case of failures #54

Open maurodoglio opened 11 years ago

maurodoglio commented 11 years ago

If the system falls over, or stops running, it needs to send an email to a canned list of people. Perhaps a simple daemon that is not part of django that runs on the system and checks the last time something was added to the database or that checks the error log and sends out an email if nothing has been added or if it detects an error.

maurodoglio commented 11 years ago

We can probably use http://getsentry.com to accomplish this.

maurodoglio commented 11 years ago

[scrapy-sentry](https://github.com/llonchj/scrapy-sentry scrapy-sentry) could help with this, at least to log failures in the scraper. This refers in part to issue #52