Open mrhappyasthma opened 1 year ago
I wonder if reporting an issue here, on GitHub (API: Create an issue) every time a data source fails during a query would be sufficient.
This way we would avoid having to deploy a separate service for just the purpose of monitoring.
Workflow variants - below.
Pros:
Cons/costs:
This project relies on a bunch of data sources. Some combination of stockrow, MSN Money, Yahoo Finance, etc.
If any of these dependencies changes their data format or removes a page that our scraping logic relies on, things may silently fail.
To protect against this, it would be a good idea to set up some (albeit fragile) tests to verify that the basic fetching logic for each data source is working. These tests can run periodically (e.g. cron job).
While this isn't a typically good practice for software testing, since we rely on scraping these (potentially unstable) data sources, I believe this will be the best early-warning system we can make.