Ability to report non-failure issues within a source's scraper

[lazd]

Description

It should be possible to report non-failure issues within a source's scraper:

• Data is stale (i.e. 1 day old)
• Totals from source differ from summed data

Basically, anything we would log out with console.log or console.warn should be passed through some sort of reporter object so it can be surfaced in logs/dashboard.

Perhaps there could be categories of errors, a debug object could be passed, etc. Not sure, but it could look something like this:

report(report.STALE_DATA, { 
  updateDate: date,
  expectedDate: scrapeDate
});

[jzohrab]

Good thoughts, some questions:

• Should that happen with the scraper, or with some after-scrape data check?
• the scrape currently tries to load the cache for the given day. If it can't find it, it will (or should!) throw an exception

My feeling is that scrapers should not do this, for reasons of atomicity. Scrapers just scrape, and that's it. I might be missing something from your question.

In terms of "totals from source differ" - is this for the scenario where the source data says something in its sum, but literally prints data that disagrees with itself? I can see that being the case.

The app should just dump stuff to the console. For logging and analysis, we'll probably want to have some kind of standardized message/warning data given, with codes, so that the log aggregation and rollup can find all of these things. Probably logging a warning with structured logging would be sufficient.

I think we should look more into logging, for sanity during operations and log aggregation etc. I'll open another issue for that.

[jzohrab]

Added https://github.com/covidatlas/li/issues/54 for more about logging.