Open anjackson opened 4 years ago
The JSON configuration of the stats pusher does not currently support responses that contain arrays. e.g. we could also query the CDX indexes directly (both crawl-time and access-time), like this:
http://cdx.api.wa.bl.uk/data-heritrix?sort=reverse&output=jsondict&limit=1&url=https://www.bl.uk/robots.txt
However, this returns an array, not a dictionary, and wa_stats can't be configured to deal with that.
wa_stats
The JSON configuration of the stats pusher does not currently support responses that contain arrays. e.g. we could also query the CDX indexes directly (both crawl-time and access-time), like this:
http://cdx.api.wa.bl.uk/data-heritrix?sort=reverse&output=jsondict&limit=1&url=https://www.bl.uk/robots.txt
However, this returns an array, not a dictionary, and
wa_statscan't be configured to deal with that.