cralwer / scraper behaviour monitoring - Githubissues

CASM-Consulting / springcrawler

Apache License 2.0

0 stars 0 forks source link

cralwer / scraper behaviour monitoring #27

Open simonwibberley opened 3 years ago

simonwibberley commented 3 years ago

We should gather and analyse behavioural statistics about the crawler and scraper.

Things to records:

# successfully imported refs (passes date and keyword)
# successfully parsed refs
# failed scraper parses (date, title, text)
# failed date parses
# accepted refs (norconex pipeline accept)
# rejected refs (norconex pipeline reject) - ignore this for the moment

From these we should calculate:

proportion of imported to parsed
proportion of parsed to accepted refs
proportion of accepted to rejected refs - ignore this for the moment

simonwibberley commented 3 years ago

Medium term: do we want to hook into JEF?