Deprecate/rewrite Data Quality Report

scrapinghub / arche

Analyze scraped data

https://arche.readthedocs.io/

MIT License

47 stars 19 forks source link

Deprecate/rewrite Data Quality Report #154

Open manycoding opened 5 years ago

manycoding commented 5 years ago

Current DQR https://arche.readthedocs.io/en/latest/nbs/DQR.html is:

scores based on schema validation and some rules\stats https://github.com/scrapinghub/arche/blob/master/src/arche/quality_estimation_algorithm.py
table of job stats
some rules summary
coverage graph (same as in the main report)
categories tables

I don't think it provides accurate information for developers. My idea is to remove it completely, focusing on the main report #119 #164

manycoding commented 4 years ago

Search find . -type f -mtime -90 -name *.ipynb -print -exec grep -l 'data_quality_report' {} \; shows that data_quality_report hasn't been used for 3 months.