scrapinghub / arche

Analyze scraped data
https://arche.readthedocs.io/
MIT License
47 stars 19 forks source link
data data-analysis data-visualization jupyter pandas python3 scrapy

Arche

PyPI PyPI - Python Version GitHub Build Status Codecov Code style: black GitHub commit activity

pip install arche

Arche (pronounced Arkey) helps to verify scraped data using set of defined rules, for example:

We use it in Scrapinghub, among the other tools, to ensure quality of scraped data

Installation

Arche requires Jupyter environment, supporting both JupyterLab and Notebook UI

For JupyterLab, you will need to properly install plotly extensions

Then just pip install arche

Why

To check the quality of scraped data continuously. For example, if you scraped a website, a typical approach would be to validate the data with Arche. You can also create a schema and then set up Spidermon

Developer Setup

pipenv install --dev
pipenv shell
tox

Contribution

Any contributions are welcome! See https://github.com/scrapinghub/arche/issues if you want to take on something or suggest an improvement/report a bug.