⚠️ This project is no longer maintained. ⚠️ It may receive security updates, but we are no longer making major changes or improvements. EDGI no longer makes active use of this toolset and it is hard to re-deploy in other contexts.
A component of the EDGI Web Monitoring Project.
This component is intended to hold various backend tools serving different tasks:
Working and Under Active Development:
web_monitoring.db
Legacy projects that may be revisited:
Get Python 3.7 or later. If you don't have the right version, we recommend using conda or pyenv to install it. (You don't need admin privileges to install or use them, and they won't interfere with any other installations of Python already on your system.)
Install libxml2 and libxslt. (This package uses lxml, which requires your system to have the libxml2 and libxslt libraries.)
On MacOS, use Homebrew:
brew install libxml2
brew install libxslt
On Debian Linux:
apt-get install libxml2-dev libxslt-dev
On other systems, the packages might have slightly different names.
Install the package.
pip install -r requirements.txt
python setup.py develop
Copy the script .env.example
to .env
and supply any local configuration
info you need. (Only some of the package's functionality requires this.)
Apply the configuration:
source .env
See module comments and docstrings for more usage information. Also see the
command line tool wm
, which is installed with the package. For help, use
wm --help
To run the tests or build the documentation, first install the development requirements.
pip install -r requirements-dev.txt
To build the docs:
cd docs
make html
To run the tests:
python run_tests.py
Any additional arguments are passed through to py.test
.
We try to make sure the code in this repo’s main
branch is always in a stable, usable state, but occasionally coordinated functionality may be written across multiple commits. If you are depending on this package from another Python program, you may wish to install from the release
branch instead:
$ pip install git+https://github.com/edgi-govdata-archiving/web-monitoring-processing@release
You can also list the git+https:
URL above in a pip requirements file.
We usually create merge commits on the release
branch that note the PRs included in the release or any other relevant notes (e.g. Release #302 and #313.
).
This repository falls under EDGI's Code of Conduct.
This project wouldn’t exist without a lot of amazing people’s help. Thanks to the following for all their contributions! See our contributing guidelines to find out how you can help.
Contributions | Name |
---|---|
💻 ⚠️ 🚇 📖 💬 👀 | Dan Allan |
💻 | Vangelis Banos |
💻 📖 | Chaitanya Prakash Bapat |
💻 ⚠️ 🚇 📖 💬 👀 | Rob Brackett |
💻 | Stephen Buckley |
💻 📖 📋 | Ray Cha |
💻 ⚠️ | Janak Raj Chadha |
💻 | Autumn Coleman |
💻 | Luming Hao |
🤔 | Mike Hucka |
💻 | Stuart Lynn |
💻 ⚠️ | Julian Mclain |
💻 | Allan Pichardo |
📖 📋 | Matt Price |
💻 | Mike Rotondo |
📖 | Susan Tan |
💻 ⚠️ | Fotis Tsalampounis |
📖 📋 | Dawn Walker |
(For a key to the contribution emoji or more info on this format, check out “All Contributors.”)
Copyright (C) 2017-2021 Environmental Data and Governance Initiative (EDGI)
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, version 3.0.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the LICENSE
file for details.