Processes for sourcing data for the Stop COVID-19 SF Bay Area Pandemic Dashboard. You can find the dashboard’s source code in the sfbrigade/stop-covid19-sfbayarea project on GitHub.
We are looking for feedback! Did you come here looking for a data API? Do you have questions, comments, or concerns? Don't leave yet - let us know how you are using this project and what you'd like to see implemented. Please leave us your two cents over in Issues under #101 Feedback Mega Thread.
This project requires Python 3 to run. It was built specifically with version 3.8.6, but it may run with other versions. However, it does take advantage of assignment expressions which are only available in 3.8+.
To install this project, you can simply run ./install.sh in your terminal. This will set up the virtual environment and install all of the dependencies from requirements.txt and requirements-dev.txt. However, it will not keep the virtual environment running when the script ends. If you want to stay in the virtual environment, you will have to run source env/bin/activate separately from the install script.
This project includes four separate scraping tools for different purposes:
You can also run each of these tools in Docker. See the “using Docker” section below.
The Legacy CDS Scraper loads Bay Area county data from the Corona Data Scraper project. Run it by typing into your terminal:
$ ./run_scraper.shThis takes care of activating the virtual environment and running the actual Python scraping script. If you are managing your virtual environments separately, you can run the Python script directly with:
$ python3 scraper.pyThe newer county website scraper loads data directly from county data portals or by scraping counties’ public health websites. Running the shell script wrapper will take care of activating the virtual environment for you, or you can run the Python script directly:
# Run the wrapper:
$ ./run_scraper_data.sh
# Or run the script directly if you are managing virtual environments youself:
$ python3 scraper_data.pyBy default, it will output a JSON object with data for all currently supported counties. Use the --help option to see a information about additional arguments (the same options also work when running the Python script directly):
$ ./run_scraper_data.sh --help
Usage: scraper_data.py [OPTIONS] [COUNTY]...
Create a .json with data for one or more counties. Supported counties:
alameda, san_francisco, solano.
Options:
--output PATH write output file to this directory
--help Show this message and exit.To scrape a specific county or counties, list the counties you want. For example, to scraper only Alameda and Solano counties:
$ ./run_scraper_data.sh alameda solano--output specifies a file to write to instead of your terminal’s STDOUT.
The news scraper finds official county news, press releases, etc. relevant to COVID-19 and formats it as news feeds. Running the shell script wrapper will take care of activating the virtual environment for you, or you can run the Python script directly:
# Run the wrapper:
$ ./run_scraper_news.sh
# Or run the script directly if you are managing virtual environments youself:
$ python3 scraper_news.pyBy default, it will output a series of JSON Feed-formatted JSON objects — one for each county. Use the --help option to see a information about additional arguments (the same options also work when running the Python script directly):
$ ./run_scraper_news.sh --help
Usage: scraper_news.py [OPTIONS] [COUNTY]...
Create a news feed for one or more counties. Supported counties: alameda,
contra_costa, marin, napa, san_francisco, san_mateo, santa_clara, solano,
sonoma.
Options:
--from CLI_DATE Only include news items newer than this
date. Instead of a date, you can specify a
number of days ago, e.g. "14" for 2 weeks
ago.
--format [json_feed|json_simple|rss]
--output PATH write output file(s) to this directory
--help Show this message and exit.To scrape a specific county or counties, list the counties you want. For example, to scrape only Alameda and Solano counties:
$ ./run_scraper_data.sh alameda solano--from sets the earliest date/time from which to include news items. It can be a date, like 2020-07-15, a specific time, like 2020-07-15T10:00:00 for 10 am on July 15th, or a number of days before the current time, like 14 for the last 2 weeks.
--format sets the output format. Acceptable values are: json_feed (see the JSON Feed spec), json_simple (a simplified JSON format), or rss (RSS v2). Specify this option multiple times to output multiple formats, e.g:
$ ./run_scraper_data.sh --format rss --format json_feed--output specifies a directory to write to instead of your terminal’s STDOUT. Each county and --format combination will create a separate file in the directory. If the directory does not exist, it will be created.
The hospitalization data scraper pulls down COVID-19-related hospitalization statistics at the county level from the California Department of Public Health via its CKAN API. To run the scraper, execute the following command in your terminal:
$ ./run_scraper_hospital.shBy default, this will print time-series data in JSON format to stdout for all nine Bay Area counties, following the structure described in the data model documentation.
Data for all California counties is also available; to select a specific county or list of counties, add them as arguments when running the script. The county should be spelled in lowercase, with underscores replacing spaces:
$ ./run_scraper_hospital.sh alameda los_angeles mendocinoYou may also pass an --output flag followed by the path to the directory where you would like the JSON data to be saved. If the directory does not exist, it will be created. The data will be saved as hospital_data.json.
As an alternative to installing and running the tools normally, you can use Docker to install and run them. This is especially helpful on Windows, where setting up Selenium and other Linux tools the scraper can be complicated.
Download and install Docker from https://www.docker.com/ (You’ll probably need to create a Docker account as well if you don’t already have one.)
Now run any of the tools by adding their command after ./run_docker.sh. For example, to run the news scraper:
$ ./run_docker.sh python scraper_data.pyUnder the hood, this builds the Docker container and then runs the specified command in it.
Docker acts kind of like a virtual machine, and you can also simply get yourself a command prompt inside the docker container by running ./run_docker.sh with no arguments:
$ ./run_docker.sh
# This will output information about the build, and then give you a
# command prompt:
root@ca87fa64d822:/app#
# You can now run commands like the data scraper as normal from the prompt:
root@ca87fa64d822:/app# python scraper_data.py
root@ca87fa64d822:/app# python scraper_news.pyThe data models are in JSON format and are located in the data_models directory. For more information, see the data model readme.
We use CircleCI to lint the code and run tests in this repository, but you can (and should!) also run tests locally.
The commands described below should all be run from within the virtual environment you’ve created for this project. If you used install.sh to get set up, you’ll need to activate your virtual environment before running them with the command:
$ source env/bin/activateIf you manage your environments differently (e.g. with Conda or Pyenv-Virtualenv), use whatever method you normally do to set up your environment.
You can run tests using pytest like so:
# In the root directory of the project:
$ python -m pytest -v .Some tests run against live websites and can be slow (or worse; they might spam a county's server with requests and get your IP blocked), so they are disabled by default. To run them, set the LIVE_TESTS environment variable. It can be '*' to run live tests against all counties, or a comma separated list of counties to test.
# Run live tests against all county websites.
$ LIVE_TESTS='*' python -m pytest -v .
# Run live tests against only San Francisco and Sonoma counties.
$ LIVE_TESTS='san_francisco,sonoma' python -m pytest -v .We use Pyflakes for linting. Many editors have support for running it while you type (either built-in or via a plugin), but you can also run it directly from the command line:
# In the root directory of the project:
$ pyflakes .We also use type annotations throughout the project. To check their validity with Mypy, run:
# In the root directory of the project:
$ mypy .PRs that are hotfixes do not require review.
PRs that add functionality/features require at least 1 passing review.
PRs that update the documentation require at least 1 passing review.
Reviewers
Everyone can review #1 hotfixes, or #3 documentation. If you want to proactively sign up to be first-string for these reviews, please add your github handle to the list below.
Experienced developers with deep knowledge of the project should be tapped for PRs that deal with complicated dependencies, language-specific implementation questions, structural/architectural concerns. If you want to be first-string for these reviews, please add your github handle to the list below.
People who have interest in data, public health, and social science should be tapped for PRs that deal with decisions that affect how data is reported, structured, and provided to the user. If you want to be first-string for these reviews, please list your github name below.