Automates retrieval and submission of ocean acidification data for the Center for Biological Diversity.
Currently contains scripts to
These steps must be taken before using the program. Most will only be required once.
This setup should only have to be run once per machine you run it on.
Install Docker. The project is designed to run in a Docker container. Therefore, the only prerequisite is Docker: Get Docker
Clone the repository. If you haven't already: git clone https://github.com/11th-Hour-Data-Science/cbd-ocean-acidification.git
Change to the root project directory: cd cbd-ocean-acidification
Build the Docker image: docker build --tag cbd .
Some steps in the 303(d) data submission process must be taken manually once before completion. The metadata/stations.csv
contains metadata about the stations that are available to retrieve data from. You can open this in Excel or any csv editor of your choice. Just be sure to save any changes as a csv.
Please refer to "Stations Table Schema" below for information on how to fill out columns.
Washington uses EIM to handle environmental data. It has three distinct types of data.
To set up submitting data to Washington:
stations.csv
to add the eim_study_id
to the relevant stationsstations.csv
and station_parameter_metadata.csv
For more information, visit the EIM page
CEDEN is California's primary portal for environmental data upload, but it does not accept time series data as of this writing. All time-series data must be submitted to the Integrated Report Document Upload Portal. To submit to California:
stations.csv
and station_parameter_metadata.csv
For new stations and locations:
stations.csv
with all relevant data. You may need to contact the source. station_parameter_metadata.csv
with all relevant data. You may need to contact the source. If you are submitting NERRS data:
Get your ipv4 address (going to a website like https://whatismyipaddress.com/ should do it)
Request a webservices account from NERRS: http://cdmo.baruch.sc.edu/web-services-request/
Wait for your confirmation email. Since most IP addresses change over time, you may have to do this before each time you acquire NERRS data, or get a static IP.
In the project root directory, run:
bash run_tool.sh <STATE> <start_date> <end_date>
where
Results will be saved in results/STATE/YYYY-MM-DDTHH-MM
with a README.txt
file explaining further instructions.
python main.py <STATE> --start <start_date> --end <end_date>
where
results/STATE/YYYY-MM-DDTHH-MM
with a README.txt
file explaining further instructions. ipacoa_measurement_lookup.csv
: Lookup table that shows IDs, names, and measurement units for all measurements provided by stations listed in the asset list.ipacoa_platform_measurements.csv
: Contains all combinations of stations and measurements accessible through IPACOA. The process
column indicates whether that information is going to be downloaded by the scraper ipacoa.py
. If you want to add additional stations or measurements, you should update process
column of the relevant row to be True
. Otherwise that information will not be included in the dataset. Currently only ocean acidification related measurements of West Coast stations are set to have process=True
.
king-county-keys.json
: Contains necessary keys to make a GET
request to King County's API. Used by the kingcounty.py
script.kingcounty_measurement_lookup.csv
: Contains information on measurements, units, and devices used in the stations listed by King County data portal.
stations.csv
: Table containing information on all stations that can be accessed through both IPACOA and King County data sources.You can contact us by opening an issue or emailing tspread at uchicago edu
Project Link: [https://github.com/chicago-cdac/cbd-ocean-acidification]()