MBKEngineers / collect

MBK Python scripts for scraping water data from the web
MIT License
3 stars 1 forks source link

Update CAWDL for new website data availability; pause CASGEM module #105

Open narlesky opened 11 months ago

narlesky commented 11 months ago

Related Issues

Closes #70 Addresses #98

Description

Extend collect.dwr.cawdl module for compatibility with data availability per the new data endpoints (CAWDL continuous data has migrated to https://wdlstorageaccount.blob.core.windows.net). Continuous data records encompass both surface and ground water records, so methods are refactored to be used for both data streams.

Temporarily remove collect.dwr.casgem module - collect methods are obsolete due to changes in CASGEM website structure. This will be revived when public endpoints are available via CASGEM's website.

Breaking Changes

Key changes in collect.dwr.cawdl:

Key changes in collect.dwr.casgem:

Example Usage

import datetime as dt
from pprint import pprint
from collect.dwr import cawdl

# surface water station example
result = cawdl.get_cawdl_continuous_data(
    'B05155',
    'Flow',
    'Daily_Mean',
    start=dt.datetime(2020, 1, 1),
    end=dt.datetime(2020, 2, 1)
)

print(result['info']['rating_tables'])
print(result['data'].head())

pprint(result)

# well example
result = get_cawdl_continuous_data(
    '01N04E36Q001M',
    'Groundwater_Level_Below_Ground_Surface',
    'Daily_Mean',
    start=dt.datetime(2020, 1, 1),
    end=dt.datetime(2020, 2, 1)
)

print(result['info']['published'])
print(result['info']['period_of_record_archive'])
print(result['data'].head())

pprint(result)