kinverarity1 / python-sa-gwdata

Python package for the Groundwater Data section of the DEW WaterConnect website
https://python-sa-gwdata.readthedocs.io/en/latest/index.html
MIT License
3 stars 0 forks source link

Add bulk download CSV water level requests #1

Closed kinverarity1 closed 4 years ago

kinverarity1 commented 5 years ago

They don't have all the data but they're a damn sight quicker than doing individual requests.

e.g. https://www.waterconnect.sa.gov.au/_layouts/15/dfw.sharepoint.wdd/WDDDMS.ashx/GetWaterLevelDownload?bulkOutput=CSV

In the POST content the key exportdata appears to have string-encoded JSON:

{"Box":+[-35.606559,139.234534,-35.158702,140.460883],
+"DHNOs":+[83454,83495,83498,83499,83502,83504,83516,83527,83577,
83644,83646,83647,83654,83663,83664,83665,83666,83703,84741,84751,
84791,84792,84793,84795,84802,84803,84868,84878,84879,148693,167427,
171071,173662,182486,182487,190264,192545,196037,196355,200067,
201427,201465,140892,212313,232824,234615,238843,240307,102041,
205988],+"Type":+"CSV",+"Anomalous":+true,+"Pumping":+true}

August update:

The full list of bulk data service names:

kinverarity1 commented 5 years ago

Here is how to do it manually:

import io, requests
import pandas as pd

wls = requests.post(
    "GetWaterLevelDownload?bulkOutput=CSV",
    data={"exportdata": json.dumps({"DHNOs": [str(d) for d in dhnos]})},
)
with io.BytesIO(wls.response.content) as buffer:
    df = pd.read_csv(buffer)
kinverarity1 commented 4 years ago

Completed and released in v0.10.0