search

NOAA-OWP / hydrotools

Suite of tools for retrieving USGS NWIS observations and evaluating National Water Model (NWM) data.
Other
53 stars 12 forks source link

Document method to obtain NWM data from GCP #170

Closed jarq6c closed 2 years ago

jarq6c commented 2 years ago

This ticket serves as a discussion area for downloading National Water Model data from Google Cloud Platform using hydrotools.

Installation

Assuming a UNIX-like environment, execute the following commands. The primary outcome should be a Python virtual environment with the hydrotools.nwm_client[gcp] package installed from PyPI.

$ python3 -m venv env
$ source env/bin/activate
(env) $ python3 -m pip install -U pip wheel
(env) $ python3 -m pip install hydrotools.nwm_client[gcp]
(env) $ touch main.py

Basic Example

Open main.py in your preferred editor and add the following code:

# Import the NWM Client for Google Cloud Platform
from hydrotools.nwm_client import gcp as nwm
import pandas as pd

# Instantiate model data service
model_data_service = nwm.NWMDataService()

# Create a location mapping with your desired comid and 
#  fake USGS site codes
#  Note: This is a work around until we fix the USGS requirement
my_mapping = pd.DataFrame({
    "nwm_feature_id": [101, 179],
    "usgs_site_code": ["my_site_a", "my_site_b"]
}).set_index("nwm_feature_id")
model_data_service.crosswalk = my_mapping

# Retrieve forecast data
forecast_data = model_data_service.get(
    configuration = "short_range",
    reference_time = "20220126T01Z"
    )

# Look at the data
print(forecast_data.info(memory_usage='deep'))
print(forecast_data[[
    'value_time',
    'value',
    'nwm_feature_id',
    'measurement_unit',
    'usgs_site_code',
    'variable_name',
    'configuration',
    'reference_time'
]].head(100))
count_row = len(forecast_data.index)  # Gives number of rows
count_col = len(forecast_data.columns)  # Gives number of columns
print("Total Rows: " + str(count_row))
print("Total Columns: " + str(count_col))

Run the script using:

(env) $ python3 main.py

Output:

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 36 entries, 0 to 35
Data columns (total 8 columns):
 #   Column            Non-Null Count  Dtype         
---  ------            --------------  -----         
 0   reference_time    36 non-null     datetime64[ns]
 1   value_time        36 non-null     datetime64[ns]
 2   nwm_feature_id    36 non-null     int64         
 3   value             36 non-null     float32       
 4   usgs_site_code    36 non-null     category      
 5   configuration     36 non-null     category      
 6   measurement_unit  36 non-null     category      
 7   variable_name     36 non-null     category      
dtypes: category(4), datetime64[ns](2), float32(1), int64(1)
memory usage: 2.0 KB
None
            value_time  value  nwm_feature_id measurement_unit usgs_site_code variable_name configuration      reference_time
0  2022-01-26 02:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
1  2022-01-26 03:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
2  2022-01-26 04:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
3  2022-01-26 05:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
4  2022-01-26 06:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
5  2022-01-26 07:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
6  2022-01-26 08:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
7  2022-01-26 09:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
8  2022-01-26 10:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
9  2022-01-26 11:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
10 2022-01-26 12:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
11 2022-01-26 13:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
12 2022-01-26 14:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
13 2022-01-26 15:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
14 2022-01-26 16:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
15 2022-01-26 17:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
16 2022-01-26 18:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
17 2022-01-26 19:00:00   0.13             101             m3/s      my_site_a    streamflow   short_range 2022-01-26 01:00:00
18 2022-01-26 02:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
19 2022-01-26 03:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
20 2022-01-26 04:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
21 2022-01-26 05:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
22 2022-01-26 06:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
23 2022-01-26 07:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
24 2022-01-26 08:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
25 2022-01-26 09:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
26 2022-01-26 10:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
27 2022-01-26 11:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
28 2022-01-26 12:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
29 2022-01-26 13:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
30 2022-01-26 14:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
31 2022-01-26 15:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
32 2022-01-26 16:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
33 2022-01-26 17:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
34 2022-01-26 18:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
35 2022-01-26 19:00:00   0.01             179             m3/s      my_site_b    streamflow   short_range 2022-01-26 01:00:00
Total Rows: 36
Total Columns: 8
jarq6c commented 2 years ago

Added with #179