schism-dev / pyschism

Python interface for handling the SCHISM model.
https://schism-dev.github.io/schism/master/getting-started/pre-processing-with-pyschism/overview.html
Apache License 2.0
23 stars 18 forks source link

Issue downloading NWM data from Google inventory #93

Closed SorooshMani-NOAA closed 11 months ago

SorooshMani-NOAA commented 11 months ago

It seems that the request_data function tries to download the file in a directory that does not exist, causing errors: https://github.com/schism-dev/pyschism/blob/7b71ae9687a66f2d49af48844b960a1d99869dc9/pyschism/forcing/source_sink/nwm.py#L640-L641 and https://github.com/schism-dev/pyschism/blob/7b71ae9687a66f2d49af48844b960a1d99869dc9/pyschism/forcing/source_sink/nwm.py#L655

Note that the added directory in the fname variable does not exist by default and should be created for wget to successfully download the data. An easy fix would be to just remove the directory since the file name already has the date in it:

f'{self.product[:12]}.channel_rt_1.{request_time.strftime("%Y%m%d%H")}.conus.nc'

To reproduce:

import datetime
from pyschism.mesh import Hgrid
from pyschism.forcing import NWM

nwm = NWM(
    nwm_file='/path/to/nwm_v2_0_hydrofabric.gdb/', cache=True
)
nwm.write(
    output_directory='/path/to/nwm_out',
    gr3=Hgrid.open('path/to/hgrid.grd', crs=4326),
    start_date=datetime.datetime(2022, 9, 21),
    end_date=datetime.timedelta(days=5)
)

And we get:

FileNotFoundError: [Errno 2] No such file or directory: './20220921/nwm.t00z.medium_range.channel_rt_1.2022092100.conus.ncpric89y6.tmp'
SorooshMani-NOAA commented 11 months ago

I think it might be unnecessary to have the directory section of the fname when fetching the data, or if we want to keep it, we need to first make sure that directory exists. Also why not just put it in the objects tmpdir? @cuill do you have any preference?

cuill commented 11 months ago

@SorooshMani-NOAA I thought I fixed this a while ago. Probably I forgot to commit it.

We want to save the downloaded nc data in order to check the raw data if necessary or re-use it in further tests. It would be better to keep logic consistent with that in class AWSHindcastInventory: https://github.com/schism-dev/pyschism/blob/7b71ae9687a66f2d49af48844b960a1d99869dc9/pyschism/forcing/source_sink/nwm.py#L521

SorooshMani-NOAA commented 11 months ago

If you already have a commit in your local repo, can you push it please? I can test

cuill commented 11 months ago

Can you test it with this branch: https://github.com/schism-dev/pyschism/tree/bugfix/nwm

SorooshMani-NOAA commented 11 months ago

@cuill thanks, I ran the same test as above and it was successful.