Custom valid hours for fetchnwp

[williamhobbs]

There are a few reasons that a user might want to modify the valid hours fetched for a given NWP with fetchnwp.

1. If they are only interested in a subset of valid hours (e.g., hour ahead 4 through 12 with the HRRR), they can reduce file sizes and total disk space by limiting to those valid hours.
2. If they want the most recent forecast possible for horizons shorter than max lead time of an NWP, it could be advantageous to end the fetch process as soon as the last valid hour needed is available, rather than wait for remining valid hour grib2 files to be posted to NOMADS.

I think an optional argument passed to fetchnwp could be a nice way to handle this.

This is semi-related to https://github.com/SolarArbiter/solarforecastarbiter-core/issues/686. I think the current solution for both is modifying solarforecastarbiter/io/fetch/nwp.py, and possibly creating a second virtual environment with those modifications if an unmodified fetch is also needed (e.g., if you want a "fast" short term HRRR and a regular full length HRRR).

[wholmgren]

Merits of an optional argument aside, the parameters are defined in the module dictionary, so, if I understand correctly, you could simply modify the dictionary in place at run time. For example:

import solarforecastarbiter.io.fetch.nwp

solarforecastarbiter.io.fetch.nwp.HRRR_HOURLY['valid_hr_gen'] = [1, 2, 3, 4, 5, 6]

[williamhobbs]

@wholmgren Thanks.

I've only run the fetch process from the command line interface, e.g., running something like solararbiter fetchnwp /home/nwp_directory/sfa hrrr_hourly in the terminal (using supervisor, supervisord.org, to start it and make sure it restarts if needed).

How would you recommend fetching with the modified dictionary? Would it be run inside of a Python script?

[wholmgren]

Good question. Not sure if I'd go so far as to say the following ideas are good ideas, but they are ideas...

You could modify solarforecastarbitercore.cli.fetch_nwp to accept a new argument (follow the existing click pattern), then conditionally use that argument to update the previously mentioned dictionaries. Note that you may run into trouble using development installs and command line scripts. That is, you might need to pip install . rather than pip install -e ..

Alternatively, you could copy that function and make a dedicated script that you have more control over.