Example Code for HSDS for production runs of reV

[Zoe-Fehlau-ES]

examples/running_with_hsds

Please note that our HSDS service is for demonstration purposes only, if you would like to use HSDS for production runs of reV please setup your own service: https://github.com/HDFGroup/hsds and point it to our public HSDS bucket: s3://nrel-pds-hsds

I am currently trying to setup my own service as described in the link (any help there would be appreciated). But I was wondering if there was some example code of how using my own service is different than the code below:

nsrdb_file = '/nrel/nsrdb/v3/nsrdb_2013.h5'
with rex.Resource(nsrdb_file, hsds=True) as f:
    meta_data = f.meta
    time_index = f.time_index
import os
import numpy as np
from reV import TESTDATADIR
from reV.config.project_points import ProjectPoints
from reV.generation.generation import Gen
from rex import init_logger
init_logger('reV', log_level='DEBUG')
lat_lons = np.array([[ 41.25, -71.66],
                     [ 41.05, -71.74],
                     [ 41.45, -71.66],
                     [ 41.97, -71.78],
                     [ 41.65, -71.74],
                     [ 41.53, -71.7 ],
                     [ 41.25, -71.7 ],
                     [ 41.05, -71.78],
                     [ 42.01, -71.74],
                     [ 41.45, -71.78]])
res_file = '/nrel/wtk/conus/wtk_conus_2012.h5'  # HSDS 'file' path
sam_file = os.path.join(TESTDATADIR,
                         'SAM/wind_gen_standard_losses_0.json')
pp = ProjectPoints.lat_lon_coords(lat_lons, res_file, sam_file)
gen = Gen.reV_run('windpower', pp, sam_file, res_file, max_workers=1,
                  out_fpath=None, output_request=('cf_mean', 'cf_profile'))
print(gen.out['cf_profile'])

[grantbuster]

I just added some useful links to the example you're using:

You might also be interested in these examples of how to set up your own local HSDS server and how to run reV on an AWS parallel cluster.

HSDS is a behind the scenes data service that streams data from AWS S3 to your local machine. "Using your own service" would not result in any changes to the code you pasted. It only changes what happens behind the scenes.

[Zoe-Fehlau-ES]

Thank you for explaining that and adding that line. What is the benefit of setting up your own local HSDS server and how to run reV on an AWS parallel cluster then? Is it just helpful for large and numerous reV jobs?

[grantbuster]

At this point i'd suggest reading through those examples...

If you're streaming a lot of data via HSDS you'll need to stand up your own HSDS server. The public NREL API won't support large volumes of data and you'll start getting errors.

If you're analyzing a lot of locations for many years (we typically do millions of locations for ~10 years), you can't do this compute on a laptop. AWS parallel cluster is a super computer in the cloud that looks just like our HPC at NREL and allows you to do large scale studies.

[Zoe-Fehlau-ES]

Thank you for answering my simple questions. You have been a lot of help.

[grantbuster]

No problem! reV can be confusing. We're working on improving the auto-docs right now but it's a big effort.

[Zoe-Fehlau-ES]

@grantbuster Sorry to bug you again but I am confused if I am following the Setting up a Local HSDS Server correctly. When I get to step 9 hsinfo, I am getting something that looks different than the instructions but I have redone them three times. I get: Vs the instructions get:

The server names are different and my endpoint should be: http://localhost:5101, right?

[grantbuster]

1) did you include the --no-docker flag? 2) I think your HSDS config is still set to the NREL API based on the hsinfo printed endpoint. Make sure everything is set up in .hscfg and make sure your shell recognizes it (you may need to restart your shell or run a source on your rc files)

[Zoe-Fehlau-ES]

I ran runall.bat --no-docker and changed my .hscfg to the following:

But now I am getting: username/password not valid for username: zfehlau@energystrat.com

[grantbuster]

Please follow the instructions EXACTLY. The only thing that should be in your hscfg is one line: hs_endpoint = http+unix://%2Ftmp%2Fhs%2Fsn_1.sock

[Zoe-Fehlau-ES]

Oh I have it working now. Sorry, I interpreted step 11 like step 6 where I just update hs_endpoint. Thank you for the help and sorry for my silly mistakes