API Issues for WIND and Wave data

[ssolson]

Quick Intro

Hello myself, my team, and users of MHKiT have been having issues requesting data from the AWS S3 buckets. I believe this is the correct place to raise this issue if not could you kindly point me to the correct point of contanct?

Recap the Problem

So trying to use h5pyd or REX I am not able to get the API to return a single data point. I have documented this issue this week in two threads (https://github.com/MHKiT-Software/MHKiT-Python/issues/246, https://github.com/MHKiT-Software/MHKiT-Python/pull/241) one for Wave and one for wind.

Demonstrate the problem (MVP)

Using h5pyd and nrel-rex I will replicate the issues above:

Wind using h5pyd

import h5pyd
data_path = '/nrel/wtk/offshore_ca/Offshore_CA_2000.h5'
f = h5pyd.File(data_path, 'r')
dset = f['windspeed_100m']
dset[1, 1]

*** ValueError: cannot reshape array of size 0 into shape (1,)

Wave using h5pyd

import h5pyd
data_path = '/nrel/US_wave/West_Coast/West_Coast_wave_1995.h5'
f = h5pyd.File(data_path, 'r')
dset = f['significant_wave_height']
dset[1, 1]

ValueError: cannot reshape array of size 0 into shape (1,)

Wind using rex

from rex import WindX
data_path = '/nrel/wtk/offshore_ca/Offshore_CA_2000.h5'
with WindX(data_path , hsds=True) as rex_wind:
    ds = rex_wind['windspeed_100m', 1, 1]

ValueError: cannot reshape array of size 0 into shape (1,)

Wave using rex

from rex import WaveX
data_path = '/nrel/US_wave/West_Coast/West_Coast_wave_1995.h5'
with WaveX(data_path, hsds=True) as rex_wave:
    ds = rex_wave['significant_wave_height', 1, 1]

ValueError: cannot reshape array of size 0 into shape (1,)

Summary

In tha above calls I use python 3.9, but a user was using python 3.10. I also test this using a number odifferent versions and installations methods using GitHub actions for our CI/CD tests so this seems indpendent of py version or packages to me.

Am I doing something wrong? or is the API at fault?

[ssolson]

@PjEdwards I was just looking back at our discussion on #285.

At that time we reasoned the problem was the amount of data being requested. However, in the above examples I am only requesting a single data point from multiple endpoints.

Any ideas?

[PjEdwards]

Hi @ssolson. Ooof, it's been a minute since you asked this. Just now reviewing all open issues and, unfortunately, I don't see a problem with your requests. The data paths and datasets you're using are valid. I tested the h5pyd versions locally just now and they both worked. I haven't used rex before so I don't have an easy env to test that. I wonder if the issue is in your hsds setup for h5pyd? Maybe your configuration for the HSDS server was broken, or the server you're using was down?

[ssolson]

PJ the issue is for sure with the API. I have had more success recently but I have been hitting it less. Could you tell me anything about who is hosting or managing the API that I could talk with?

[PjEdwards]

Let me find a point of contact who can help move this along. It's the team that maintains this repo, however I believe that repo is basically unmaintained and/or monitored. I'll get back to you as soon as I find anything out.

[PjEdwards]

@ssolson While we wait for a response to my emails...

I suspect that the public HSDS service is failing intermittently due to load. The public facing HSDS service NREL maintains does not have a lot of power to support concurrent throughput. You will almost certainly find it more reliable to run your own HSDS service and reconfigure h5pyd and/or rex to use dedicated resources. I was poking around that repo and found this suggested addition to the docs that describes how to do this.

[ssolson]

PJ This is interesting. I will need to try it out but it could be something I direct people looking to get large amounts of data from the server too if it proves simple enough but is outside the scope of MHKiT generally speaking.

[PjEdwards]

@ssolson I have identified the correct party, however they are out of the office on jury duty which has an indeterminate end date. Hopefully they'll be able to respond soon.

[rolson2]

Hello @ssolson, Sorry for this late response. I've just tried the examples you've provided and they are working (following the setup instructions here: https://github.com/NREL/hsds-examples). Are you still having the same issue?