No results querying data from brown.slideruleearth.io

kmilo9999 commented 8 months ago

Hello,

Is there anything wrong with https://brown.slideruleearth.io ? I am using the following params, but getting no results:

params_yapc = {
    "srt": 1,
    "len": 20,
    "ats": 3,
    "res": 10,
    "dist_in_seg": False,  # if False units of len and res are in meters
    "track": 0,
    "pass_invalid": False,
    "cnf": 2,
    "cnt": 20,
    "sigma_r_max": 4,  # maximum standard deviation of photon in extend
    "maxi": 10,
    "yapc": dict(
        knn=0, win_h=6, win_x=11, min_ph=4, score=100
    ),  # use the YAPC photon classifier; these are the recommended parameters, but the results might be more specific with a smaller win_h value, or a higher score cutoff
    #   "yapc": dict(knn=0, win_h=3, win_x=11, min_ph=4, score=50),  # use the YAPC photon classifier; these are the recommended parameters, but the results might be more specific with a smaller win_h value, or a higher score cutoff
    "atl03_geo_fields": ["dem_h"],
}
gdf = icesat2.atl06p(params_yapc, resources=[ATL03_track_name])

Thank you for your help

jpswinski commented 8 months ago

I'll take a look. Can you provide the name of the granule you are processing - the one in the ATL03_track_name variable?

kmilo9999 commented 8 months ago

Hello, the value of that variable is: ATL03_20190502052058_05180312_005_01.h5

Thank you!

jpswinski commented 8 months ago

I've confirmed that the file does not get processed on the brown cluster - on the backend, it is reporting an Access Denied error response. I've also confirmed that this file is successfully processed on our public cluster. I will continue to drill down and will let you know what I find.

cpaniaguam commented 8 months ago

@jpswinski How does one access the public cluster? Doing something like

icesat2.init("slideruleearth.io")
gdf = icesat2.atl06p(params_yapc, resources=[ATL03_track_name])

still retrieves no data. Thanks!

jpswinski commented 8 months ago

I've gotten to the bottom of the issue and here is a summary of what happened and some options for moving forward. Thanks for reporting this issue, and working with us to get it resolved.

The problem

We recently updated our base AMI that we used for our production servers - this activity is part of an ongoing process of keeping up with the latest security patches and practices. The new base image defaults to only allowing the latest protocol within AWS for obtaining credentials to access things like data in S3 buckets.
The brown cluster was pinned to v4.0.4 which is an older version of our server code. This version uses the old way of obtaining AWS credentials which is not supported on the new base image.
As a result, the code was not able to access ATL03 files in S3 because it could not obtain the proper credentials for doing so.
This is a mistake on our end as I missed that your cluster was running an older version of the code, and I also missed that the new AMI didn't just use the newer protocol, but also forbid the older protocol.

Possible Solution 1 (preferred)

I updated your cluster configuration to specify using the latest server code (v4.2.3), and I verified that the problem is resolved.
You will need to update your Python client to the latest version (e.g. conda update sliderule).

Possible Solution 2

I can build a new base image for production servers that allows the old way of getting AWS credentials. This will allow you to continue using the pinned v4.0.4 for the time being.
At some point in the future, we may fully deprecate the old way of obtaining credentials, at which point you will be forced to update. I wish I could tell you when that would be, but I really don't know. Right now, using the old way is not forbidden by any security policies we must comply with, but we are always responding to new things and this is one of the things that could come up at any time.

jpswinski commented 8 months ago

@jpswinski How does one access the public cluster? Doing something like
icesat2.init("slideruleearth.io")
gdf = icesat2.atl06p(params_yapc, resources=[ATL03_track_name])
still retrieves no data. Thanks!

@cpaniaguam That looks right. When I tested it on the public cluster, it worked fine for me. If you are using a Jupyter Notebook, maybe try restarting your kernel as I wonder if the organization setting is getting cached in the client.

kmilo9999 commented 8 months ago

It works on my side again!
First options works for us as well. We can set our project to work with the latest python client 👍 Thank you !

SlideRuleEarth / sliderule