Closed amfriesz closed 11 months ago
ntabris
commented
11 months ago Might be helpful if you could run this and let me know what you see:
from coiled.utils import get_aws_identity
# check local identity
get_aws_identity()
# check aws identity from cluster
client.submit(coiled.utils.get_aws_identity)
mrocklin
commented
11 months ago Aaron sent this by e-mail. We weren't sure if this information was sensitive or not.
On Thu, Jul 20, 2023 at 12:07 PM Nat Tabris @.***> wrote:
Might be helpful if you could run this and let me know what you see:
from coiled.utils import get_aws_identity# check local identityget_aws_identity()# check aws identity from clusterclient.submit(coiled.utils.get_aws_identity)
— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/253#issuecomment-1644200527, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTBVUZHW3FXZ2KAVKKDXRFJUVANCNFSM6AAAAAA2RUX3GM . You are receiving this because you are subscribed to this thread.Message ID: @.***>
--
Matthew Rocklin CEO, Dask Maintainer
mrocklin
commented
11 months ago The result is that local and remote look the same, so STS appears to be working as expected.
@ntabris had the recommendation that maybe we're running the Coiled cluster in a different region, and the bucket has permissions so that it can only be read in that one region. That would certainly explain things.
Some things to do:
region= keyword to coiled.Cluster to specify the region explicitly (@amfriesz can you do this?)
shughes-uk
commented
11 months ago I think i'm -1 on changing the cluster creation region if we identify we're launching a cluster from a certain region, that's unexpected magic that might be very annoying to those who use the default region feature. A warning might be ok.
mrocklin
commented
11 months ago I think i'm -1 on changing the cluster creation region if we identify we're launching a cluster from a certain region, that's unexpected magic that might be very annoying to those who use the default region feature. A warning might be ok.
Yeah, not pushing (and I think I agree with you). Just bringing up the idea.
amfriesz
commented
11 months ago @mrocklin,
Try the region= keyword to coiled.Cluster to specify the region explicitly (@amfriesz can you do this?)
The region keyword was specified in the coiled.Cluster. It was set to us-west-2
mrocklin
commented
11 months ago In what region is your 2i2c hub running?
On Thu, Jul 20, 2023, 4:15 PM Aaron Friesz @.***> wrote:
@mrocklin https://github.com/mrocklin,
Try the region= keyword to coiled.Cluster to specify the region explicitly @.*** https://github.com/amfriesz can you do this?)
The region keyword was specified in the coiled.Cluster. It was set to us-west-2
— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/253#issuecomment-1644541467, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTES73PRB7J5QV6ERETXRGGXJANCNFSM6AAAAAA2RUX3GM . You are receiving this because you were mentioned.Message ID: @.***>
amfriesz
commented
11 months ago The 2i2c hub is in us-west-2. So is the bucket we are trying to write to.
shughes-uk
commented
11 months ago Can you try defining your s3 filesystem like this instead?
s3 = s3fs.S3FileSystem(anon=False, client_kwargs={'endpoint_url':'https://s3-us-west-2.amazonaws.com/'})I'm slightly stumped at the moment.
I did just remember that https://github.com/coiled/benchmarks/issues/207 showed the same error message, so it might be worth trying to pass region_name to s3fs.
amfriesz
commented
11 months ago I'm now having a problem starting the cluster.
import coiled
cluster = coiled.Cluster(
n_workers=10,
region='us-west-2',
account='nasa-openscapes',
name="play",
shutdown_on_close=False,
)
client = cluster.get_client()The above code gives me the Traceback below
amfriesz
commented
11 months ago Okay, not sure why the cluster wasn't starting, but it's working now.
@shughes-uk ,
Your recommendation worked! I was able to write outputs to the s3 bucket after adding client_kwargs={'endpoint_url':'https://s3-us-west-2.amazonaws.com/'} to the s3 filesystem setup.
mrocklin
commented
11 months ago I'd love to figure out what was going on here if anyone has any ideas. This is an odd setup, but one that we're likely to run into I think with other NASA adjacent groups.
shughes-uk
commented
11 months ago Some weirdness with S3FS and the way it interacts with dask + remote credential setups I think?
dchudz
commented
11 months ago I'm trying to understand why it would work locally but not the cluster (until we specified that argument).
My only guess is that somehow locally the endpoint_url was configured as a default, but then it wasn't on the cluster.
https://docs.aws.amazon.com/sdkref/latest/guide/feature-ss-endpoints.html has some info about how to configure these as defaults.
It'd be nice to confirm whether JupyterHub had this stuff explicitly configured.
(Maybe we'd want to mirror this sort of config to the cluster.)
mrocklin
commented
11 months ago @amfriesz are you able to connect the right person here from the JHub team? We're a little curious if there was some custom configuration set up somewhere.
amfriesz
commented
11 months ago @mrocklin, I've reached out to some contacts. We'll see if they have any insights.
amfriesz
commented
11 months ago Looks like all the hub config files are publicly available in https://github.com/2i2c-org/infrastructure/tree/master/config/clusters/openscapes
There's also documentation on https://infrastructure.2i2c.org/ on how to interpret that config.
mrocklin
commented
11 months ago I'm curious, who are you asking these questions of. Is this Yuvi? Chris? James? Someone else? (I'm happy to ping them here directly) Ideally I'd avoid sifting through a bunch of yaml. My hope is that they'll say either:
amfriesz
commented
11 months ago Yuvi pointed me to the resources.
@yuvipanda, would you have insights into @mrocklin's question above, or is there someone else we should reach out to regarding the Openscapes 2i2c setup?
mrocklin
commented
11 months ago Hi @yuvipanda 👋
If you have a moment and can confirm if y'all have done anything to configure S3 access as David mentions above (https://github.com/coiled/feedback/issues/253#issuecomment-1646112379) that would be helpful. We're trying to track down some odd behavior and would like to exclude JHub if we can.
dchudz
commented
11 months ago Hi @amfriesz -- we have a guess about how we could have done the right thing automatically. If you're up for it, there a couple steps you could do to help us confirm that guess:
import boto3; boto3.session.Session().region_name. We're guessing that this is us-west-2.environ={"AWS_DEFAULT_REGION": "us-west-2"} to the coiled.Cluster(...) call. Then see if your original code (before we suggested client_kwargs={'endpoint_url' ...) works.If our guess is right then it should be easy for us to change Coiled so it detects the local default region, and mirror that to the cluster so that all of this would have worked the first time.
amfriesz
commented
11 months ago @dchudz ,
That seems to work! Running boto3.session.Session().region_name does indeed give me us-west-2. I was able to write output files to our S3 bucket after adding that new argument to the cluster and running the code without the client_kwargs in the s3fs.S3FileSystem.
dchudz
commented
11 months ago Nice! Thanks so much for testing that.
Should be an easy fix to get Coiled to do this automatically.
On Wed, Jul 26, 2023, 1:39 PM Aaron Friesz @.***> wrote:
@dchudz https://github.com/dchudz ,
That seems to work! Running boto3.session.Session().region_name does indeed give me us-west-2. I was able to write output files to our S3 bucket after adding that new argument to the cluster and running the code without the client_kwargs in the s3fs.S3FileSystem.
— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/253#issuecomment-1652239218, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAJKQRUGTDFG523RQJNC3B3XSFI47ANCNFSM6AAAAAA2RUX3GM . You are receiving this because you were mentioned.Message ID: @.***>
ntabris
commented
11 months ago Hi, @amfriesz. In coiled=0.9.0 (release this morning), the AWS_DEFAULT_REGION env var on the cluster will automatically be set based on your local session, so you shouldn't need the workaround anymore.
(More specifically, if you have local AWS credentials, Coiled by default ships an STS token so that the cluster can access the same AWS resources—e.g., S3—that you can access locally. Now when that happens, it also sets AWS_DEFAULT_REGION from your local default region.)
mrocklin
commented
11 months ago Thanks Nat.
On Thu, Jul 27, 2023 at 9:39 AM Nat Tabris @.***> wrote:
Closed #253 https://github.com/coiled/feedback/issues/253 as completed.
— Reply to this email directly, view it on GitHub https://github.com/coiled/feedback/issues/253#event-9936750690, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACKZTBFD5HNTBWXHN62Z23XSJ4SJANCNFSM6AAAAAA2RUX3GM . You are receiving this because you were mentioned.Message ID: @.***>
--
Matthew Rocklin CEO, Dask Maintainer
(this is Matt on Aaron's computer)
We're running into an issue where we're unable to access S3 data on a Coiled worker. We are able to access that data locally. This is running on a 2I2C openscapes deployment.
https://cloud.coiled.io/clusters/245329/overview?account=nasa-openscapes
We're doing something like the following:
If we use Dask + Coiled we fail with a permission error, which makes me think that the STS forwarding isn't working (although I'm not seeing anything bad in logs). Maybe permissions in a 2I2C environment are different somehow? He's also put in his credentials in his
.awsdirectory.Traceback
```python-traceback import xarray as xr import dask import ujson import s3fs import fsspec from glob import glob import pathlib import hvplot.xarray import coiled from kerchunk.hdf import SingleHdf5ToZarr from kerchunk.combine import MultiZarrToZarr # The xarray produced from the reference file throws a SerializationWarning for each variable. Will need to explore why import warnings warnings.simplefilter("ignore") Start up a Dask Client to monitor Dask processing import coiled cluster = coiled.Cluster( n_workers=10, region='us-west-2', account='nasa-openscapes', name="play", shutdown_on_close=False, ) client = cluster.get_client() ╭────────────────────────────────────────── Package Sync ──────────────────────────────────────────╮ │ Fetching latest package priorities ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │ │ Scanning 477 conda packages ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │ │ Scanning 305 python packages ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │ │ Running pip check ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:02 │ │ Validating environment ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:02 │ │ Requesting package sync build ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 0:00:00 │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ jupyterlab-server 2.22.1 has requirement requests>=2.28, but you have requests 2.27.1. Package - jupyterlab-server, Pip check had the following issues that need resolving: jupyterlab-server 2.22.1 has requirement requests>=2.28, but you have requests 2.27.1. ╭──────────────────────────────────── Not Synced with Cluster ─────────────────────────────────────╮ │ ╷ ╷ │ │ Package │ Error │ Risk │ │ ╶───────────────────┼────────────────────────────────────────────────────────────────┼─────────╴ │ │ jupyterlab-server │ Pip check had the following issues that need resolving: │ Warning │ │ │ jupyterlab-server 2.22.1 has requirement requests>=2.28, but │ │ │ │ you have requests 2.27.1. │ │ │ ╵ ╵ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭───────────────────────────────────────── Coiled Cluster ─────────────────────────────────────────╮ │ https://cloud.coiled.io/clusters/245329/overview?account=nasa-openscapes │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ ╭─────────────────── Overview ───────────────────╮╭──────────────── Configuration ─────────────────╮ │ ││ │ │ Cluster Name: nasa-openscapes-84441220-a ││ Region: us-west-2 │ │ ││ │ │ Scheduler Status: started ││ Scheduler Instance Type: m6i.xlarge │ │ ││ │ │ Dashboard Address: ││ Worker Instance Type(s): m6i.xlarge (10) │ │ [https://cluster-zddwe.dask.host?token=y6qr0LME](https://cluster-zddwe.dask.host/?token=y6qr0LME_3ZHXMBZ) ││ │ │ [_3ZHXMBZ](https://cluster-zddwe.dask.host/?token=y6qr0LME_3ZHXMBZ) ││ Workers Requested: 10 │ │ ││ │ ╰────────────────────────────────────────────────╯╰────────────────────────────────────────────────╯ ╭─────────────────────────────────── (2023/07/20 15:47:55 UTC) ────────────────────────────────────╮ │ │ │ Most of your workers have arrived. Cluster ready for use. │ │ │ │ │ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯ #from dask.distributed import Client #client = Client(n_workers=4) #client Identify the S3 bucket bucket = 's3://npwbanalres' Connect to the S3 bucket and list files it contains s3 = s3fs.S3FileSystem(anon=False) #s3.ls(bucket) urls_all = s3.glob(f'{bucket}/*.nc4') urls_all rain_urls = s3.glob(f'{bucket}/*rain.nc4') aet_urls = s3.glob(f'{bucket}/*aet.nc4') Create Kerchunk file for each data asset/.nc4 file def gen_json(u): so = dict( mode= "rb", anon= False, default_fill_cache= False, default_cache_type= "none" ) with s3.open(u, **so) as infile: h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300) with s3.open(f"s3://openscapes-cloud-experiments/{u.split('/')[-1]}.json", 'wb') as outf: # write to s3 outf.write(ujson.dumps(h5chunks.translate()).encode()) # def gen_json(u): # so = dict( # mode= "rb", # anon= False, # default_fill_cache= False, # default_cache_type= "none" # ) # with s3.open(u, **so) as infile: # h5chunks = SingleHdf5ToZarr(infile, u, inline_threshold=300) # with open(f"jsons/{u.split('/')[-1]}.json", 'wb') as outf: # outf.write(ujson.dumps(h5chunks.translate()).encode()) Create a json directory to write Kurchunk files to #pathlib.Path('./jsons/').mkdir(exist_ok=True) Use Dask to process multiple files in parallel reference_files = [] for url in aet_urls: ref = dask.delayed(gen_json)(url) reference_files.append(ref) reference_files_compute = dask.compute(*reference_files) --------------------------------------------------------------------------- ClientError Traceback (most recent call last) File /opt/coiled/env/lib/python3.9/site-packages/s3fs/core.py:113, in _error_wrapper() File /opt/coiled/env/lib/python3.9/site-packages/aiobotocore/client.py:371, in _make_api_call() ClientError: An error occurred (AccessDenied) when calling the PutObject operation: No AWSAccessKey was presented. The above exception was the direct cause of the following exception: PermissionError Traceback (most recent call last) Cell In[12], line 6 3 ref = dask.delayed(gen_json)(url) 4 reference_files.append(ref) ----> 6 reference_files_compute = dask.compute(*reference_files) File /srv/conda/envs/notebook/lib/python3.9/site-packages/dask/base.py:599, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs) 596 keys.append(x.__dask_keys__()) 597 postcomputes.append(x.__dask_postcompute__()) --> 599 results = schedule(dsk, keys, **kwargs) 600 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)]) File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:3224, in Client.get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs) 3222 should_rejoin = False 3223 try: -> 3224 results = self.gather(packed, asynchronous=asynchronous, direct=direct) 3225 finally: 3226 for f in futures.values(): File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/client.py:2359, in Client.gather(self, futures, errors, direct, asynchronous) 2357 else: 2358 local_worker = None -> 2359 return self.sync( 2360 self._gather, 2361 futures, 2362 errors=errors, 2363 direct=direct, 2364 local_worker=local_worker, 2365 asynchronous=asynchronous, 2366 ) File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:351, in SyncMethodMixin.sync(self, func, asynchronous, callback_timeout, *args, **kwargs) 349 return future 350 else: --> 351 return sync( 352 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs 353 ) File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:418, in sync(loop, func, callback_timeout, *args, **kwargs) 416 if error: 417 typ, exc, tb = error --> 418 raise exc.with_traceback(tb) 419 else: 420 return result File /srv/conda/envs/notebook/lib/python3.9/site-packages/distributed/utils.py:391, in sync.