search

ray-project / ray

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.
https://ray.io
Apache License 2.0
33.08k stars 5.6k forks source link

InvalidLocationConstraint Message: The specified location-constraint is not valid for storage option #29309

Open rajexp opened 1 year ago

rajexp commented 1 year ago

What happened + What you expected to happen

I intended to use workflow in my ray cluster mode application. For that I provided the following option in the ray start command --storage s3://mybucket/mypath where the bucket is in us-east-1 region. After running the ray application, I am getting following error during head setup.

  File "/data/anaconda3/envs/openpit_monitoring/bin/ray", line 8, in <module>
    sys.exit(main())
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/scripts/scripts.py", line 2588, in main
    return cli()
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/click/core.py", line 829, in __call__
    return self.main(*args, **kwargs)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/click/core.py", line 782, in main
    rv = self.invoke(ctx)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/click/core.py", line 1259, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/click/core.py", line 610, in invoke
    return callback(*args, **kwargs)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/autoscaler/_private/cli_logger.py", line 852, in wrapper
    return f(*args, **kwargs)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/scripts/scripts.py", line 746, in start
    node = ray._private.node.Node(
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/_private/node.py", line 202, in __init__
    storage._init_storage(ray_params.storage, is_head=True)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/_private/storage.py", line 337, in _init_storage
    _init_filesystem(create_valid_file=True)
  File "/data/anaconda3/envs/openpit_monitoring/lib/python3.9/site-packages/ray/_private/storage.py", line 383, in _init_filesystem
    _filesystem.create_dir(_storage_prefix)
  File "pyarrow/_fs.pyx", line 460, in pyarrow._fs.FileSystem.create_dir
  File "pyarrow/error.pxi", line 112, in pyarrow.lib.check_status
OSError: When creating bucket 'mybucket': AWS Error [code 100]: Unable to parse ExceptionName: InvalidLocationConstraint Message: The specified location-constraint is not valid

It looks like pyarrow is not able to handle the us-east-1 bucket. Something related to this AWS Issue

Versions / Dependencies

Python: 3.9.6 Ray: 2.0.0

Reproduction script

ray start --head --port=6379 --dashboard-host 0.0.0.0 --object-manager-port=8076 --storage s3://mybucket/consumer --autoscaling-config=~/ray_bootstrap_config.yaml

Issue Severity

No response

DmitriGekhtman commented 1 year ago

Running ray start with an autoscaling config directly is not well supported. Try using the cluster launcher (ray up) instead.