aws-samples / aws-samples-for-ray

MIT No Attribution
64 stars 23 forks source link

Ray Error when running ray.init() on a Jupyter Notebook with an EMR cluster attached #4

Closed egelberg closed 2 years ago

egelberg commented 2 years ago

When attempting to initialize Ray on an EMR cluster that bootstraps the init script in this repo, I'm hitting the following error: My goal is to utilize Ray in a Jupyter notebook that has an EMR cluster attached. I created a small cluster which bootstraps the init script in this repo. I've created a PySpark notebook, where I run

import ray
ray.init()

This then produces the following error:

An error was encountered:
Interpreter died:

ERROR:fake_shell:execute_reply
Traceback (most recent call last):
  File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 263, in parse_code_into_nodes
    nodes.append(NormalNode(code))
  File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 215, in __init__
    self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
  File "<stdin>", line 1
    !sh databricks_init.sh
    ^
SyntaxError: invalid syntax

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 309, in execute_request
    nodes = parse_code_into_nodes(code)
  File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 295, in parse_code_into_nodes
    nodes.append(NormalNode(chunk))
  File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 215, in __init__
    self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
  File "<stdin>", line 1
    !sh databricks_init.sh
    ^
SyntaxError: invalid syntax
[2022-10-13 20:22:07,389 E 19151 19151] core_worker.cc:137: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory

I'm not doing anything fancy with the EMR cluster, and the logs show that the bootstrapping action was without error.