When attempting to initialize Ray on an EMR cluster that bootstraps the init script in this repo, I'm hitting the following error:
My goal is to utilize Ray in a Jupyter notebook that has an EMR cluster attached. I created a small cluster which bootstraps the init script in this repo. I've created a PySpark notebook, where I run
import ray
ray.init()
This then produces the following error:
An error was encountered:
Interpreter died:
ERROR:fake_shell:execute_reply
Traceback (most recent call last):
File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 263, in parse_code_into_nodes
nodes.append(NormalNode(code))
File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 215, in __init__
self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
File "<stdin>", line 1
!sh databricks_init.sh
^
SyntaxError: invalid syntax
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 309, in execute_request
nodes = parse_code_into_nodes(code)
File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 295, in parse_code_into_nodes
nodes.append(NormalNode(chunk))
File "/mnt2/yarn/usercache/livy/appcache/application_1665681676614_0004/container_1665681676614_0004_01_000001/tmp/7559468942980752460", line 215, in __init__
self.code = compile(code, '<stdin>', 'exec', ast.PyCF_ONLY_AST, 1)
File "<stdin>", line 1
!sh databricks_init.sh
^
SyntaxError: invalid syntax
[2022-10-13 20:22:07,389 E 19151 19151] core_worker.cc:137: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory
I'm not doing anything fancy with the EMR cluster, and the logs show that the bootstrapping action was without error.
When attempting to initialize Ray on an EMR cluster that bootstraps the init script in this repo, I'm hitting the following error: My goal is to utilize Ray in a Jupyter notebook that has an EMR cluster attached. I created a small cluster which bootstraps the init script in this repo. I've created a PySpark notebook, where I run
This then produces the following error:
I'm not doing anything fancy with the EMR cluster, and the logs show that the bootstrapping action was without error.