GokuMohandas / Made-With-ML

Learn how to design, develop, deploy and iterate on production-grade ML applications.
https://madewithml.com
MIT License
37.07k stars 5.88k forks source link

Ray Configuration on Mac M1 #255

Open rcontesti opened 6 months ago

rcontesti commented 6 months ago

Hi, I'm facing some issues with Ray when running it locally on my mac (M1, Sonoma 14.1.1). I wonder whether for those of us running locally, a different configuration or setup of Ray would be required.

Many thanks

2024-03-08 12:31:00,522 ERROR services.py:1222 -- Failed to start the dashboard , return code 1
2024-03-08 12:31:00,524 ERROR services.py:1247 -- Error should be written to 'dashboard.log' or 'dashboard.err'. We are printing the last 20 lines for you. See 'https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure' to find where the log file is.
2024-03-08 12:31:00,529 ERROR services.py:1291 -- 
The last 20 lines of /tmp/ray/session_2024-03-08_12-30-58_875047_6306/logs/dashboard.log (it contains the error message from the dashboard): 
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/rubencontesti/Code/madewithml/venv/lib/python3.11/site-packages/ray/dashboard/utils.py", line 134, in get_all_modules
    raise e
  File "/Users/rubencontesti/Code/madewithml/venv/lib/python3.11/site-packages/ray/dashboard/utils.py", line 121, in get_all_modules
    importlib.import_module(name)
  File "/opt/homebrew/Cellar/python@3.11/3.11.4_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1204, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/Users/rubencontesti/Code/madewithml/venv/lib/python3.11/site-packages/ray/dashboard/modules/test/test_agent.py", line 6, in <module>
    import ray.dashboard.modules.test.test_utils as test_utils
  File "/Users/rubencontesti/Code/madewithml/venv/lib/python3.11/site-packages/ray/dashboard/modules/test/test_utils.py", line 3, in <module>
    import async_timeout
ModuleNotFoundError: No module named 'async_timeout'
2024-03-08 12:31:00,654 INFO worker.py:1642 -- Started a local Ray instance.
(raylet) [2024-03-08 12:31:01,402 E 6333 155252] (raylet) agent_manager.cc:70: The raylet exited immediately because one Ray agent failed, agent_name = dashboard_agent/470211272.
(raylet) The raylet fate shares with the agent. This can happen because
(raylet) - The version of `grpcio` doesn't follow Ray's requirement. Agent can segfault with the incorrect `grpcio` version. Check the grpcio version `pip freeze | grep grpcio`.
(raylet) - The agent failed to start because of unexpected error or port conflict. Read the log `cat /tmp/ray/session_latest/logs/{dashboard_agent|runtime_env_agent}.log`. You can find the log file structure here https://docs.ray.io/en/master/ray-observability/ray-logging.html#logging-directory-structure.
(raylet) - The agent is killed by the OS (e.g., out of memory).