Open Whakamua opened 4 years ago
Hi @JorDikk, thanks for raising this issue.
The Tune tab depends on the following dependencies:
pip install pandas tensorboard tabulate
The usual recommendation is to install pip install "ray[tune]"
even after installing the latest wheels, as this will install dependencies such as pandas and tabulate. However, it currently does not install tensorboard, as it is tensorflow specific. So the Tune tab still wouldn't show.
cc @simon-mo can we make the dependency on tensorboard optional and just not show a tensorboard tab/display a note if it is not found?
Thanks @krficke, doing the steps you mentioned works!
I couldn't find this information on the docs, maybe I missed it. Otherwise I think it's useful to add it to the Installing ray or Ray Dashboard page.
Yeah, updating the docs is definitely a good idea. I'd argue that we want to have a Tune
tab in any case, but if the dependencies are not found, display a message with these install commands instead.
@krfricke that's a great idea. I'm going to add a task to do that next sprint. Thanks for the suggestion.
Hey @edoakes I don't think this is P1. I will modify it to P2
I have made the tune tab work for a while. However, it disappeared now. I think it's due to my failure of installing tensorflow and some packages broken. Would you please give me some hints on finding out what is missing? I don't know how to debug this since no error or even warnings are given, the tune tab just disappeared.
Sorry this sounds bad. Did you get to find any logs from /tmp/ray/session_latest/logs/dashboard.log
?
2021-02-02 17:34:23,575 INFO head.py:48 -- Dashboard head grpc address: 114.212.23.25:33921
2021-02-02 17:34:23,575 INFO head.py:50 -- Dashboard head http address: 127.0.0.1:8265
2021-02-02 17:34:23,575 INFO dashboard.py:82 -- Setup static dir for dashboard: /home/yangjq/anaconda3/envs/hmm/lib/python3.8/site-packages/ray/new_dashboard/client/build
2021-02-02 17:34:23,578 INFO head.py:161 -- Connect to GCS at b'114.212.23.25:38691'
2021-02-02 17:34:23,579 INFO utils.py:201 -- Get all modules by type: DashboardHeadModule
2021-02-02 17:34:23,835 INFO head.py:129 -- Loading DashboardHeadModule: <class 'ray.new_dashboard.modules.log.log_head.LogHead'>
2021-02-02 17:34:23,835 INFO head.py:129 -- Loading DashboardHeadModule: <class 'ray.new_dashboard.modules.logical_view.logical_view_head.LogicalViewHead'>
2021-02-02 17:34:23,836 INFO head.py:129 -- Loading DashboardHeadModule: <class 'ray.new_dashboard.modules.reporter.reporter_head.ReportHead'>
2021-02-02 17:34:23,836 INFO head.py:129 -- Loading DashboardHeadModule: <class 'ray.new_dashboard.modules.stats_collector.stats_collector_head.StatsCollector'>
2021-02-02 17:34:23,836 INFO head.py:129 -- Loading DashboardHeadModule: <class 'ray.new_dashboard.modules.tune.tune_head.TuneController'>
2021-02-02 17:34:23,836 INFO head.py:134 -- Loaded 5 modules.
2021-02-02 17:34:23,837 INFO head.py:207 -- <ResourceRoute [GET]
I still have it with ray 1.8.0 have both ray[default] and ray[tune], but no tune tab in the dashboard.
I didn't have a tune tab either, even though I installed all the dependency with pip install "ray[tune]"
. Though looking at the log /tmp/ray/session_latest/logs/dashboard.log
, the dashboard wasn't finding tensorboard
. It was installed with the above command though. I installed it independently with conda and the tune tab now appears in the dashboard.
@krfricke Is the tune tab supposed to only show on the legacy dashboard?
@peterghaddad @p-enel @dvirginz @ThyrixYang @Whakamua In Ray 2.0, we changed the experimental dashboard to be the default dashboard. We plan to deprecate the old dashboard in the future. In the new dashboard, we currently don't support the tune tab / tune results table.
For better feature planning for the new dashboard, I have some questions:
@scottsun94 I think the tune tab is a useful feature since we can monitor the score and parameters conveniently without more coding. Tensorboard and ML flow are powerful tools, but we need to add more tensorboard logs manually, distributed logging is also problematic for tensorboard.
It would be nice if ray can write a tensorboard log and start a local TB server automatically. In such case the tune tab is unnecessary and the power of tensorboard is utilized.
@ThyrixYang Thanks for the feedback. Added the request to our backlog.
It's been a while, just want to check what the status is for bringing back some basic monitoring functionalities ray tune runs on the dashboard. Staring at the table in stdout has been a bit frustrating.
This is not planned in the next 3 months. Need @matthewdeng to triage and prioritize.
This is not prioritized right now.
Can people share their feedback on what functionalities they would find helpful to expose in the dashboard, and what's lacking in the console output? Thanks!
What is the problem?
Ray Dashboard does not always display the Tune tab. And it seems random when it does display. The first time that I used Ray the Tune tab worked fine(this was about a week ago), but from the day after until now the tab is not showing. Except for yesterday, I did
ray.shutdown()
and thenray.init()
again, and suddenly the tune tab was there(the web url went up to http://127.0.0.1:8266 from http://127.0.0.1:8265). The problem now was that when I was putting in the path to the tuning results nothing happened when pressing submit. When I now doray.shutdown()
and thenray.init()
again, this doesn't work. (the web url also doesn't go up)I am running MacOS 10.15.5 with python 3.7.7
Reproduction (REQUIRED)
I used a new pyenv environment and did the following:
pip install -U https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-1.1.0.dev0-cp37-cp37m-macosx_10_13_intel.whl
python
import ray
ray.init()
This is what is showing on http://127.0.0.1:8265:
Note: I get this warning:
File descriptor limit 256 is too low for production servers and may result in connection errors. At least 8192 is recommended. --- Fix with 'ulimit -n 8192'