Open dhirschfeld opened 5 years ago
ian-r-rose
commented
5 years ago @dhirschfeld, thanks for the report. My intention was that the "search" icon next to the URL would attempt to automatically query the dashboard URL from the currently active notebook/console. Is it not working for your use-case? We may need to harden that a bit.
Or are the panes not correctly loading a valid URL? If so, can you provide a sequence of commands so I can try to reproduce (ideally with a local cluster)?
deKeijzer
commented
5 years ago @ian-r-rose I'm having the same issue. Not sure what the search icon is doing, but it's not getting the correct ip. This is what happens after i restart the kernel and run everything up to the client part in cell [4]. It would be nice if this problem can get fixed, that way i can ''reload'' the current open windows, after restarting the kernel. Without having to re-open them and creating the layout i like to have.
TLDR: the search icon makes it get 'http://192.168.0.100/20840/1:8787/' while it should get 'http://localhost:8787/status'.
mrocklin
commented
5 years ago It looks like this is a problem within the dask/distributed library. The Cluster.dashboard_link property gets the host address of the scheduler, but if the scheduler is running using the inproc:// protocol rather than tcp:// then this gives a wacky address defined by the PID and port number.
This can be fixed here:
Probably we want to replace host with localhost if self.scheduler.address starts with inproc://
@deKeijzer is this something that you have the time to contribute?
also cc @jacobtomlinson
deKeijzer
commented
5 years ago @mrocklin Sadly not at the moment, but thanks for asking. I'm still quite busy with actually learning Dask haha.
mrocklin
commented
5 years ago Well then short term I recommend setting Client(processes=True) or just putting http://localhost:8787 into the search bar manually.
On Mon, Oct 1, 2018 at 11:27 AM Brian de Keijzer notifications@github.com wrote:
@mrocklin https://github.com/mrocklin Sadly not at the moment, but thanks for asking. I'm still quiet busy with actually learning Dask haha.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/dask/dask-labextension/issues/26#issuecomment-425950069, or mute the thread https://github.com/notifications/unsubscribe-auth/AASszNBkb8SQBhGaASh4mnXqJiG-ujzCks5ugjRQgaJpZM4W17u2 .
dhirschfeld
commented
5 years ago My intention was that the "search" icon next to the URL would attempt to automatically query the dashboard URL from the currently active notebook/console. Is it not working for your use-case? Or are the panes not correctly loading a valid URL?
What I'm observing is that when I close a dask cluster I see the 404: Not Found message in the open panes. When I then start a new cluster (using a different random port) and put in the correct url I observe the buttons all turn from grey to orange (so it has connected correctly) however the panes don't automatically refresh and recognise the new url - they continue to display the 404: Not Found message.
The only way to get the panes to recognise the new url is to close and then re-open them from the sidebar. This necessitates a lot of mucking about rearranging the layout you had previously so it would be great if the panes automatically refreshed the connection when you entered a new url.
This sounds like a different problem to what @deKeijzer is observing as my url is correct and the extension is connected (buttons are orange) - it's just the existing panes aren't recognising a new url.
ian-r-rose
commented
5 years ago @dhirschfeld I am having a tough time reproducing this. I have a notebook with a single cell:
from dask.distributed import Client
client = Client()
clientand am performing the following steps:
What browser are you using? I can reproduce this behavior in both firefox and chrome on Ubuntu.
dhirschfeld
commented
5 years ago The dashboard panel becomes empty
You mean you see the same 404: Not Found message?
I'm using chrome. My distributed scheduler is remote and every time I restart it gets mapped to a new port - that could be the key here.
I'll see if I can test further to narrow it down...
ian-r-rose
commented
5 years ago This should be fixed by #36, once that lands.
vprzybylo
commented
5 years ago Any updates on this issue? I am experiencing similar greyed out buttons to access the dashboard and receive a connection refused when opening 'http://localhost:8888/proxy/8787/status' in a new tab in chrome.
Likewise, I use a PBS cluster and pangeo on cheyenne and port forward to see the lab notebook running remotely. I was once able to connect to a client and start some workers but also saw the 404: Not Found for the dask task stream.
Would love to get this up and running to figure out my worker load balance and see the task stream.
bmcfee
commented
3 years ago Chiming in here to say that I'm also having this problem, using dask-labextension on a slurm cluster behind a vpn. I can access the dashboard through an ssh tunnel, but putting the tunneled dashboard url (http://localhost:8787) into the search box does not appear to do anything.
jacobtomlinson
commented
3 years ago @bmcfee are you also accessing Jupyter at localhost?
Just to check you can view the dashboard at http://localhost:8787 but if you put it into the box and press enter the buttons remain grey?
bmcfee
commented
3 years ago No, our jupyter setup runs a bit differently, and is not exposed as localhost.
Otherwise, that's correct: I can tunnel a port on localhost to the machine running the dask dashboard, visit the dashboard directly, but pasting the link doesn't work.
jacobtomlinson
commented
3 years ago No, our jupyter setup runs a bit differently, and is not exposed as localhost.
This could be the problem then.
If you remove /lab from the end of your Jupyter URL and add /proxy/8787/status to the end are you able to view the dashboard?
bmcfee
commented
3 years ago If you remove
/labfrom the end of your Jupyter URL and add/proxy/8787/statusto the end are you able to view the dashboard?
Aha, that does seem to work!
Is there any way (eg from a cell magic or something) to push that over to labextension from a notebook cell? (Background: I'm using this for teaching purposes, and it'd be best if students didn't have to copy-mangle-paste to get the dashboard working.)
bmcfee
commented
3 years ago Actually, I spoke too soon -- it doesn't quite work. Modifying the url as you suggest does connect to the dashboard, and turns the links orange, but none of the dashboard contents show up.
However, if I connect to the dashboard through the ssh tunnel, everything does appear as expected. This might be down to a quirk of how our network is set up, in which case, feel free to ignore this. Seems strange though!
jacobtomlinson
commented
3 years ago Is there any way (eg from a cell magic or something) to push that over to labextension from a notebook cell?
You can set the dashboard url templarte in your Dask config which is picked up by the Lab extension.
Modifying the url as you suggest does connect to the dashboard, and turns the links orange, but none of the dashboard contents show up.
Does the browser developer tools show any error relating to the plot contents? Are you able to right click the plot tabs and open in a new browser tab and see it working there?
This might be down to a quirk of how our network is set up, in which case, feel free to ignore this. Seems strange though!
This likely is the case, but it would be nice to get to the bottom of what is causing it.
bmcfee
commented
3 years ago Does the browser developer tools show any error relating to the plot contents? Are you able to right click the plot tabs and open in a new browser tab and see it working there?
Loading the proxy url directly does bring up the page, but it's generally blank (except for the info tab, which works). Inspecting the dev console, I see:
[bokeh] Lost websocket 0 connection, 1006 ()
[bokeh] Websocket connection 0 disconnected, will not attempt to reconnectInteresting, we've seen similar issues in a few other places where whatever proxy is being placed in front of Jupyter is not correctly handling websocket connections.
our jupyter setup runs a bit differently
Are you able to share any more detail on how you are accessing Jupyter? Specifically what proxying technology is being used.
Am I correct in thinking the regular dashboard works via the /proxy/... url but the inidividual plots do not?
bmcfee
commented
3 years ago Are you able to share any more detail on how you are accessing Jupyter? Specifically what proxying technology is being used.
We use Open OnDemand for this, but I don't know much more than that, sorry!
Am I correct in thinking the regular dashboard works via the
/proxy/...url but the inidividual plots do not?
That's right -- the menu at the top shows up, but all tabs (except info) are blank / nonexistent.
jacobtomlinson
commented
3 years ago That's right -- the menu at the top shows up, but all tabs (except info) are blank / nonexistent
Ok it sounds like all the websocket connections are being dropped, which is how we update the plots. I suspect that Open OnDemand is proxying the HTTP traffic and perhaps not handling the websocket connections correctly.
Jupyter also uses websockets, so things must be working correctly in that case. I'm not sure what is going wrong here though.
We've seen a similar problem in dask/dask#5432 on AWS Sagemaker.
jacobtomlinson
commented
3 years ago Looks like this also came up in #176.
Out of curiosity @bmcfee what bokeh version is in your environment?
bmcfee
commented
3 years ago Out of curiosity @bmcfee what bokeh version is in your environment?
2.3.0.
jacobtomlinson
commented
3 years ago Thanks @bmcfee that rule it out as a 1.x issue.
My worry here is that it is an issue being caused by the Open OnDemand proxy which we have no control over.
A common workflow for me when using distributed is to layout my JupyterLab with a couple of
daskpanes so I can keep an eye on the cluster. I'm often starting new clusters and pasting new/statusurls in thedask-labextensionsidebar. When I paste in a new url the currentdaskpanes show a 404 message and AFAICS the only way to get them to recognise the new connection is to close the existing panes, open new panes and then recreate the layout as I had it previously.It would be great if the
daskpanes could recognise a new url and connect to it automatically. If there's a reason that can't work automatically, perhaps rather than a 404 message the panes could have a refresh button which would allow them to connect to a new url?