search

nebari-dev / nebari

🪴 Nebari - your open source data science platform
https://nebari.dev
BSD 3-Clause "New" or "Revised" License
281 stars 93 forks source link

[bug] Unable to connect to dashboard using Dask-Gateway and Labextension #1012

Closed viniciusdc closed 2 years ago

viniciusdc commented 2 years ago

Describe the bug

Running the following on qhub v0.3.13 and v0.3.14 generates an accessible link for the dask cluster dashboards, but we are unable to reuse that link to access the dask extension.

from dask_gateway import Gateway
gateway = Gateway()

options = gateway.cluster_options()
options

cluster = gateway.new_cluster(options)
cluster

successful generates a working URL for the dashboards (can be accessed): image

Dashboard URL does not work: image

Expected behavior

How to reproduce

costrouc commented 2 years ago

@viniciusdc has ideas for future work on this. For now we are going to move this to the v0.4.x release. Ideally the clusters should just show up without users needing to add the url.

dharhas commented 2 years ago

Do we know why it isn't working and how to fix?

Ideally the clusters should just show up without users needing to add the url. <- It is fine to push this to a later date.

But being able to add the url manually should be fixed in my opinion for 0.4 release. Folks use this and it is important for demos. How difficult is the fix? @viniciusdc @costrouc

costrouc commented 2 years ago

Personally I don't understand why that feature isn't working. When I looked at the http traffic it is making a request and getting a reposonse. I don't see any 400/500 http errors.

image

costrouc commented 2 years ago

image

Seeing some redirect when I try to visit the dashboard without the status

costrouc commented 2 years ago

We will look at this issue for another 2 hours and then likely make the decision to move this to the 0.4.1 release.

viniciusdc commented 2 years ago

Moving this to v0.4.x

dharhas commented 2 years ago

@viniciusdc @costrouc This is still a regression. And is used by @rsignell-usgs heavily and by me and others in demo's what is the plan for investigating and fixing.

viniciusdc commented 2 years ago

Hi @dharhas, sorry for the delay with this, here are some details regarding this issue:

I will open a detailed issue on the lab-extension repo, as they might have a solution that we could implement.

costrouc commented 2 years ago

@viniciusdc is going to test this theory out but I believe I know the issue. The user is authenticated in the browser via the single sign on flow. This means that the cookies are stored in the browser not on the jupyterlab server.

The dask lab extension has two parts code that runs in the browser and code that runs on the server. This check https://github.com/dask/dask-labextension/blob/main/dask_labextension/dashboardhandler.py is running on the server which does not have credentials.

The way to test this is to run the code that is done on the lab extension and run it on both the server and browser. If this theory is correct the browser code will be the only one to run properly. I think that the lab extension broke once we started using forward auth in a v0.3.1X version.

viniciusdc commented 2 years ago

This issue seems to corroborate our hypothesis https://github.com/dask/dask-labextension/issues/190

viniciusdc commented 2 years ago

So as we originally had expected, here is a simple test suit that shows the above-mentioned issue (now verified):

gateway = Gateway() options = gateway.cluster_options() options

- To mimic what lab-extension does (see the corresponding function [here](https://github.com/dask/dask-labextension/blob/9433ecd21842417dda1579b78127dd4babfb8a4e/dask_labextension/dashboardhandler.py#L18)) we can then proceed with:
```python
import requests

def fetch(url):
    "Added this just in case we got stuck with redirects"
    request = requests.get(url)
    if request.status_code != 200:
        print(request.headers)
        msg = 'Error Code: %i on request'
        raise Exception(msg % request.status_code)
    return request

class Request():
    protocol = ""
    host = ""

url = _normalize_dashboard_link(parse.unquote(url), Request())  # just copied the function from source
individual_plots_url = url_path_join(url,  "individual-plots.json",)

individual_plots_response = fetch(individual_plots_url)

# inspect fetch response, look for individual plots response
response_content = individual_plots_response.content

try:
    assert individual_plots_response.headers.get('content-type') == 'application/json'
except AssertionError as err:
    with open("response.html", "w+") as file:
        file.write(individual_plots_response.text)

from the above code snippet we got the following body response (rendered HTML from response.html): image

viniciusdc commented 2 years ago

Just a bit more info, from ForwardAuth we can inspect the redirection to Keycloak for authentication:

viniciusdc commented 2 years ago

@costrouc i think this issue still needs to be open right? as this is related to how labextension handles the oauth requests...

viniciusdc commented 2 years ago

I will be testing what was proposed in https://github.com/dask/dask-labextension/issues/190#issuecomment-1075559415

jabbera commented 2 years ago

@viniciusdc did you ever have a chance to try the workaround? I'm having the same issue.

viniciusdc commented 2 years ago

@viniciusdc did you ever have a chance to try the workaround? I'm having the same issue.

Hi @jabbera, sorry for the late feedback on this. Yes, the workaround fixed the issue. To accomplish this just add an else if in here with a fetch. don't forget to add the same return object as the MakeRequest function. I will propose a PR to lab-extension with those changes.

viniciusdc commented 2 years ago

Some feedback on this, we have the fix already merged into dask-labextension repository and we will be including this in the next release! https://github.com/dask/dask-labextension/pull/229.

viniciusdc commented 2 years ago

to close this we need: