search

coiled / feedback

A place to provide Coiled feedback
14 stars 3 forks source link

Miscellaneous issues with Jupyter Lab on Coiled Cloud #259

Closed dym-ok closed 1 year ago

dym-ok commented 1 year ago

I launched a notebook instance in eu-central-1 region, cluster notebook-bebd61b9 All I could see after it launched is a 404 error page:

image

The instance url was https://cluster-xvzeg.dask.host/jupyter/lab

ntabris commented 1 year ago

Hi, @dym-ok. Your local software environment is being synced to the cluster, and from what I see it looks like maybe your local environment doesn't have jupyterlab, or at least not in a way that we're picking up (I do see jupyter metapackage, as well as some other jupyter components, but not jupyterlab itself).

Mind trying an environment that has jupyterlab explicitly installed and see if that works better?

ntabris commented 1 year ago

For reference, you can get the list of packages we synced to cluster at https://cloud.coiled.io/software/alias/37467/build/28221

jrbourbeau commented 1 year ago

Thanks for the issue @dym-ok. Yeah, @ntabris is correct that we should avoid spinning up a notebook and emit a useful message when jupyterlab isn't installed locally (and automatic package sync) is being used. @dym-ok we've got a fix for this on our end that should be out in the next day a two. With the fix I get the following when in an environment without jupyterlab installed

$ coiled notebook start
Jupyterlab must be installed locally to launch notebooks.

Installing jupyterlab locally will fix the 404

dym-ok commented 1 year ago

Thanks @ntabris @jrbourbeau Somehow I didn't realise it needs to be present locally 😅 I am able to launch working Jupyter Lab environment after I added it, however I got this error when I added the --sync option:

Traceback (most recent call last):
  File "/Users/dym/mambaforge/envs/dask-labs/bin/coiled", line 8, in <module>
    sys.exit(cli())
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/coiled/cli/notebook/notebook.py", line 333, in start_notebook
    connect_mutagen_sync(cloud, cluster_id, widget.live.console)
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/site-packages/coiled/cli/notebook/notebook.py", line 145, in connect_mutagen_sync
    result = subprocess.run(
  File "/Users/dym/mambaforge/envs/dask-labs/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['mutagen', 'sync', 'create', '--name', 'coiled-270317', '--label', 'managed-by=coiled', '--label', 'cluster-id=270317', '--ignore-vcs', '--max-staging-file-size=1 GiB', '.', 'ubuntu@3.79.186.236:/scratch/synced']' returned non-zero exit status 1.
dym-ok commented 1 year ago

I installed mutagen@0.16 via Homebrew.

dym-ok commented 1 year ago

I also have a few other questions:

  1. how to achieve some persistence of Jupyter Lab configuration (extension collection, settings, etc.)?
  2. I tried to connect to https://cluster-kmtwl.dask.host (with token included in the URL) from VS Code and it kept getting HTTP status 403, is there something I can tweak on my end to make it work?
dym-ok commented 1 year ago

This VS Code issue might be related to this configuration.

ntabris commented 1 year ago

@dym-ok any chance you're using 1Password ssh integration (or some other non-default ssh agent)?

dym-ok commented 1 year ago

Hi @ntabris, I do use 1Password as an ssh agent. Do you I need some extra configuration in ~/.ssh/config?

ntabris commented 1 year ago

Do you I need some extra configuration in ~/.ssh/config?

Yes, the problem is that we the syncing happens over ssh connection and the 1Password agent doesn't let us automatically add the (temporary) keypair.

This won't work yet but would you be comfortable adding this to the top of your ~/.ssh/config

Host *.dask.host
  IdentityAgent SSH_AUTH_SOCK

?

(I also need to make a change on our side to make this work, since currently we're using IP address rather than hostname for ssh).

dym-ok commented 1 year ago

This won't work yet but would you be comfortable adding this to the top of your ~/.ssh/config

Absolutely, this looks like a very reasonable configuration change. Thanks @ntabris 🙏🏾

dym-ok commented 1 year ago

BTW, what about my question about connecting to Jupyter Lab from VS Code? Have you guys tried it out?

ntabris commented 1 year ago

@dym-ok coiled=0.9.29 is on pypi and will be up on conda-forge shortly (as soon as their bots get to it). This will address the issue with 1Password. You won't have to manually edit your ssh config (unless you want to), running coiled notebook --sync will check your ssh config and give you the option to let it automatically update the config if necessary.

Don't have an answer for you yet on VS Code.

dym-ok commented 1 year ago

Awesome @ntabris, I can confirm that --sync option works for me now 🙏🏾

image
dym-ok commented 1 year ago

Now the last hurdle in the way of my happiness is connecting there from VS Code. Is there a way to make sure this configuration is in place?

Or is there a different way to get rid of this error?

image
dym-ok commented 1 year ago

I checked the Developer Console in VS Code and this is the error I'm getting there, so I don't think it's related to CORS configuration anymore:

h [FetchError]: invalid json response body at https://cluster-bwhaw.dask.host/api/kernels?1695330122181 reason: Unexpected token '<', "<html>
   "... is not valid JSON
    at /Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:1131055
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Function.create (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:68017)
    at Object.t.listRunning (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:55954)
    at d.requestRunning (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:37794) {
  type: 'invalid-json'
}
h [FetchError]: invalid json response body at https://cluster-bwhaw.dask.host/api/sessions?1695330122199 reason: Unexpected token '<', "<html>
   "... is not valid JSON
    at /Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:1131055
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Function.create (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:68017)
    at Object.t.listRunning (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:78913)
    at d.requestRunning (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:77235) {
  type: 'invalid-json'
}
h [FetchError]: invalid json response body at https://cluster-bwhaw.dask.host/api/kernelspecs?1695330122126 reason: Unexpected token '<', "<html>
   "... is not valid JSON
    at /Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:1131055
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Function.create (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:68017)
    at Object.t.getSpecs (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:63552)
    at u.requestSpecs (/Users/dym/.vscode/extensions/ms-toolsai.jupyter-2023.8.1002501831-darwin-arm64/out/extension.node.js:2:63075)
    at async Promise.all (index 0) {
  type: 'invalid-json'
dym-ok commented 1 year ago

I guess the problem is caused by your authorisation mechanism, which is different from what VS Code expects.

ntabris commented 1 year ago

I guess the problem is caused by your authorisation mechanism, which is different from what VS Code expects.

I'm taking a look at VS Code stuff now... will let you know what I find.

ntabris commented 1 year ago

@dym-ok I tried various things with VS Code + jupyter and so far haven't gotten it working. Like you said, I think it's related to auth, but I haven't yet found a reasonable way for us to do auth that also works with VS Code.

My apologies but I don't expect that we'll have a solution for this in the near future.