When using elyra dev tag error is main.js: SyntaxError: expected expression, got '?'

[dummys]

Hi, using kubernetes and the elyra image I get the following error:

SyntaxError: expected expression, got '?'main.e41ae1cf1107c2b063a2.js:46:31

It seems to be this line: _CONFIG_DATA = configData ?? Object.create(null); that fail.

when launching with helm my chart for jupyterhub I get this warning:

coalesce.go:196: warning: cannot overwrite table with non table for extraConfig (map[])

My config:

hub:
  db:
    type: sqlite-memory

  extraConfig: |-
    from kubespawner import KubeSpawner
    from tornado import gen
    import yaml

    class CustomKubeSpawner(KubeSpawner):
        def get_env(self):
            env = super().get_env()

            env['KG_HTTP_USER'] =  self.user.name
            env['KERNEL_USERNAME'] =  self.user.name

            return env
    c.JupyterHub.spawner_class = CustomKubeSpawner
    config = '/etc/jupyter/jupyter_notebook_config.py'
    c.Spawner.start_timeout = 500
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: traefik
  hosts:
    - ""
singleuser:
  # force using new UI from jupyterlab
  defaultUrl: "/lab"
  image:
    name: elyra/elyra
    tag: dev
    # disable this in a production environment
    pullPolicy: "Always"
  storage:
    dynamic:
      storageClass: nfs-dynamic
  extraEnv:
    JUPYTER_GATEWAY_URL: http://myurl/gateway
    KG_REQUEST_TIMEOUT: "120"

[lresende]

This is good timing, as I am just about updating my jupyterhub scripts/environment.

By taking a quick look at your config, I noticed it has a couple of differences: https://github.com/lresende/ansible-kubernetes-cluster/blob/master/roles/jupyterhub/templates/jupyterhub-config.yaml.j2

Let me make progress on my side and I will let you know.

BTW, what version of Hub or Z2JH are you using?

[dummys]

Hello,

first I tried the 1.2.1 but then I get warning because I'm not using 1.1.0, then I downgraded to 1.1.0

Unfortunately I can't use your ansible playbook, as the computer is installed under ubuntu. So I reuse most of your script, but I install everything using Helm rather than manual to track installation.

@lresende just a follow up, commenting /lab made it, but then I loose the jupyterlab interface. I have still some problem making the gateway works with traefik for now. I can work only without the gateway. I'm debugging the stuff but I'm really a noob in k8s :D And when i go to /gateway I get this: {"reason": "Not Found", "message": ""}

Something must be wrong with my traefik stuff, but I can't figured it out EDIT: traefik igress is working on the gateway because when I go to /gateway I get log on the enterprise-gateway:

[W 201126 14:44:00 web:2250] 404 GET /gateway (192.168.62.135) 1.06ms
[W 201126 14:44:05 web:2250] 404 GET /gateway (192.168.62.135) 1.01ms
[W 201126 14:44:09 web:2250] 404 GET /gateway (192.168.62.135) 1.01ms
[W 201126 14:44:11 web:2250] 404 GET /gateway (192.168.62.135) 1.01ms

Wondering why I get a 404 on the gateway log itself. It means that traefik redirect correctly the query, but enterprise gateway is not understanding.

But if I understand correctly, normaly when I go to /gateway, traefik should redirect the query to / on the gateway right ? Not /gateway

[lresende]

A couple of updates here:

coalesce.go:196: warning: cannot overwrite table with non table for extraConfig (map[])

I got this warning, but AFAIK it is not affecting the configuration at all, and I moved this to a lower priority

But if I understand correctly, normally when I go to /gateway, traefik should redirect the query to / on the gateway right ? Not /gateway

Yes, you will need to configure stripPrefix, and with the new traefik, it should look like this or the old way.

Current status: I am now tweaking the Elyra image to properly work with the volume mounts, as the current one is configuring things on the jovyan user home directory that gets overridden by the nfs storage configured here.

[dummys]

ok to be honest I'm fed up debugging this traefik things, so much time I'm on it and it didn't work. I always get a "404 ERROR for /gateway" It's like the PathPrefixStrip is not working nor used by this F**** traefik....

EDIT: OMFG ITS WORKING !!!! thanks to your new file, now traefik do the pathprefixstrip. To be honest I didn't found any examples like the one you wrote in their documentation !! Just for your information, with new traefik your old way is not working anymore.

Yes now when I want to launch a notebook I get the error: Permission denied: Untitled.ipynb I think it should be related to your NFS error ? why it try to save to: filemanager:468] Saving /home/jovyan/work/Untitled.ipynb ? Because I don't want to use this, in my nfs deployment I changed the home url. Is there something to change inside the config of the notebook ?

EDIT2: I think I found the error of the permission denied, the folder /home/jovyan/work is owned by root in the pod

$ pwd
/home/jovyan
$ ls -all
total 20
drwxrwsrwx 5 root   root 4096 Nov 25 16:17 .
drwxr-xr-x 1 root   root 4096 Jan 30  2020 ..
drwxr-sr-x 3 jovyan root 4096 Nov 25 16:17 .cache
drwxr-sr-x 3 jovyan root 4096 Nov 25 16:17 .local
drwxr-sr-x 2 root   root 4096 Nov 25 16:17 work

And something weird is that in your image, the permission are correct:

jovyan@00092dfdcf65:~$ ls -all
total 56
drwsrwsr-x 1 jovyan users 4096 Nov 24 01:42 .
drwxr-xr-x 1 root   root  4096 Jan 30  2020 ..
-rw-rw-r-- 1 jovyan users  220 Apr  4  2018 .bash_logout
-rw-rw-r-- 1 jovyan users 3770 Jan 30  2020 .bashrc
drwxr-sr-x 1 jovyan users 4096 Nov 24 01:47 .cache
drwsrwsr-x 1 jovyan users 4096 Jan 30  2020 .conda
drwsrws--- 1 jovyan users 4096 Jan 30  2020 .config
drwsrwsr-x 2 jovyan users 4096 Jan 30  2020 .empty
drwsrws--- 1 jovyan users 4096 Nov 24 01:51 .jupyter
drwxr-sr-x 4 jovyan users 4096 Nov 24 01:42 .npm
drwxr-sr-x 5 jovyan users 4096 Nov 24 01:42 .npm-global
-rw-r--r-- 1 jovyan users   63 Nov 24 01:42 .npmrc
-rw-rw-r-- 1 jovyan users  807 Apr  4  2018 .profile
drwsrwsr-x 2 jovyan users 4096 Jan 30  2020 work

And last but not least, is it possible to make it work with ubuntu ? It would be nice to deploy everything from scratch.

[lresende]

Yes, these are some of the issues I have fixed either on the ansible-scripts and/or the new image quay.io/elyra/elyra:dev. Once you pass these, the one extra remaining thing is that in order to have this working with lab3, we need to set JUPYTERHUB_SINGLEUSER_APP: "jupyter_server.serverapp.ServerApp" on the hub config.

With these changes, I seem to have all working now, let me push these changes to the respective repositories and as you to do a final test.

BTW, Thank you for the patience, I know how it is, as I was also very frustrated with ingress and some of the K8s configuration stuff. But I am glad we made good progress.

[lresende]

I have pushed the latest JupyterHub config and you could use the command below to update your hub deployment:

helm upgrade hub jupyterhub/jupyterhub --namespace hub --version 0.10.6 --values /tmp/ansible-install/jupyterhub-config.yaml

Note that the config is now using the elyra image from quay.io, which I have been updating with the fixes to some of the permission issues you described above.

@dummys Please let me know if that works for you.

[lresende]

And last but not least, is it possible to make it work with ubuntu ? It would be nice to deploy everything from scratch.

In ubuntu you could use microk8s which will give you a working k8s environment with storage and ingress, and you just need to cherry-pick the commands to deploy gateway and hub from the ansible scripts.

[lresende]

The changes to Elyra docker image have been released as part of Elyra 1.4.2, and configuration updates have been updated to the ansible scripts. Note that, Elyra 1.4.2 is still based on JupyterLab 2.x while master (and the Eyra image dev tag) is based on JupyterLab 3.0 release candidates.

[dummys]

ok but then changing the version of the hub means changing the version of the gateway ? I have made a small fix to fix the permissions issue:

singleuser:
  # force using new UI from jupyterlab
  defaultUrl: "/lab"
  image:
    name: elyra/elyra
    tag: dev
    # disable this in a production environment
    pullPolicy: "Always"
  storage:
    dynamic:
      storageClass: nfs-dynamic
  extraEnv:
    JUPYTER_GATEWAY_URL: http://
    KG_REQUEST_TIMEOUT: "120"
    GRANT_SUDO: "yes"
  uid: 0
  cmd: null
  lifecycleHooks:
    postStart:
      exec:
        command: ["chown", "-R", "jovyan:root", "/home/jovyan"]

Will test your new stuff now to check if its working.

[lresende]

The hub and gateway versions are independent.

[dummys]

ok I see. For now its not working

https://github.com/jupyter/enterprise_gateway/releases/download/v2.3.0/jupyter_enterprise_gateway_helm-2.3.0.tgz I use this gateway

and the jupyterhub config:

singleuser:
  # force using new UI from jupyterlab
  defaultUrl: "/lab"
  image:
    name: quay.io/elyra/elyra
    tag: dev
    # disable this in a production environment
    pullPolicy: "Always"

When I go to the /gateway I get a 404 error on the enterprise gateway pod again ...

When I login on jupyterhub now I get this: A Jupyter Server is running. but thats it no more jupyterlab interface etc

And here is the log of the jupyterhub-user container:

Executing the command: jupyterhub-singleuser --ip=0.0.0.0
[I 2020-12-07 13:58:27.932 SingleUserNotebookApp manager:298] elyra | extension was successfully linked.
[W 2020-12-07 13:58:27.936 LabApp] 'ip' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2020-12-07 13:58:27.936 LabApp] 'port' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2020-12-07 13:58:27.936 LabApp] 'open_browser' has moved from NotebookApp to ServerApp. This config will be passed to ServerApp. Be sure to update your config before our next release.
[W 2020-12-07 13:58:27.942 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2020-12-07 13:58:27.942 SingleUserNotebookApp manager:298] jupyterlab | extension was successfully linked.
[W 2020-12-07 13:58:27.950 SingleUserNotebookApp configurable:190] Config option `open_browser` not recognized by `SingleUserNotebookApp`.  Did you mean `browser`?
[I 2020-12-07 13:58:28.158 SingleUserNotebookApp nbserver:121] nbgitpuller | extension was found and enabled by nbclassic. Consider moving the extension to Jupyter Server's extension paths.
[I 2020-12-07 13:58:28.158 SingleUserNotebookApp manager:298] nbgitpuller | extension was successfully linked.
[I 2020-12-07 13:58:28.176 SingleUserNotebookApp nbserver:121] nbresuse | extension was found and enabled by nbclassic. Consider moving the extension to Jupyter Server's extension paths.
[I 2020-12-07 13:58:28.176 SingleUserNotebookApp manager:298] nbresuse | extension was successfully linked.
[I 2020-12-07 13:58:28.176 SingleUserNotebookApp manager:298] nbclassic | extension was successfully linked.
[I 2020-12-07 13:58:28.222 SingleUserNotebookApp processor:57] Registering processor "EntryPoint('kfp', 'elyra.pipeline.processor_kfp', 'KfpPipelineProcessor', Distribution('elyra', '2.0.0.dev0'))" with type -> kfp
[I 2020-12-07 13:58:28.222 SingleUserNotebookApp processor:57] Registering processor "EntryPoint('local', 'elyra.pipeline.processor_local', 'LocalPipelineProcessor', Distribution('elyra', '2.0.0.dev0'))" with type -> local
[I 2020-12-07 13:58:28.222 SingleUserNotebookApp manager:307] elyra | extension was successfully loaded.
[I 2020-12-07 13:58:28.223 LabApp] JupyterLab extension loaded from /opt/conda/lib/python3.8/site-packages/jupyterlab
[I 2020-12-07 13:58:28.223 LabApp] JupyterLab application directory is /opt/conda/share/jupyter/lab
[I 2020-12-07 13:58:28.225 SingleUserNotebookApp manager:307] jupyterlab | extension was successfully loaded.
[I 2020-12-07 13:58:28.228 SingleUserNotebookApp manager:307] nbclassic | extension was successfully loaded.
[I 2020-12-07 13:58:28.229 SingleUserNotebookApp manager:307] nbgitpuller | extension was successfully loaded.
[I 2020-12-07 13:58:28.229 SingleUserNotebookApp manager:307] nbresuse | extension was successfully loaded.
[I 2020-12-07 13:58:28.229 SingleUserNotebookApp mixins:558] Starting jupyterhub-singleuser server version 1.2.2
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811] Serving notebooks from local directory: /home/jovyan
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811] Jupyter Server 1.0.8 is running at:
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811] http://jupyter-user:8888/user/user/
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811]  or http://127.0.0.1:8888/user/user/
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811] Kernels will be managed by the Gateway server running at:
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1811] http://server/gateway
[I 2020-12-07 13:58:28.236 SingleUserNotebookApp serverapp:1812] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
[I 2020-12-07 13:58:28.242 SingleUserNotebookApp mixins:538] Updating Hub with activity every 300 seconds
[I 2020-12-07 13:58:28.911 SingleUserNotebookApp log:181] 200 GET /user/user/ (@192.168.62.135) 17.41ms

I think the /etc/jupyter/jupyter_notebook_config.py is a bit outdated:

from jupyter_core.paths import jupyter_data_dir
import subprocess
import os
import errno
import stat

c = get_config()  # noqa: F821
c.NotebookApp.ip = '0.0.0.0'
c.NotebookApp.port = 8888
c.NotebookApp.open_browser = False

NotebookApp should be ServerApp right ?

edit: Tried to update gateway to 2.4.0 same shit again. the A Jupyter Server is running. but no more notebook ui etc

@lresende are you on irc /or slack or whatever ? Then we can debug it quickly

edit2: so if I go to the http://jupyterhub/user/myuser/tree I see the notebook. But when I try the /gateway I get a 404 still How to be sure that the gateway works ?

when connecting to /lab, it didn't found elrya namespace:

Failed to load resource: the server responded with a status of 404 (Not Found)
elyra/schema/code-snippets?1607414775968:1 Failed to load resource: the server responded with a status of 404 (Not Found)
elyra/schema/runtimes?1607414776045:1 Failed to load resource: the server responded with a status of 404 (Not Found)
registry.npmjs.org/-/v1/search?text=+keywords%3A%22jupyterlab-extension%22&size=250&from=0:1 Failed to load resource: net::ERR_CONNECTION_RESET
-F63fjptAgt5VM-kVkqdyU8n1i8q131nj-o.woff2:1 Failed to load resource: net::ERR_CONNECTION_RESET

We are running on a airgapped kubernetes cluster, is he trying to get those resource on the internet ?

It seems that something is missing from the elrya image, but I don't know why.

EDIT2: If i use your old image with elrya/elrya tag dev it's working. When I move to quay.io it's not working.

And last but not least, when my user is login to the jupyterhub interface, they are not redirected to /lab they are welcoming here: http://jupyterhub01.sli.local/user/myuser thus showing the string A Jupyter Server is running.

I'm reviewing the configuration of your image and I don't see where this parameter is used inside the KubeSpawner: defaultUrl: "/lab"

And I think as you are using a customer KubeSpawner, we need to handle the case manually for the: c.Spawner.default_url

There is also a typo in your start-elyra.sh script:

#!/bin/bash
#!/bin/bash

And to be honest I don't know if that crap is running in the process:

 extraConfig: |-
    from kubespawner import KubeSpawner
    from tornado import gen
    import yaml

    class CustomKubeSpawner(KubeSpawner):
        def get_env(self):
            env = super().get_env()

            env['KG_HTTP_USER'] =  self.user.name
            env['KERNEL_USERNAME'] =  self.user.name

            return env
    LOLLOLOLMDR
    c.JupyterHub.spawner_class = CustomKubeSpawner
    config = '/etc/jupyter/jupyter_notebook_config.py'

    c.Spawner.start_timeout = 500

I tried to add bogus python code, and nothing happend, no error or whatever ...

[lresende]

Find me or others on the Elyra team at: https://gitter.im/elyra-ai/community We also have dev meetings every Thursday, see project readme for details (at the bottom of the readme)

[dummys]

ok but I didn't received any response on the gitter.

[lresende]

@dummys please reopen if we still need to discuss this issue.