Jupyterhub fails to spawn servers when installed with Binderhub via Helm

[oyamin]

I am running binderhub on AWS EKS 1.21. Binderhub version 0.2.0-n880.hd19a36a. Using Okta authenticator via GenericOAuthenticator.

When I install binderhub with helm, it works perfectly fine, but jupyterhub does not work. If I install jupyterhub alone without binderhub, using:

helm upgrade --cleanup-on-fail --install jupyterhub jupyterhub/jupyterhub -n jhub --version=1.2.0 -f values.yaml

It works perfectly fine.

The error I am getting when trying to start the server on Jupyterhub when installed with binderhub is:

In the UI:

Error in Authenticator.pre_spawn_start: HTTPError HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for oyamin@example.com may contain details.)```

and in the logs:

[E 2022-03-28 19:12:09.437 JupyterHub user:718] Unhandled error starting oyamin@example.com's server: HTTP 400: Bad Request (token required)
[E 2022-03-28 19:12:09.456 JupyterHub gen:623] Exception in Future <Task finished name='Task-28' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py:900> exception=HTTPError()> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/base.py", line 907, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 736, in spawn
        raise e
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/user.py", line 632, in spawn
        f = maybe_future(spawner.start())
      File "<string>", line 92, in start
    tornado.web.HTTPError: HTTP 400: error (token required)

[E 2022-03-28 19:12:09.457 JupyterHub pages:261] Failed to spawn single-user server with form
    Traceback (most recent call last):
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py", line 257, in post
        return await self._wrap_spawn_single_user(
      File "/usr/local/lib/python3.8/dist-packages/jupyterhub/handlers/pages.py", line 314, in _wrap_spawn_single_user
        raise web.HTTPError(
    tornado.web.HTTPError: HTTP 500: Internal Server Error (Error in Authenticator.pre_spawn_start: HTTPError HTTP 500: Internal Server Error (Spawner failed to start [status=1]. The logs for oyamin@example.com may contain details.))```

[welcome[bot]]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

[oyamin]

Looks like there is also same issue that was posted on the forum back in January with no acknowledgement: https://discourse.jupyter.org/t/unable-to-create-jupyter-server-from-ztbh-install/12752

[oyamin]

After testing some older versions, the issue started happening after upgrading to the following version:

0.2.0-n683.h469f5e5 | 09 September 2021 - Error started appearing 0.2.0-n676.h0cbee48 | 09 September 2021 - Error does not appear

from: https://jupyterhub.github.io/helm-chart/#development-releases-binderhub

But now when i try to access my server at: https://example.com/user/oyamin@example.com

404 : Not Found
You are requesting a page that does not exist!

Same with: https://example.com/hub/tree

404 : Not Found
Jupyter has lots of moons, but this is not one...

[oyamin]

The 404 errors where resolved by adding:

  singleuser:
    cmd: jupyterhub-singleuser

[oyamin]

Issue was resolved when I added the following fields to my secret.yaml

jupyterhub:
  hub:
    services:
      binder:
        apiToken: "<output of `openssl rand -hex 32` command>"
    config:
      GenericOAuthenticator:
        client_id: <okta client_id> # since we use okta
        client_secret: <client_secret>
  proxy:
    secretToken: "<output of `openssl rand -hex 32` command>"

[JColl88]

I encountered this, and found the answer here (https://discourse.jupyter.org/t/unable-to-create-jupyter-server-from-ztbh-install/12752/3) to be useful too; to be able to spawn environments in the regular way with the BinderHub's JupyterHub, you need to set:

hub:
  config:
    BinderSpawner:
      auth_enabled: true