Closed PerilousApricot closed 3 months ago
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
This currently bites me too on trying to upgrade from z2jh 2 to z2jh 3...
Maybe a possible workaround would be to use a Spawner.pre_spawn_hook
!?
Although from reading the code, I can not figure out why it does not work in the first place (i.e. why is https://github.com/jupyterhub/jupyterhub/blob/ca6032381aa137f03cb9349265fa74ed5e4c5c19/jupyterhub/spawner.py#L1101 apparently not relevant?)...
KubeSpawner calls the parent function to evaluate callables as you've pointed out: https://github.com/jupyterhub/kubespawner/blob/f99b98fc731517129ebfb7ef3fee293f28d75a75/kubespawner/spawner.py#L2161
but in addition it adds template expansion of values, which is causing the problem: https://github.com/jupyterhub/kubespawner/blob/f99b98fc731517129ebfb7ef3fee293f28d75a75/kubespawner/spawner.py#L2169
I think the correct fix is to only expand non-callable values (since if you're using a callable you most likely don't want the result to be interpreted as a template).
@manics I still don't get why the template expansion should cause this - it should ignore anything that is not list, dict or str, shouldn't it?
I really appreciate this issue writeup @PerilousApricot - thank you! My favorite part is the inclusion of the docs reference that makes it definite its a bug in the project!
And, according to the docs, this should be supported:
The environment configurable should be set by JupyterHub administrators to add installation specific environment variables. It is a dict where the key is the name of the environment variable, and the value can be a string or a callable. If it is a callable, it will be called with one parameter (the spawner instance), and should return a string fairly quickly (no blocking operations please!).
Thank you @manics for the fix and thank you @jabbera and @moschlar for reporting on this as well!
Bug description
When upgrading kubespawner (via an associated z2jh helm upgrade from 2.0.0 to 3.2.1), kubespawner lost the ability to pass in callable to the environment variables dict. In my jupyterhub config, I have the following:
And, according to the docs, this should be supported:
However, attempting to spawn a notebook with the previous configuration snippet results in an error:
Which (naively) appears to be an issue where the callable isn't first called to get a string back.
How to reproduce
In a jupyterhub + kubespawner configuration, set the following configuration value:
Expected behaviour
I would expect a new user pod to be spawned with the HOME environment variable set to the user's home directory
Actual behaviour
The spawning fails.
Your personal set up
This cluster is running k8s version:
with z2jh version 3.2.1.
root@hub-6f754c444-tlt2j:/srv/jupyterhub# /usr/local/bin/jupyterhub --version 4.0.2 root@hub-6f754c444-tlt2j:/srv/jupyterhub# /usr/local/bin/python --version Python 3.11.6
Full environment
``` accre==4.0.0 aiohttp==3.9.1 aiosignal==1.3.1 alembic==1.12.1 asn1crypto==1.5.1 async-generator==1.10 attrs==23.1.0 bcrypt==4.0.1 certifi==2023.11.17 certipy==0.1.3 cffi==1.16.0 charset-normalizer==3.3.2 cryptography==41.0.5 dcim==0.3.0 escapism==1.0.1 frozenlist==1.4.0 greenlet==3.0.1 idna==3.6 Jinja2==3.1.2 jsonschema==4.20.0 jsonschema-specifications==2023.11.1 jupyter-telemetry==0.1.0 jupyterhub==4.0.2 jupyterhub-firstuseauthenticator==1.0.0 jupyterhub-hmacauthenticator==1.0 jupyterhub-idle-culler==1.2.1 jupyterhub-kubespawner==6.2.0 jupyterhub-ldapauthenticator==1.3.2 jupyterhub-ltiauthenticator==1.6.1 jupyterhub-nativeauthenticator==1.2.0 jupyterhub-tmpauthenticator==1.0.0 kubernetes-asyncio==28.2.1 ldap3==2.9.1 Mako==1.3.0 MarkupSafe==2.1.3 multidict==6.0.4 mwoauth==0.4.0 nullauthenticator==1.0.0 oauthenticator==16.2.1 oauthlib==3.2.2 onetimepass==1.0.1 packaging==23.2 pamela==1.1.0 pg8000==1.30.3 prometheus-client==0.19.0 proxmoxer==2.0.1 psycopg2==2.9.9 py-spy==0.3.14 pyasn1==0.5.1 pycparser==2.21 pycurl==7.45.2 PyJWT==2.8.0 PyMySQL==1.1.0 pyOpenSSL==23.3.0 pystache==0.6.5 python-dateutil==2.8.2 python-json-logger==2.0.7 python-slugify==8.0.1 PyYAML==6.0.1 referencing==0.31.0 requests==2.31.0 requests-oauthlib==1.3.1 rpds-py==0.13.1 ruamel.yaml==0.18.5 ruamel.yaml.clib==0.2.8 scramp==1.4.4 six==1.16.0 SQLAlchemy==2.0.23 sqlalchemy-cockroachdb==2.0.1 statsd==4.0.1 text-unidecode==1.3 tornado==6.3.3 traitlets==5.13.0 typing_extensions==4.8.0 urllib3==2.1.0 yarl==1.9.3 zxcvbn==4.4.28 ```Configuration
I'm happy to post the configuration, though z2jh builds it dynamically, is there a way to get z2jh to spit out the "rendered" version of the config? This is a pprint.pprint() dump of the spawner ``` {'_cross_validation_lock': False, '_fs_gid_metadata': {}, '_gid_metadata': {}, '_oauth_client_allowed_scopes_metadata': {}, '_oauth_roles_metadata': {}, '_options_form_metadata': {}, '_profile_list_metadata': {}, '_server': Server(url=http://hub-6f754c444-tlt2j:58283/user/meloam/, bind_url=http://*:58283/user/meloam/), '_server_token_scopes_metadata': {}, '_spawn_future':Logs
``` [E 2024-01-02 19:44:56.877 JupyterHub user:884] Unhandled error starting meloam's server: expected string or bytes-like object, got 'function' Traceback (most recent call last): File "/usr/local/lib/python3.11/site-packages/jupyterhub/user.py", line 798, in spawn url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 2718, in _start pod = await self.get_pod_manifest() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/kubespawner/spawner.py", line 1998, in get_pod_manifest return make_pod( ^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/kubespawner/objects.py", line 490, in make_pod "deps": _get_env_var_deps(env), ^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/site-packages/kubespawner/objects.py", line 470, in _get_env_var_deps deps = set(re.findall(re_k8s_env_reference_pattern, env.value)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/local/lib/python3.11/re/__init__.py", line 216, in findall return _compile(pattern, flags).findall(string) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: expected string or bytes-like object, got 'function' [D 2024-01-02 19:44:56.882 JupyterHub user:982] Stopping meloam [D 2024-01-02 19:44:56.900 JupyterHub user:1002] Deleting oauth client jupyterhub-user-meloam [D 2024-01-02 19:44:57.004 JupyterHub user:1005] Finished stopping meloam [E 2024-01-02 19:44:57.202 JupyterHub gen:630] Exception in Future