Open Hoeze opened 3 years ago
@Hoeze in your logs I don't see something like
[I 2020-11-05 07:34:13.757 JupyterHub log:181] 200 POST /hub/api/batchspawner (...) 12.12ms
That's when batchspawner-singleuser
calls back to the hub to report in what port its using. I see you've got the requisite import batchspawner
so I'm not quite sure why it's not showing up, unless there's some sensitivity about where in the config file that import happens. FWIW it's dead last in my config.
Thanks for your answer @rcthomas. I moved the import to the end but this does not change anything.
Could this be a problem with ConfigProxy
?
I noticed that the TestServer configuration does not work as well:
It just throws a lot of
404
's like 17:18:16.450 - error: [ConfigProxy] 404 GET /custom/custom.css
.
Logs:
@rcthomas Here is a minimal example configuration that does not work:
import batchspawner
c.JupyterHub.bind_url = 'http://:8686/jupyter/'
c.JupyterHub.default_url = 'home'
c.JupyterHub.hub_connect_ip = 'ouga01'
c.JupyterHub.hub_ip = '0.0.0.0'
c.JupyterHub.hub_port = 8687
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
(
"Test server",
'local-test',
'jupyterhub.spawner.LocalProcessSpawner',
{
'ip':'0.0.0.0'
}
)
]
This has no user authentication, etc.
I run this with jupyterhub -f config_min.py
.
At least this minimal example should work out-of-the-box, right?
This goes away if I specify traitlets<5
in the build. Can you try that. See also: https://github.com/jupyterhub/jupyterhub/issues/3170 --- maybe this is the reproducer we're looking for there.
An even more minimal reproducer is just:
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
(
"Test server",
'local-test',
'jupyterhub.spawner.LocalProcessSpawner',
{
'ip':'0.0.0.0'
}
)
]
Full details. I tested by building this Dockerfile. To make it fail, take out traitlets<5
FROM ubuntu:focal
LABEL maintainer="Rollin Thomas <rcthomas@lbl.gov>"
WORKDIR /srv
ENV DEBIAN_FRONTEND noninteractive
ENV LANG C.UTF-8
RUN \
apt-get update && \
apt-get upgrade --yes && \
apt-get install --yes \
--no-install-recommends \
git \
npm \
python3-pip \
python3-setuptools \
tzdata \
vim
ENV TZ=America/Los_Angeles
RUN \
ln -snf /usr/share/zoneinfo/$TZ /etc/localtime && \
echo $TZ > /etc/timezone
RUN \
pip3 install \
--no-cache-dir \
jupyterhub \
jupyterlab \
batchspawner \
git+https://github.com/jupyterhub/wrapspawner \
'traitlets<5'
RUN \
npm install -g configurable-http-proxy
# Some dummy users
RUN \
adduser -q --gecos "" --disabled-password user1 && \
echo user1:user1 | chpasswd
ADD jupyterhub_config.py /srv/jupyterhub_config.py
For the config it's just
c.JupyterHub.spawner_class = 'wrapspawner.ProfilesSpawner'
c.ProfilesSpawner.profiles = [
(
"Test server",
'local-test',
'jupyterhub.spawner.LocalProcessSpawner',
{
'ip':'0.0.0.0'
}
)
]
Thank you so much @rcthomas!
Yes, installing traitlets<5
is EXACTLY the solution for both problems.
Is there somewhere I can see the logs for wrapspawner? I'd like to track down a fix for this. Can't seem to find them on the client or the hub - if possible I'd like to use traitlets v5 as it's causing issues with other packages to downgrade to 4.
@hakasapl it would normally be in the regular JupyterHub log output. But if you look in wrapspawner you'll see that there are no log messages coming from it. Since traitlets<5 is working just fine for me I was going to let this slide till I absolutely had to do anything about it or it got fixed. My brilliant plan for that day was to add logging messages and trace through wrapspawner and hub code. Maybe you'll give that a shot? Does the reproducer work for you?
Sounds good, I'll try to add some debug messages and see where I get. The main issue I'm running into is that conda doesn't let me install both traitlets<5 and python 3.9 - 3.8 works fine, so it's fine for now, but may present an issue in the near future. I believe PyPi doesn't have this issue, you can install traitlets<5 and python 3.9.
Finally was able to get enough time to look at this. What is happening is that something has changed in traitlets 4 to traitlets 5 where HasTraits._trait_values
behaves differently. This impacts the computation of common_traits
, specifically the second set here (self.child_spawner._trait_values.keys()
in wrapspawner):
common_traits = (
set(self._trait_values.keys()) &
set(self.child_spawner._trait_values.keys()) -
set(self.child_config.keys())
)
Under traitlets 4.3.3 this is a long list of things, under traitlets 5.0.5 it's much smaller (basically only stuff passed to the constructor, plus ip
from the config file and maybe one or two other things).
I traced through the last few years of changes in traitlets. One of the things that I noticed is that along the way was this accepted MR which adds a less private-looking trait_values()
. I note that when using this (swapping in trait_values()
for _trait_values
) in traitlets 5.0.5 I get everything I was getting in 4.3.3. and more, including things that I seemed to need to be adding back by hand. TBH when that worked I stopped trying to track down what happened to _trait_values
and just tried it, I don't know if we should come back to that.
I've tested this on the reproducer, and I'm going to come up with a PR that at least allows for the old behavior on traitlets 4 but uses this method with traitlets 5. We could try getting that second term of the computation more right for traitlets 4. Stay tuned.
This issue has been mentioned on Jupyter Community Forum. There might be relevant details there:
Hi @rcthomas, is this issue solved as of today? Can I upgrade to the latest traitlets now?
@Hoeze the part about about common_traits
was fixed in release 1.0.1 I believe:
https://github.com/jupyterhub/wrapspawner/releases/tag/v1.0.1
So I think the traitlets-related part of this is fixed with that release.
Hi, apologies for the double post, I moved this issue from https://github.com/jupyterhub/batchspawner/issues/194.
We currently cannot spawn any workers with ProfilesSpawner enabled. The worker starts normally but the jupyterhub directly kills it.
Logs of the worker:
Logs of jupyterhub:
I am using python 3.7, jupyterhub 1.2, batchspawner 1.0.1 and the current git version of wrapspawner. When directly applying batchspawner, everything is working fine.
My configuration to reproduce: