Closed clkao closed 4 years ago
I think https://github.com/jupyterhub/kubespawner/pull/355 is another k8s 1.16 related issue (and fix).
@betatim right, i think that's exactly the 3rd one i listed. let me link that as well.
Now merged https://github.com/jupyterhub/kubespawner/pull/356. Maybe it is a temporary fix but it'll get people unblocked and we can make a better fix later. WDYT?
@betatim Unfortunately, https://github.com/jupyterhub/kubespawner/pull/356/ did not solve the issue for me:
[E 2019-10-14 14:15:11.268 JupyterHub web:1788] Uncaught exception GET /hub/api/users/304957/server/progress (192.168.2.183)
HTTPServerRequest(protocol='http', host='jupyter.rwth-aachen.de', method='GET', uri='/hub/api/users/304957/server/progress', version='HTTP/1.1', remote_ip='192.168.2.183')
Traceback (most recent call last):
File "/usr/local/lib/python3.6/dist-packages/tornado/web.py", line 1699, in _execute
result = await result
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/apihandlers/users.py", line 592, in get
async for event in events:
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 199, in __next__
return self._invoke(self._it.__next__)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/utils.py", line 555, in iterate_until
await yield_(item_future.result())
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/usr/local/lib/python3.6/dist-packages/jupyterhub/spawner.py", line 976, in _generate_progress
async for event in progress:
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 366, in step
return await ANextIter(self._it, start_fn, *args)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 197, in __next__
return self._invoke(first_fn, *first_args)
File "/usr/local/lib/python3.6/dist-packages/async_generator/_impl.py", line 209, in _invoke
result = fn(*args)
File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1541, in progress
events = self.events
File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1512, in events
for event in self.event_reflector.events:
File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 73, in events
key=lambda x: x.last_timestamp if x.last_timestamp is not None else 0.,
TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'
@betatim I have the same issue as @stv0g:
'<' not supported between instances of 'float' and 'datetime.datetime'
In #1422 I'm now testing against 1.16 but allowing allow for failures.
https://travis-ci.org/jupyterhub/zero-to-jupyterhub-k8s/jobs/599062620
https://github.com/jupyterhub/kubespawner/pull/357 is up for people who are having trouble running with kubernetes 1.16 to take over and finish off. I won't get around to pushing it over the finish line.
Help pushing https://github.com/jupyterhub/kubespawner/pull/357 through the finish line would be appreciated. Note that the current development setup described in CONTRIBUTING.md may make it a plausible to locally develop a functional solution of z2jh+kubespawner+k8s 1.16. I suggest using the dev
script to create a kind cluster with k8s 1.16 and run tests etc.
To get the test to use a locally developed kubespawner, I suggest to git clone kubespawner into the images/hub folder, and in the images/hub/Dockerfile add some steps where we copy the local kubespawner repo along with local changes and install it with pip install -e kubespawner
or similar.
Quick and dirty way:
Replace jupyterhub-kubespawner==0.10.1
with https://github.com/jupyterhub/kubespawner/archive/884f2f6f68e13401a39265d4617ce465a716a4a3.zip
:
https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/be710d7f0107f64219bbd2eb52c22e2f89840715/images/hub/requirements.txt#L7
And test as normal (or push to GitHub as a temporary branch and let travis test it for you)
@clkao, all these issues are now being addressed, mostly in #1483:
It appears there are several issues with running z2jh on kubernetes 1.16, running the latest master with kind:
scheduling.userScheduler.enabled: true
causing user pod to stuck in Pending state, without any eventsingleuser.events: true
causing launch failures, as events seem not to be what EventReflector is expecting: (https://github.com/jupyterhub/kubespawner/issues/354)I haven't looked further but each of them should probably be of its own issue