Current container fails on OCP 4.3 due to old jupiterhub version and k8s 1.16

[therevoman]

There's an issue with JupiterHub and k8s 1.16 which is bundled in OCP 4.3 This workshop spawner is suffering this issue. Update the image to the latest jupiterhub

https://github.com/jupyterhub/zero-to-jupyterhub-k8s/issues/1444

[GrahamDumpleton]

Are you sure it doesn't work. It was using a fork of JupyterHub which had the fix for the date time issue in events which stopped it working. Can you provide a proper explanation of the problem you are seeing.

[GrahamDumpleton]

More specifically, was installing from a specific commit of KubeSpawner GitHub repository.

• https://github.com/openshift-homeroom/workshop-spawner/blob/develop/jupyterhub/src/requirements.txt#L5

[GrahamDumpleton]

BTW, if you are having an issue, it may be because the specific workshop was never updated to use the 2.x branch of the workshop deployment scripts, and is still using 1.x, which would pick up an older version of the spawner.

What is the URL for the Git repo of the workshop? What does the .gitmodules file in it contain?

[bttnns]

I am using the same commit in my requirements.txt and hit an issue on OCP 4.3

ERROR:asyncio:Task exception was never retrieved                                                                                             
future: <Task finished coro=<BaseHandler.spawn_single_user() done, defined at /opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/b
ase.py:697> exception=TypeError("'<' not supported between instances of 'datetime.datetime' and 'float'",)>                                  
Traceback (most recent call last):                                                                                                           
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 889, in spawn_single_user                               
    timedelta(seconds=self.slow_spawn_timeout), finish_spawn_future                                                                          
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/handlers/base.py", line 807, in finish_user_spawn                               
    await spawn_future                                                                                                                       
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 642, in spawn                                                    
    raise e                                                                                                                                  
  File "/opt/app-root/lib/python3.6/site-packages/jupyterhub/user.py", line 546, in spawn                                                    
    url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)                                                                
  File "/opt/app-root/lib/python3.6/site-packages/tornado/gen.py", line 748, in run                                                          
    yielded = self.gen.send(value)                                                                                                           
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 1661, in _start                                              
    events = self.events                                                                                                                     
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 1512, in events                                              
    for event in self.event_reflector.events:                                                                                                
  File "/opt/app-root/lib/python3.6/site-packages/kubespawner/spawner.py", line 73, in events                                                
    key=lambda x: x.last_timestamp if x.last_timestamp is not None else 0.,                                                                  
TypeError: '<' not supported between instances of 'datetime.datetime' and 'float'                                                            
[E 2020-03-03 15:26:47.029 JupyterHub pages:284] Previous spawn for rkwjm failed: '<' not supported between instances of 'datetime.datetime' 
and 'float'                                                                                                                                  
[E 2020-03-03 15:26:47.034 JupyterHub log:166] {                                                                                             
      "X-Forwarded-For": "192.168.130.1,::ffff:10.128.0.1",                                                                                  
      "Forwarded": "for=192.168.130.1;host=odo-odoooo.apps-crc.testing;proto=https;proto-version=\"\"",                                      
      "X-Forwarded-Proto": "https,http",                                                                                                     
      "X-Forwarded-Port": "443,80",                                                                                                          
      "X-Forwarded-Host": "odo-odoooo.apps-crc.testing",                                                                                     
      "Host": "odo-odoooo.apps-crc.testing",                                                                                                 
      "Accept-Encoding": "gzip, deflate, br",                                                                                                
      "Referer": "https://odo-odoooo.apps-crc.testing/hub/home",                                                                             
      "Accept-Language": "en-us",                                                                                                            
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.0.5 Safari/605.1.15", 
      "Cookie": "jupyterhub-hub-login=[secret]; jupyterhub-session-id=[secret]",                                                             
      "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",                                                           
      "Connection": "close"                                                                                                                  
    }    ```

[jankleinert]

@GrahamDumpleton I also haven't been able to get any of our existing workshops that I've tried to work on 4.3. The ones I've tried are using the 2.x branch: https://github.com/openshift-labs/starter-guides/blob/ocp-4.2/.gitmodules https://github.com/openshift-labs/lab-tekton-pipelines/blob/master/.gitmodules https://github.com/openshift-labs/lab-developing-with-odo/blob/master/.gitmodules

[GrahamDumpleton]

Not sure what happened to reference to fix I thought I was using. I was part of the discussion back about this issue with Kubernetes 1.16 in October.

• https://github.com/jupyterhub/kubespawner/issues/354#issuecomment-542464427

It is possible I disabled the event stream and never came back to and updated to newer version after I identified the change for them , not sure.

Anyway, as a quick workaround, you can add to custom JUPYTERHUB_CONFIG template parameter when deploying:

c.KubeSpawner.events_enabled = False

I'll track down what happened to any prior fix.

[GrahamDumpleton]

Should be fixed.

For workshops using version 2.x of workshop-scripts, run:

git submodule update --remote

to refresh the copy of the scripts.

Then do a git add, commit, push etc of the workshop so that anyone checking out workshop in future will get the newer version of the scripts as well.

No strict need to update image version in doing this, but you could still update image version number an tag the workshop as appropriate.

Note that the default OpenShift console version is still 4.2.

[jankleinert]

@GrahamDumpleton thank you!!!