Kubespray deployment updated to Kubernetes 1.17.6

[zonca]

I updated Kubespray from 2.8.2 to 2.13.1, which installs a very recent Kubernetes 1.17.6.

Made a new version of the tutorial:

https://zonca.dev/2020/06/kubernetes-jetstream-kubespray.html

@julienchastang @fengggli would you like to test it and provide feedback? Also improvements/clarifications on the docs would be great, you can make a PR directly to https://github.com/zonca/zonca-blog/blob/master/_posts/2020-06-15-jetstream_kubernetes_kubespray.md

[zonca]

@julienchastang this doesn't interfere at all with version 2.8.2, so you can continue to use that for the time being.

[julienchastang]

OK, I'll try to provide feedback shortly.

[fengggli]

Hi @zonca, I tried the branch v2.13.1 in IU jetstream following the new documentation, it works fine.

[julienchastang]

cluster.tf is now cluster.tfvars. Is that intentional?

[zonca]

Yes, that is the convention with the newer versions of kubespray

On Wed, Jun 17, 2020, 11:51 Julien Chastang notifications@github.com wrote:

cluster.tf is now cluster.tfvars. Is that intentional?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/33#issuecomment-645556948, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5Q4VIFFL5KI57ZJFOPDDRXEGDRANCNFSM4N7J2VSA .

[julienchastang]

Please update this in https://zonca.dev/2020/06/kubernetes-jetstream-kubespray.html that still refers to cluster.tf. Thanks.

[zonca]

good catch, fixed, thanks!

On Wed, Jun 17, 2020 at 2:48 PM Julien Chastang notifications@github.com wrote:

Please update this in https://zonca.dev/2020/06/kubernetes-jetstream-kubespray.html that still refers to cluster.tf. Thanks.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/issues/33#issuecomment-645643214, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAC5Q4S2NIURNUF6GOUKGZ3RXE23DANCNFSM4N7J2VSA .

[julienchastang]

Warning: Quoted type constraints are deprecated

  on ../../contrib/terraform/openstack/variables.tf line 134, in variable "network_dns_domain":
 134:   type        = "string"

Warning: Interpolation-only expressions are deprecated

  on ../../contrib/terraform/openstack/modules/compute/main.tf line 11, in resource "openstack_compute_keypair_v2" "k8s":
  11:   public_key = "${chomp(file(var.public_key_path))}"

[zonca]

yeah, those are in Kubespray itself, as they are only warning, I don't think we need to fix them, hope Kubespray will fix them in next release.

[julienchastang]

  File "/usr/lib/python2.7/site-packages/openstack/utils.py", line 13, in <module>
    import queue
ImportError: No module named queue

Is something missing from requirements.txt?

[zonca]

python 2.7??

[julienchastang]

My information may be out-of-date, but yes, last time I checked OpenStack was still Python 2. "Currently, the clients do not support Python 3" :-(

[zonca]

that must be an error in the docs

python
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openstack
>>> openstack.version.__version__
'0.46.0'

[zonca]

ah no I think the right package is:

python
Python 3.7.3 (default, Mar 27 2019, 22:11:17) 
[GCC 7.3.0] :: Anaconda, Inc. on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import openstackclient
>>> openstackclient.__version__
'3.18.0'

[zonca]

definitely they support Python 3, see the PyPI page: https://pypi.org/project/python-openstackclient/

[zonca]

I plan to update the tutorial to use supplementary_addresses_in_ssl_keys, see https://github.com/zonca/jetstream_kubespray/pull/13#issuecomment-645011752

[julienchastang]

In the end, I ran into this issue: https://github.com/jupyterhub/kubespawner/issues/354

[E 2020-07-02 23:48:24.322 JupyterHub log:158] 500 GET /hub/user/julienchastang/ (julienchastang@216.160.128.60) 297.09ms
[I 2020-07-02 23:48:26.115 JupyterHub log:158] 200 GET /hub/admin (julienchastang@216.160.128.60) 519.18ms
[E 2020-07-02 23:48:39.481 JupyterHub user:477] Unhandled error starting julienchastang's server: '<' not supported between instances of 'datetime.datetime' and 'NoneType'
[E 2020-07-02 23:48:40.307 JupyterHub gen:593] Exception in Future <Task finished coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py:629> exception=TypeError("'<' not supported between instances of 'datetime.datetime' and 'NoneType'",)> after timeout
    Traceback (most recent call last):
      File "/usr/local/lib/python3.6/dist-packages/tornado/gen.py", line 589, in error_callback
        future.result()
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/handlers/base.py", line 636, in finish_user_spawn
        await spawn_future
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 489, in spawn
        raise e
      File "/usr/local/lib/python3.6/dist-packages/jupyterhub/user.py", line 409, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1636, in _start
        events = self.events
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 1491, in events
        for event in self.event_reflector.events:
      File "/usr/local/lib/python3.6/dist-packages/kubespawner/spawner.py", line 72, in events
        key=lambda x: x.last_timestamp,
    TypeError: '<' not supported between instances of 'datetime.datetime' and 'NoneType'

[W 2020-07-02 23:48:40.308 JupyterHub base:699] 4 consecutive spawns failed.  Hub will exit if failure count reaches 5 before succeeding
[W 2020-07-02 23:48:40.308 JupyterHub web:1782] 500 POST /hub/api/users/julienchastang/server (216.160.128.60): Spawner failed to start [status=1]. The logs for julienchastang may contain details.
[E 2020-07-02 23:48:40.310 JupyterHub log:150] {
      "Cookie": "jupyterhub-hub-login=\"2|1:0|10:1593733166|20:jupyterhub-hub-login|44:N2FjZTJlYzk4NmY4NDVmMThiOTI0M2M5ZDBlOGVkZTE=|7e8c7d059d42cdd2554c7a00d4ee52f4429dc096127c3fa334e87f9a55beb040\"; _ga=GA1.2.488392192.1586536558; jupyterhub-session-id=c00600d64c154b1db7c74aace60ad5bc",
      "Accept-Language": "en-US,en;q=0.9,fr;q=0.8",
      "Accept-Encoding": "gzip, deflate, br",
      "Referer": "https://js-171-160.jetstream-cloud.org/hub/admin",
      "Sec-Fetch-Dest": "empty",
      "Sec-Fetch-Mode": "cors",
      "Sec-Fetch-Site": "same-origin",
      "Origin": "https://js-171-160.jetstream-cloud.org",
      "Content-Type": "application/json",
      "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36",
      "X-Requested-With": "XMLHttpRequest",
      "Accept": "*/*",
      "Content-Length": "0",
      "X-Scheme": "https",
      "X-Forwarded-Proto": "https,http",
      "X-Forwarded-Port": "443,80",
      "X-Forwarded-Host": "js-171-160.jetstream-cloud.org",
      "X-Forwarded-For": "216.160.128.60,10.233.64.0",
      "X-Real-Ip": "216.160.128.60",
      "X-Request-Id": "ed0690fb72130969128971b107eec327",
      "Host": "js-171-160.jetstream-cloud.org",
      "Connection": "close"
    }
[E 2020-07-02 23:48:40.310 JupyterHub log:158] 500 POST /hub/api/users/julienchastang/server (julienchastang@216.160.128.60) 2504.12ms

[julienchastang]

The proposed fix of moving to 0.9.0-beta.3 actually seems to work though it took a bit of time for the hub to come up.

[zonca]

why not 0.9.0? that is now default in https://github.com/zonca/jupyterhub-deploy-kubernetes-jetstream/blob/master/install_jhub.sh

[zonca]

so this seems to be working fine for now, I'm closing this, please open another issue if anything breaks!