teamhephy / controller

Hephy Workflow Controller (API)
https://teamhephy.com
MIT License
14 stars 26 forks source link

set deployment API to apps/v1 if k8s >= 1.9 #86

Closed n0n0x closed 5 years ago

n0n0x commented 5 years ago

this closes #85 this closes teamhephy/workflow#57

related issue worth reading: teamhephy/workflow#57

To sum this up based on the discussion that took place at issue teamhephy/workflow#57:

What we're seeking here is the same version on the HPA that the Deployment is defined as. The HPA cannot be looking for the Deployment in an API group/version that it is not in. I recommend changing your Deployment to the apiVersion that your version uses by default. For 1.9 looks like it's actually apps/v1. source

Looks like the issue here was not that Hephy was using the wrong HPA api version, but that deployments api compatibility has changed. Sounds that deployment API version will need to be migrated to apps/v1 . source

I guess this will actually require manual editing of each existing deployment using kubectl -n <namespace> edit deploy <deployment-name> and change the apiVersion to apps/v1 for deis autoscale to work properly on old deployments, right?

Cryptophobia commented 5 years ago

Have you tested this @n0n0x on any cluster yet?

I'm okay with having to manually edit to fix this for existing deployments as upgrading clusters to newer versions is a manual process anyways. :+1:

kingdonb commented 5 years ago

We're getting reports in slack that this patch has problems when the version is reported as 1.11+, which happens on some clusters.

api.exceptions.DeisException: Invalid version: '1.11+'
Traceback (most recent call last):
  File "/app/api/models/build.py", line 65, in create
    self.app.deploy(new_release)
  File "/app/api/models/app.py", line 517, in deploy
    self._check_deployment_in_progress(deploys, force_deploy)
  File "/app/api/models/app.py", line 603, in _check_deployment_in_progress
    kwargs.get("replicas"), kwargs.get("tags")
  File "/app/scheduler/resources/deployment.py", line 204, in in_progress
    ready, _ = self.are_replicas_ready(namespace, name)
  File "/app/scheduler/resources/deployment.py", line 264, in are_replicas_ready
    deployment = self.get(namespace, name).json()
  File "/app/scheduler/resources/deployment.py", line 34, in get
    url = self.api(url, *args)
  File "/app/scheduler/resources/__resource.py", line 28, in api
    return "/{}/{}".format(self.api_prefix, self.api_version) + tmpl.format(*args)
  File "/app/scheduler/resources/deployment.py", line 16, in api_version
    if self.version() >= parse("1.9.0"):
  File "/app/scheduler/__init__.py", line 87, in version
    return Version('{}.{}'.format(data['major'], data['minor']))
  File "/usr/local/lib/python3.5/dist-packages/packaging/version.py", line 202, in __init__
    raise InvalidVersion("Invalid version: '{0}'".format(version))
tolstenko commented 5 years ago

Thanks @kingdonb! I am facing this problem. I upgraded from v2.20.0 to v2.20.1 and now I am having this issue @kingdonb kindly described here. Here is the kubectl version output BTW, I am using google GKE

Client Version: version.Info{Major:"1", Minor:"10", GitVersion:"v1.10.7", GitCommit:"0c38c362511b20a098d7cd855f1314dad92c2780", GitTreeState:"clean", BuildDate:"2018-08-20T10:09:03Z", GoVersion:"go1.9.3", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"11+", GitVersion:"v1.11.6-gke.2", GitCommit:"04ad69a117f331df6272a343b5d8f9e2aee5ab0c", GitTreeState:"clean", BuildDate:"2019-01-04T16:19:46Z", GoVersion:"go1.10.3b4", Compiler:"gc", Platform:"linux/amd64"}
kingdonb commented 5 years ago

These version numbers are not good, but we've seen this before:

https://github.com/teamhephy/controller/pull/73