search

keel-hq / keel

Kubernetes Operator to automate Helm, DaemonSet, StatefulSet & Deployment updates
https://keel.sh
Mozilla Public License 2.0
2.46k stars 282 forks source link

Initial deploy not update image if need image without additional tag but last image in repository have an additional tag. #271

Closed compozitor closed 5 years ago

compozitor commented 6 years ago

@rusenask If last container in repository have a version tag with additional metadata such as 3.0.0-dev but keel needs update container with tag without additional metadata such as 3.0.1 then update don't work.

The reverse is also true.

If last container in repository have a version tag with additional metadata such as 3.0.0-dev and keel needs update container with tag with additional metadata such as 3.0.1-dev then update successfully.

@rusenask Can you make that the keel does not stop on the last image in the repository, but looks for the appropriate last image with or without an additional tag, depending on the container being updated?

Do you mean if there are 1.5.0-dev and 1.6.0 available, keel won't upgrade your deployment if it's currently 1.0.0-dev? Do I understand this correctly?

Yes right.

@rusenask Such a situation arises when the initial deploy. Then, if the required containers 1.6.0-dev appear in the repository, everything is updated normally.

time="2018-09-06T14:34:22Z" level=info msg="bot: registered" name=slack
time="2018-09-06T14:34:22Z" level=info msg="keel starting..." arch=amd64 build_date=2018-09-06T125536Z go_version=go1.10.3 os=linux revision=03510c10 version=0.10.0
time="2018-09-06T14:34:22Z" level=info msg="provider.kubernetes: using in-cluster configuration"
time="2018-09-06T14:34:22Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"
time="2018-09-06T14:34:22Z" level=info msg="extension.credentialshelper: helper registered" name=secrets
time="2018-09-06T14:34:22Z" level=info msg="bot.slack.Configure(): Slack approval bot is not configured"
time="2018-09-06T14:34:22Z" level=error msg="bot.Run(): can not get configuration for bot [slack]"
time="2018-09-06T14:34:22Z" level=info msg="webhook trigger server starting..." port=9300
time="2018-09-06T14:34:22Z" level=info msg=started context=buffer
time="2018-09-06T14:34:22Z" level=info msg=started context=watch resource=deployments
time="2018-09-06T14:34:22Z" level=info msg="trigger.poll.manager: polling trigger configured"
time="2018-09-06T14:34:22Z" level=info msg=started context=watch resource=cronjobs
time="2018-09-06T14:34:22Z" level=info msg=started context=watch resource=statefulsets
time="2018-09-06T14:34:22Z" level=info msg=started context=watch resource=daemonsets
time="2018-09-06T14:35:05Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:93ddbd957274b9f6437814fd8ced73b53d920b1c6f98ba8de980e2ac8c402113" image="anatoliy/mysql-nafta:0.0.1" job_name="git.technotrade.ua:4567/anatoliy/mysql-nafta" schedule="@every 1m"
time="2018-09-06T14:35:05Z" level=info msg="trigger.poll.WatchRepositoryTagsJob: submiting event to providers" new_tag=0.0.3-dev repository="git.technotrade.ua:4567/anatoliy/mysql-nafta"
time="2018-09-06T14:35:05Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta" tag=0.0.3-dev
time="2018-09-06T14:35:05Z" level=info msg="provider.kubernetes.checkVersionedDeployment: keel policy found, checking resource..." kind=deployment name=server-nafta-mysql namespace=technotrade policy=major
time="2018-09-06T14:35:05Z" level=info msg="provider.kubernetes: checking image" image="git.technotrade.ua:4567/anatoliy/mysql-nafta:0.0.1" kind=deployment name=server-nafta-mysql namespace=technotrade parsed_image_name="git.technotrade.ua:4567/anatoliy/mysql-nafta:0.0.1" policy=major target_image_name="git.technotrade.ua:4567/anatoliy/mysql-nafta" target_tag=0.0.3-dev
time="2018-09-06T14:35:05Z" level=info msg="provider.kubernetes: no plans for deployment updates found for this event" image="git.technotrade.ua:4567/anatoliy/mysql-nafta" tag=0.0.3-dev
rusenask commented 6 years ago

the culprit is this function https://github.com/keel-hq/keel/blob/master/util/version/version.go#L78-L115, I don't have time to fix it at the moment, maybe over the weekend or next week :)

compozitor commented 6 years ago

OK I'll wait :)

rusenask commented 6 years ago

I am still on it, just wanted to give an update. It's getting there but require quite a bit of changes and trying to avoid hitting API rate limits :)

compozitor commented 6 years ago

@rusenask In version 0.12.0, the problem went nowhere.

rusenask commented 6 years ago

Hi, the scope really increased of the issue and I don't have a lot of time these days to work on Keel. PR is still in progress.

rusenask commented 6 years ago

Merged, will be available in the next release.

compozitor commented 5 years ago

@rusenask I was try version 0.13.0-rc2 The keel of this version does not work as I would like. I have in the repository an images with tags and without: 0.0.1, 0.0.2, 0.0.3, 0.0.1-dev, 0.0.2-dev, 0.0.3-dev, 0.0.4-dev, 0.0.5-dev After deploying the image 0.0.1 automatic update does not occur. The keel does not see version 0.0.3 but only sees the last 0.0.5-dev.

time="2018-11-26T14:35:56Z" level=info msg="extension.credentialshelper: helper registered" name=aws
time="2018-11-26T14:35:56Z" level=info msg="bot: registered" name=slack
time="2018-11-26T14:35:56Z" level=info msg="keel starting..." arch=amd64 build_date=2018-11-24T235258Z go_version=go1.10.3 os=linux revision=0519fd8d version=0.13.0-rc2
time="2018-11-26T14:35:56Z" level=info msg="provider.kubernetes: using in-cluster configuration"
time="2018-11-26T14:35:56Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"
time="2018-11-26T14:35:56Z" level=info msg="extension.credentialshelper: helper registered" name=secrets
time="2018-11-26T14:35:56Z" level=info msg="trigger.poll.manager: polling trigger configured"
time="2018-11-26T14:35:56Z" level=info msg="bot.slack.Configure(): Slack approval bot is not configured"
time="2018-11-26T14:35:56Z" level=error msg="bot.Run(): can not get configuration for bot [slack]"
time="2018-11-26T14:35:56Z" level=info msg="webhook trigger server starting..." port=9300
time="2018-11-26T14:35:56Z" level=info msg=started context=watch resource=statefulsets
time="2018-11-26T14:35:56Z" level=info msg=started context=buffer
time="2018-11-26T14:35:56Z" level=info msg=started context=watch resource=daemonsets
time="2018-11-26T14:35:56Z" level=info msg=started context=watch resource=cronjobs
time="2018-11-26T14:35:56Z" level=info msg=started context=watch resource=deployments
time="2018-11-26T14:36:01Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:da3ae76cb7b367751d096f95cb59253a3841e49c17b0b7869d834510d9322414" image="anatoliy/mysql-nafta:0.0.1" job_name="git.technotrade.ua:4567/anatoliy/mysql-nafta" schedule="@every 1m"
time="2018-11-26T14:36:01Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:bc833ebbb5c204a34de93b36567eea5543bcb04114040ee15c532cebfb126dd0" image="future-technologies/resturlstransmitter:0.0.1" job_name="git.technotrade.ua:4567/future-technologies/resturlstransmitter" schedule="@every 1m"
time="2018-11-26T14:36:02Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:efa25965a808cd6c995d90f5785c00e120a307d40858dc512d3f0915fb959f89" image="future-technologies/web-gui:0.0.1" job_name="git.technotrade.ua:4567/future-technologies/web-gui" schedule="@every 1m"
time="2018-11-26T14:36:02Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:7a22de3405f00f23aee4451ab09d55e9e12a053ea40041ecd0ef719b7a09297f" image="anatoliy/mysql-nafta-update:0.0.1" job_name="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" schedule="@every 1m"
time="2018-11-26T14:36:03Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.5-dev
time="2018-11-26T14:37:03Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.5-dev
time="2018-11-26T14:38:03Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.5-dev
rusenask commented 5 years ago

Hi, can you have a look at these tests https://github.com/keel-hq/keel/blob/master/tests/acceptance_polling_test.go#L47-L105 ? here are the tags https://hub.docker.com/r/keelhq/push-workflow-example/tags/

Test is testing several cases where we have both a version with pre-release tag and without.

compozitor commented 5 years ago

I added an image version 0.0.6 to the repository. After the image 0.0.1-dev was deployed, the 0.0.1-dev image was automatically updated to 0.0.5-dev. The situation modeled in the test works for me. Try to simulate the situation in the test that I quoted above.

time="2018-11-26T17:00:34Z" level=info msg="extension.credentialshelper: helper registered" name=aws
time="2018-11-26T17:00:34Z" level=info msg="bot: registered" name=slack
time="2018-11-26T17:00:34Z" level=info msg="keel starting..." arch=amd64 build_date=2018-11-24T235258Z go_version=go1.10.3 os=linux revision=0519fd8d version=0.13.0-rc2
time="2018-11-26T17:00:34Z" level=info msg="provider.kubernetes: using in-cluster configuration"
time="2018-11-26T17:00:34Z" level=info msg="provider.defaultProviders: provider 'kubernetes' registered"
time="2018-11-26T17:00:34Z" level=info msg="extension.credentialshelper: helper registered" name=secrets
time="2018-11-26T17:00:34Z" level=info msg="trigger.poll.manager: polling trigger configured"
time="2018-11-26T17:00:34Z" level=info msg="bot.slack.Configure(): Slack approval bot is not configured"
time="2018-11-26T17:00:34Z" level=error msg="bot.Run(): can not get configuration for bot [slack]"
time="2018-11-26T17:00:34Z" level=info msg="webhook trigger server starting..." port=9300
time="2018-11-26T17:00:34Z" level=info msg=started context=buffer
time="2018-11-26T17:00:34Z" level=info msg=started context=watch resource=deployments
time="2018-11-26T17:00:34Z" level=info msg=started context=watch resource=statefulsets
time="2018-11-26T17:00:34Z" level=info msg=started context=watch resource=daemonsets
time="2018-11-26T17:00:34Z" level=info msg=started context=watch resource=cronjobs
time="2018-11-26T17:00:40Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:da3ae76cb7b367751d096f95cb59253a3841e49c17b0b7869d834510d9322414" image="anatoliy/mysql-nafta:0.0.1" job_name="git.technotrade.ua:4567/anatoliy/mysql-nafta" schedule="@every 1m"
time="2018-11-26T17:00:41Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:bc833ebbb5c204a34de93b36567eea5543bcb04114040ee15c532cebfb126dd0" image="future-technologies/resturlstransmitter:0.0.1" job_name="git.technotrade.ua:4567/future-technologies/resturlstransmitter" schedule="@every 1m"
time="2018-11-26T17:00:42Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:efa25965a808cd6c995d90f5785c00e120a307d40858dc512d3f0915fb959f89" image="future-technologies/web-gui:0.0.1" job_name="git.technotrade.ua:4567/future-technologies/web-gui" schedule="@every 1m"
time="2018-11-26T17:00:43Z" level=info msg="trigger.poll.RepositoryWatcher: new watch repository tags job added" digest="sha256:7fb2cc3785e34e118eba0ca62910c21fc6771f9da4b76018d9d5333a5d3f91a6" image="anatoliy/mysql-nafta-update:0.0.1-dev" job_name="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" schedule="@every 1m"
time="2018-11-26T17:00:43Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.6
time="2018-11-26T17:00:43Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.5-dev
time="2018-11-26T17:00:43Z" level=info msg="provider.kubernetes: impacted deployment container found" parsed_image="git.technotrade.ua:4567/anatoliy/mysql-nafta-update:0.0.1-dev" policy=major raw_image_name="git.technotrade.ua:4567/anatoliy/mysql-nafta-update:0.0.1-dev" target_image="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" target_image_tag=0.0.5-dev
time="2018-11-26T17:00:43Z" level=info msg="provider.kubernetes: resource updated" kind=deployment name=update-nafta-mysql namespace=technotrade new=0.0.5-dev previous=0.0.1-dev
time="2018-11-26T17:01:45Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.6
time="2018-11-26T17:02:43Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.6
time="2018-11-26T17:03:43Z" level=info msg="provider.kubernetes: processing event" registry= repository="git.technotrade.ua:4567/anatoliy/mysql-nafta-update" tag=0.0.6
compozitor commented 5 years ago

@rusenask A test without a pre-release tag gives a positive result for the reason that the maximum version in the repository is 0.10.0. Add an image with version 0.11.0-dev to the repository and test again.

rusenask commented 5 years ago

thanks, replicated

rusenask commented 5 years ago

Hi @compozitor just did a bit of a refactoring to the polling trigger. Now passing policy to the watcher and getting all tags there as well.

Additionally, added an e2e test for this case when there's a higher level tag with a prerelease for it and it did pass. Can you try out alpha tag?

docker push keelhq/keel:alpha
The push refers to repository [docker.io/keelhq/keel]
a58c4fa39102: Pushed 
465d99f26841: Layer already exists 
df64d3292fd6: Layer already exists 
alpha: digest: sha256:1b6726390046e4b4dfd1adba3476ddd9f611280d7f74385d20589f92e194d756 size: 949
compozitor commented 5 years ago

Hi @rusenask I tried the image with alpha tag. Thank you very much!!! Everything is working.