search

fluxcd / flux

Successor: https://github.com/fluxcd/flux2
https://fluxcd.io
Apache License 2.0
6.9k stars 1.08k forks source link

Flux support in Kubernetes 1.6? #516

Closed BenHall closed 6 years ago

BenHall commented 7 years ago

On Kubernetes 1.6, everything looks to deploy correctly, but I'm unable to view the images. Is this a known issue or something with my configuration?

$ fluxctl list-images --service=sock-shop/front-end
SERVICE  CONTAINER  IMAGE  CREATED

$ fluxctl release --service=sock-shop/front-end --update-all-images
Submitting release job...
Release job submitted, ID 607101aa-4ed4-7ca5-6b0c-10340bf0848e
Status: Complete.

Here's what happened:
SERVICE   STATUS   UPDATES
Took 16.119216s
awh commented 7 years ago

Thanks for the report! Looking into it now.

awh commented 7 years ago

@BenHall a few questions:

Thanks!

BenHall commented 7 years ago

This is related to Weave Cloud, using the latest release.

I applied a RBAC fix from the Cortex issue (https://github.com/weaveworks/cortex/issues/392#issuecomment-292713631). This allowed images to be viewed via list-images but rolling out changes would indicate 0 services required updating. This happened via the CLI and the web UI.

awh commented 7 years ago

Hrm, I could not replicate the '0 services required updating' issue - instead I get this:

$ fluxctl-0.2.0 --token=<REDACTED> list-images --service=sock-shop/front-end
SERVICE              CONTAINER  IMAGE                                      CREATED
sock-shop/front-end  front-end  index.docker.io/weaveworksdemos/front-end  
                                |   master-ac9ca707                        07 Apr 17 13:06 UTC
                                |   master-3fad5eda                        30 Mar 17 08:51 UTC
                                |   master-bdc6f3ff                        29 Mar 17 10:45 UTC
                                '-> 0.3.12                                 21 Mar 17 11:32 UTC
                                    latest                                 21 Mar 17 11:32 UTC
                                    master-b2308a3e                        21 Mar 17 11:24 UTC
                                    0.3.11                                 20 Mar 17 12:13 UTC
                                    master-fe7f9828                        17 Mar 17 14:24 UTC
                                    0.3.10                                 16 Mar 17 13:31 UTC
                                    master-30f06738                        16 Mar 17 13:30 UTC
$ fluxctl-0.2.0 --token=<REDACTED> release --service=sock-shop/front-end --update-all-images
Submitting release job...
Release job submitted, ID f7636083-65ff-3b4c-4b0c-958ce1eb1778
Status: Failed: sock-shop/front-end: applying definition to sock-shop/front-end: running kubectl: error: group map[federation:0xc8203caf50 :0xc8203cafc0 apps:0xc8203cb030 autoscaling:0xc8203cb110 componentconfig:0xc8203cb1f0 policy:0xc8203cb2d0 rbac.authorization.k8s.io:0xc8203cb340 authorization.k8s.io:0xc8203cb0a0 batch:0xc8203cb180 extensions:0xc8203cb260 authentication.k8s.io:0xc8203cb420] is already registered

Here's as far as we got:
 1) Queued.
 2) Calculating updates for release.
 3) Cloning git repository.
 4) Finding defined services.
 5) Found service sock-shop/front-end
 6) Found 1 services.
 7) Looking up images.
 8) Will update sock-shop/front-end container front-end: weaveworksdemos/front-end:0.3.12 -> master-ac9ca707
 9) Pushing changes.
 10) Applying changes.
 11) Sending notifications.
 12) Failed: sock-shop/front-end: applying definition to sock-shop/front-end: running kubectl: error: group map[federation:0xc8203caf50 :0xc8203cafc0 apps:0xc8203cb030 autoscaling:0xc8203cb110 componentconfig:0xc8203cb1f0 policy:0xc8203cb2d0 rbac.authorization.k8s.io:0xc8203cb340 authorization.k8s.io:0xc8203cb0a0 batch:0xc8203cb180 extensions:0xc8203cb260 authentication.k8s.io:0xc8203cb420] is already registered

Took 19.718682s
== Error ==

Internal error: sock-shop/front-end: applying definition to sock-shop/front-end: running kubectl: error: group map[federation:0xc8203caf50 :0xc8203cafc0 apps:0xc8203cb030 autoscaling:0xc8203cb110 componentconfig:0xc8203cb1f0 policy:0xc8203cb2d0 rbac.authorization.k8s.io:0xc8203cb340 authorization.k8s.io:0xc8203cb0a0 batch:0xc8203cb180 extensions:0xc8203cb260 authentication.k8s.io:0xc8203cb420] is already registered

We don't have a specific help message for the error above.

It would help us remedy this if you log an issue at

    https://github.com/weaveworks/flux/issues

saying what you were doing when you saw this, and quoting the message
at the top.

This is with flux 0.2.0 installed on a 1.6.0 k8s cluster via the launch generator:

kubectl apply -f 'https://cloud.weave.works/k8s/flux.yaml?service-token=<REDACTED>'

which includes the required service account.

ISTM that the 'group already registered' error is due to an incompatibility with the kubectl shipped inside fluxd (v1.3.5) - I'm about to open a PR that updates that, and with which I have successfully released sock-shop/front-end to a 1.6 cluster...

BenHall commented 7 years ago

The service 0 was from the Flux / Weave Cloud UI

Release via fluxctl 0.2.0

root@master:~/weave-front-end# fluxctl release --service=sock-shop/front-end --update-all-images
Submitting release job...
Release job submitted, ID 15dc7bd0-2ea5-2c2f-9621-2cf1192c33a7
Status: Complete.

Here's what happened:
SERVICE   STATUS   UPDATES
Took 18.771451s
root@master:~/weave-front-end# fluxctl list-images --service=sock-shop/front-end
SERVICE              CONTAINER  IMAGE                                                             CREA
TED
sock-shop/front-end  front-end  2886795273-4567-kitek01.environments.katacoda.com/root/front-end
                                |   1d8e9cd73cc781c91aa4fcbc87afda2a464da343                      10 A
pr 17 16:50 UTC
                                '-> latest                                                        28 O
ct 16 20:22 UTC
root@master:~/weave-front-end#

 DryRun via Weave Cloud

 Dry Run: Calculating plan to release sock-shop/front-end
Release job submitted, ID 855b5454-0df1-abdd-f381-9c10287f4b2b
Status: Complete.

1) Queued.
2) Calculating updates for release.
3) Cloning git repository.
4) Finding defined services.
5) Found 0 services.
6) Looking up images.
7) No updates to do, finishing.
Success

 Release

 Releasing sock-shop/front-end
Release job submitted, ID 000891f0-0c74-e6df-477e-d18665a180fc
Status: Complete.

1) Queued.
2) Calculating updates for release.
3) Cloning git repository.
4) Finding defined services.
5) Found 0 services.
6) Looking up images.
7) No updates to do, finishing.
Success
awh commented 7 years ago

Sorry, didn't mean to close this yet - you clearly have some other issue as well (although you would hit the k8s 1.6 problem once we get past whatever it is)

awh commented 7 years ago

@BenHall can you paste the result of running kubectl --namespace sock-shop get all -o wide against that cluster? Thanks

BenHall commented 7 years ago 
NAME                               READY     STATUS              RESTARTS   AGE       IP          NODE
po/carts-153328538-fvncf           1/1       Running             0          6m        10.40.0.2   node
po/carts-db-1968389027-wlpvg       1/1       Running             0          6m        10.40.0.1   node
po/catalogue-114596073-w6h21       1/1       Running             0          6m        10.40.0.7   node
po/catalogue-db-1956862931-l4ktv   1/1       Running             0          6m        10.40.0.4   node
po/front-end-537982664-s53gw       1/1       Running             0          5m        10.40.0.3   node
po/orders-2365168879-mx6b8         1/1       Running             0          5m        10.40.0.6   node
po/orders-db-3036494983-88vc9      1/1       Running             0          5m        10.40.0.5   node
po/payment-1968871107-rfgpn        1/1       Running             0          5m        10.40.0.8   node
po/queue-master-2798459664-bfz41   0/1       ContainerCreating   0          5m        <none>      node
po/rabbitmq-2262854187-ghnld       0/1       ContainerCreating   0          5m        <none>      node
po/shipping-2899287913-51t6g       0/1       ContainerCreating   0          5m        <none>      node
po/user-468431046-ff8sx            0/1       ContainerCreating   0          5m        <none>      node
po/user-db-1166754267-t12k1        0/1       ContainerCreating   0          5m        <none>      node
NAME               CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE       SELECTOR
svc/carts          10.107.86.207    <none>         80/TCP         6m        name=carts
svc/carts-db       10.103.227.2     <none>         27017/TCP      6m        name=carts-db
svc/catalogue      10.108.231.190   <none>         80/TCP         6m        name=catalogue
svc/catalogue-db   10.101.213.150   <none>         3306/TCP       6m        name=catalogue-db
svc/front-end      10.107.188.158   ,172.17.0.13   80:30284/TCP   5m        name=front-end
svc/orders         10.111.22.215    <none>         80/TCP         5m        name=orders
svc/orders-db      10.100.28.123    <none>         27017/TCP      5m        name=orders-db
svc/payment        10.104.246.205   <none>         80/TCP         5m        name=payment
svc/queue-master   10.97.136.240    <none>         80/TCP         5m        name=queue-master
svc/rabbitmq       10.111.195.90    <none>         5672/TCP       5m        name=rabbitmq
svc/shipping       10.107.14.149    <none>         80/TCP         5m        name=shipping
svc/user           10.103.100.168   <none>         80/TCP         5m        name=user
svc/user-db        10.106.209.225   <none>         27017/TCP      5m        name=user-db
NAME                  DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE       CONTAINER(S)   IMAGE(S)                                                           SELECTOR
deploy/carts          1         1         1            1           6m        carts          weaveworksdemos/carts:0.4.8                                        name=carts
deploy/carts-db       1         1         1            1           6m        carts-db       mongo:3.2.11                                                       name=carts-db
deploy/catalogue      1         1         1            1           6m        catalogue      weaveworksdemos/catalogue:0.3.5                                    name=catalogue
deploy/catalogue-db   1         1         1            1           6m        catalogue-db   weaveworksdemos/catalogue-db:0.3.0                                 name=catalogue-db
deploy/front-end      1         1         1            1           5m        front-end      2886795275-4567-kitek01.environments.katacoda.com/root/front-end   name=front-end
deploy/orders         1         1         1            1           5m        orders         weaveworksdemos/orders:0.4.7                                       name=orders
deploy/orders-db      1         1         1            1           5m        orders-db      mongo:3.2.11                                                       name=orders-db
deploy/payment        1         1         1            1           5m        payment        weaveworksdemos/payment:0.4.3                                      name=payment
deploy/queue-master   1         1         1            0           5m        queue-master   weaveworksdemos/queue-master:0.3.1                                 name=queue-master
deploy/rabbitmq       1         1         1            0           5m        rabbitmq       rabbitmq:3                                                         name=rabbitmq
deploy/shipping       1         1         1            0           5m        shipping       weaveworksdemos/shipping:0.4.8                                     name=shipping
deploy/user           1         1         1            0           5m        user           weaveworksdemos/user:0.4.4                                         name=user
deploy/user-db        1         1         1            0           5m        user-db        weaveworksdemos/user-db:0.4.0                                      name=user-db
NAME                         DESIRED   CURRENT   READY     AGE       CONTAINER(S)   IMAGE(S)                                                           SELECTOR
rs/carts-153328538           1         1         1         6m        carts          weaveworksdemos/carts:0.4.8                                        name=carts,pod-template-hash=153328538
rs/carts-db-1968389027       1         1         1         6m        carts-db       mongo:3.2.11                                                       name=carts-db,pod-template-hash=1968389027
rs/catalogue-114596073       1         1         1         6m        catalogue      weaveworksdemos/catalogue:0.3.5                                    name=catalogue,pod-template-hash=114596073
rs/catalogue-db-1956862931   1         1         1         6m        catalogue-db   weaveworksdemos/catalogue-db:0.3.0                                 name=catalogue-db,pod-template-hash=1956862931
rs/front-end-537982664       1         1         1         5m        front-end      2886795275-4567-kitek01.environments.katacoda.com/root/front-end   name=front-end,pod-template-hash=537982664
rs/orders-2365168879         1         1         1         5m        orders         weaveworksdemos/orders:0.4.7                                       name=orders,pod-template-hash=2365168879
rs/orders-db-3036494983      1         1         1         5m        orders-db      mongo:3.2.11                                                       name=orders-db,pod-template-hash=3036494983
rs/payment-1968871107        1         1         1         5m        payment        weaveworksdemos/payment:0.4.3                                      name=payment,pod-template-hash=1968871107
rs/queue-master-2798459664   1         1         0         5m        queue-master   weaveworksdemos/queue-master:0.3.1                                 name=queue-master,pod-template-hash=2798459664
rs/rabbitmq-2262854187       1         1         0         5m        rabbitmq       rabbitmq:3                                                         name=rabbitmq,pod-template-hash=2262854187
rs/shipping-2899287913       1         1         0         5m        shipping       weaveworksdemos/shipping:0.4.8                                     name=shipping,pod-template-hash=2899287913
rs/user-468431046            1         1         0         5m        user           weaveworksdemos/user:0.4.4                                         name=user,pod-template-hash=468431046
rs/user-db-1166754267        1         1         0         5m        user-db        weaveworksdemos/user-db:0.4.0                                      name=user-db,pod-template-hash=1166754267
awh commented 7 years ago

@BenHall thanks. Is the config repository public?

BenHall commented 7 years ago

It's part of Katacoda so it's only short lived. It is public at the time. Why do you ask? To reproduce?

awh commented 7 years ago

To reproduce?

Yes. The set of services listed by list-images is the one obtained from k8s, whereas the services used by release are further filtered by what's defined in the config repo. Since the former is successfully finding sock-shop/front-end and it is missing from the latter ('Found 0 services.'), the finger points at the config repo...

I gather from the subject of your issue that you think the problem is caused by k8s 1.6 - is there a way to check that this doesn't also happen with earlier versions?

BenHall commented 7 years ago

The current Katacoda (https://www.weave.works/docs/tutorials/core/interactive/part-2-deploy-continuous-delivery/) is using v1.5 and works as expected.

The same repository with a Kubernetes v1.6 cluster fails

fluxctl release --service=sock-shop/front-end --update-all-images
Submitting release job...
Release job submitted, ID 97e450be-09e3-cfba-081a-1b84c57a0b08
Status: Complete.

Here's what happened:
SERVICE              STATUS   UPDATES
sock-shop/front-end  skipped  not found in repository
Took 10.98256s
BenHall commented 7 years ago

A copy of the microservices-demo repo we use can be found at http://assets.joinscrapbook.com/weavecloud/weave-micro-0604.tar.

Would the fact that I'm using :latest be causing an issue? I noticed the image Kubernetes returns is 2886795275-4567-kitek01.environments.katacoda.com/root/front-end

samb1729 commented 6 years ago

Flux 0. is no longer supported. Please file a new issue if you are having difficulties with Flux 1., Ben.