search

fluxcd / flux

Successor: https://github.com/fluxcd/flux2
https://fluxcd.io
Apache License 2.0
6.9k stars 1.08k forks source link

Flux not applying new yamls.. #2463

Closed arjunsbabu closed 5 years ago

arjunsbabu commented 5 years ago

Describe the bug I have flux installed and its running for many days without any issues till last friday. After that i have noticed flux have restarted almost 113 times and new yamls are not properly applied. A restart of flux solved the issues temporarily but the errors are still coming in yaml.

To Reproduce Steps to reproduce the behaviour:

  1. What's your setup? flux is deployed on openshift 3.11. Not sure how to reproduce..it came after few days

Expected behavior No error messages and flux should apply all yamls automaticallu

Logs

arjun@DESKTOP-QO0863U:~/workspace/git/fluxdeployment/deploy$ oc logs flux-9bccc6b8b-v2kqs -f
ts=2019-09-23T07:29:12.995355588Z caller=main.go:225 version=1.13.3
ts=2019-09-23T07:29:12.995442824Z caller=main.go:317 msg="using in cluster config to connect to the cluster"
ts=2019-09-23T07:29:13.033172705Z caller=main.go:396 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-09-23T07:29:13.033213685Z caller=main.go:397 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQAJ2WsGuu9fjFK4RunN5lmHc4qFMWssiLVjhIJpuAXsRm2AMwMg9+sFiaEJLfmGV1Iczo+k955sT5lG0dZbYLVLNJGHdEllYIKHxVi9+n+k+hQqT0feaDK64wZYRpfQ3K44LO8/5tzTGBzP5aDXsl1BnV7EQmZkZ1sp1X2krz5fRrCo3uYjJnaKz2uashEBW1IYisEI8keqeW069xc3U0OHhHE2G/bjMHI9utwB1SpoAtI7fYJTYHsNKIF+VCOdl1eDjhY7cXP2G1zg4jwNyP9ZduZDKat"
ts=2019-09-23T07:29:13.03324022Z caller=main.go:402 host=https://172.30.0.1:443 version=kubernetes-v1.11.0+d4cacc0
ts=2019-09-23T07:29:13.033287391Z caller=main.go:414 kubectl=/usr/local/bin/kubectl
ts=2019-09-23T07:29:13.034002686Z caller=main.go:426 ping=true
ts=2019-09-23T07:29:13.074933078Z caller=main.go:562 url=git@github.ourgit.com:openshiftha/dev-projects.git user="Weave Flux" email=support@weave.works signing-key= verify-signatures=false sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-09-23T07:29:13.074999076Z caller=main.go:623 upstream="no upstream URL given"
ts=2019-09-23T07:29:13.075075618Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-09-23T07:29:13.085064576Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2019-09-23T07:29:13.085073271Z caller=main.go:652 metrics-addr=:3031
ts=2019-09-23T07:29:13.085117288Z caller=loop.go:85 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-09-23T07:29:13.086114345Z caller=main.go:644 addr=:3030
ts=2019-09-23T07:29:14.000809128Z caller=checkpoint.go:21 component=checkpoint msg="update available" latest=1.14.2 URL=https://github.com/weaveworks/flux/releases/tag/1.14.2
ts=2019-09-23T07:29:19.301126725Z caller=loop.go:111 component=sync-loop event=refreshed url=git@github.ourgit.com:openshiftha/dev-projects.git branch=master HEAD=d2b2dd1bf4d59e756401a54
ts=2019-09-23T07:32:11.813055187Z caller=sync.go:479 method=Sync cmd=apply args= count=193
ts=2019-09-23T07:32:12.23922921Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=426.023655ms err="running kubectl: " output=
ts=2019-09-23T07:32:12.797153701Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=557.797994ms err=null output="rolebinding.rbac.authorization.k8s.io/rb1 configured"
ts=2019-09-23T07:32:13.20543492Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=408.221192ms err=null output="rolebinding.rbac.authorization.k8s.io/rb2 configured"
ts=2019-09-23T07:32:13.60672595Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=401.239883ms err=null output="rolebinding.rbac.authorization.k8s.io/rb3 configured"

ts=2019-09-23T07:32:56.183179278Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=376.311074ms err=null output="rolebinding.rbac.authorization.k8s.io/rb4 configured"
ts=2019-09-23T07:32:56.596380815Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=413.131117ms err=null output="rolebinding.rbac.authorization.k8s.io/rb5 configured"
ts=2019-09-23T07:32:57.002380099Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=405.891291ms err=null output="rolebinding.rbac.authorization.k8s.io/rb5 configured"
ts=2019-09-23T07:32:57.298410286Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=295.956363ms err="running kubectl: " output=
ts=2019-09-23T07:32:57.671736942Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=373.270976ms err="running kubectl: " output=
ts=2019-09-23T07:32:57.976032989Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=304.236859ms err="running kubectl: " output=
ts=2019-09-23T07:32:58.272461159Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.351868ms err="running kubectl: " output=
ts=2019-09-23T07:32:58.574869054Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=302.346466ms err="running kubectl: " output=
ts=2019-09-23T07:32:58.87559508Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=300.665336ms err="running kubectl: " output=
ts=2019-09-23T07:32:59.109992107Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=234.336774ms err="running kubectl: " output=
ts=2019-09-23T07:32:59.473767787Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=363.705921ms err="running kubectl: " output=
ts=2019-09-23T07:32:59.775798513Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=301.92334ms err="running kubectl: " output=
ts=2019-09-23T07:33:00.072776483Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.913215ms err="running kubectl: " output=
ts=2019-09-23T07:33:00.31277672Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=239.938249ms err="running kubectl: " output=
ts=2019-09-23T07:33:00.673035726Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=360.199219ms err="running kubectl: " output=
ts=2019-09-23T07:33:00.97548632Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=302.385581ms err="running kubectl: " output=
ts=2019-09-23T07:33:01.275608379Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=300.061542ms err="running kubectl: " output=
ts=2019-09-23T07:33:01.50690675Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=231.20882ms err="running kubectl: " output=
ts=2019-09-23T07:33:01.799472797Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=292.501946ms err="running kubectl: " output=
ts=2019-09-23T07:33:02.094427753Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=294.887669ms err="running kubectl: " output=
ts=2019-09-23T07:33:02.481425204Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=386.930835ms err="running kubectl: " output=
ts=2019-09-23T07:33:02.791056666Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=309.533498ms err="running kubectl: " output=
ts=2019-09-23T07:33:03.098131263Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=306.978652ms err="running kubectl: " output=
ts=2019-09-23T07:33:03.396197488Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=297.998919ms err="running kubectl: " output=
ts=2019-09-23T07:33:03.697952215Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=301.696262ms err="running kubectl: " output=
ts=2019-09-23T07:33:03.997349969Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.333676ms err="running kubectl: " output=
ts=2019-09-23T07:33:04.298457395Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=301.044051ms err="running kubectl: " output=
ts=2019-09-23T07:33:04.594973477Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.450534ms err="running kubectl: " output=
ts=2019-09-23T07:33:04.897758926Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=302.729922ms err="running kubectl: " output=
ts=2019-09-23T07:33:05.19601295Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=298.193603ms err="running kubectl: " output=
ts=2019-09-23T07:33:05.571855615Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=375.787632ms err="running kubectl: " output=
ts=2019-09-23T07:33:05.876350575Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=304.419897ms err="running kubectl: " output=
ts=2019-09-23T07:33:06.172811604Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.398131ms err="running kubectl: " output=
ts=2019-09-23T07:33:06.408488247Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=235.61225ms err="running kubectl: " output=
ts=2019-09-23T07:33:06.700239671Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=291.688652ms err="running kubectl: " output=
ts=2019-09-23T07:33:07.07316061Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=372.846439ms err="running kubectl: " output=
ts=2019-09-23T07:33:07.31038495Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=237.157479ms err="running kubectl: " output=
ts=2019-09-23T07:33:07.675818975Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=304.083138ms err="running kubectl: " output=
ts=2019-09-23T07:33:07.974904113Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.02831ms err="running kubectl: " output=
ts=2019-09-23T07:33:08.20665231Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=231.690972ms err="running kubectl: " output=
ts=2019-09-23T07:33:08.583572Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=376.855999ms err="running kubectl: " output=
ts=2019-09-23T07:33:08.888338557Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=304.709914ms err="running kubectl: " output=
ts=2019-09-23T07:33:09.181132223Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=292.727939ms err="running kubectl: " output=
ts=2019-09-23T07:33:09.484406274Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=303.204674ms err="running kubectl: " output=
ts=2019-09-23T07:33:09.775889032Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=291.410941ms err="running kubectl: " output=
ts=2019-09-23T07:33:10.075670124Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.714214ms err="running kubectl: " output=
ts=2019-09-23T07:33:10.373189419Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=297.455841ms err="running kubectl: " output=
ts=2019-09-23T07:33:10.607134884Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=233.88635ms err="running kubectl: " output=
ts=2019-09-23T07:33:10.898824035Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=291.626851ms err="running kubectl: " output=
ts=2019-09-23T07:33:11.195518219Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.622023ms err="running kubectl: " output=
ts=2019-09-23T07:33:11.592353275Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=396.730941ms err="running kubectl: " output=
ts=2019-09-23T07:33:11.975590474Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=383.16332ms err="running kubectl: " output=
ts=2019-09-23T07:33:12.279809096Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=304.160171ms err="running kubectl: " output=
ts=2019-09-23T07:33:12.572732031Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=292.863583ms err="running kubectl: " output=
ts=2019-09-23T07:33:12.8001747Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=227.384378ms err="running kubectl: " output=
ts=2019-09-23T07:33:13.180513915Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=380.284721ms err="running kubectl: " output=
ts=2019-09-23T07:33:13.494678005Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=314.096276ms err="running kubectl: " output=
ts=2019-09-23T07:33:13.884574057Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=389.84284ms err="running kubectl: " output=
ts=2019-09-23T07:33:14.190361549Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=305.737862ms err="running kubectl: " output=
ts=2019-09-23T07:33:14.502634235Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=312.206889ms err="running kubectl: " output=
ts=2019-09-23T07:33:14.875977448Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=373.282763ms err="running kubectl: " output=
ts=2019-09-23T07:33:15.176037435Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.920381ms err="running kubectl: " output=
ts=2019-09-23T07:33:15.472638926Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=296.541502ms err="running kubectl: " output=
ts=2019-09-23T07:33:15.77565328Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=302.953347ms err="running kubectl: " output=
ts=2019-09-23T07:33:16.073033352Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=297.311098ms err="running kubectl: " output=
ts=2019-09-23T07:33:16.372856727Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.763011ms err="running kubectl: " output=
ts=2019-09-23T07:33:16.610239062Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=237.322861ms err="running kubectl: " output=
ts=2019-09-23T07:33:16.899629099Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=289.334369ms err="running kubectl: " output=
ts=2019-09-23T07:33:17.194905197Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=295.215986ms err="running kubectl: " output=
ts=2019-09-23T07:33:17.57949126Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=384.524571ms err="running kubectl: " output=
ts=2019-09-23T07:33:17.888490039Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=308.83348ms err="running kubectl: " output=
ts=2019-09-23T07:33:18.18096543Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=292.412592ms err="running kubectl: " output=
ts=2019-09-23T07:33:18.472822198Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=291.799568ms err="running kubectl: " output=
ts=2019-09-23T07:33:18.77628722Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=303.334963ms err="running kubectl: " output=
ts=2019-09-23T07:33:19.180080857Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=403.630934ms err="running kubectl: " output=
ts=2019-09-23T07:33:19.490509057Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=310.354333ms err="running kubectl: " output=
ts=2019-09-23T07:33:19.789796244Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=299.195793ms err="running kubectl: " output=
ts=2019-09-23T07:33:20.095489669Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=305.632658ms err="running kubectl: " output=
ts=2019-09-23T07:33:20.483888342Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=388.333125ms err="running kubectl: " output=
ts=2019-09-23T07:33:20.789938154Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=305.944319ms err="running kubectl: " output=
ts=2019-09-23T07:33:21.091108548Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=301.109662ms err="running kubectl: " output=
ts=2019-09-23T07:36:00.727019482Z caller=sync.go:152 component=daemon err="opentoolchain:rolebinding/rb5: running kubectl: ; estorestage:rolebinding/rb6: running kubectl: ; estore:rolebinding/rb6: running kubectl: ; estoredev:rolebinding/rb6: running kubectl: ; estoretest:rolebinding/rb6: running kubectl: ; dev-pr1:rolebinding/rb7: running kubectl: ; dmb:rolebinding/rb8: running kubectl: ; devopscom:rolebinding/rb8: running kubectl: ; dev-pr1:rolebinding/rb8:

Additional context Add any other context about the problem here, e.g

We are not using for any images..We are just using for project onboarding ( project , rolebinding , quota , limitrange )

arjunsbabu commented 5 years ago

please see line 12 . I got this error while i checked why flux havent updated the changes done 30 minutes back. ( sync interval is 5minutes and git poll interval is 1m) After 10 minutes the changes are deployed automatically. Ie some temporary issue. Line 12 gives some clue i guess 

ts=2019-09-23T14:06:00.619549213Z caller=sync.go:148 info="cluster resource not in resources to be synced; deleting" dry-run=false resource=bankstage:rolebinding/rb1
ts=2019-09-23T14:06:00.619580711Z caller=sync.go:148 info="cluster resource not in resources to be synced; deleting" dry-run=false resource=<cluster>:project/bankstage
ts=2019-09-23T14:06:00.619610646Z caller=sync.go:153 warning="resource to be synced has not been updated; skipping" dry-run=false resource=eda-demo:resourcequota/eda-demo-quota
ts=2019-09-23T14:06:00.619633347Z caller=sync.go:148 info="cluster resource not in resources to be synced; deleting" dry-run=false resource=bank:limitrange/bank-limits
ts=2019-09-23T14:06:00.619648944Z caller=sync.go:148 info="cluster resource not in resources to be synced; deleting" dry-run=false resource=bank:rolebinding/rb1
ts=2019-09-23T14:06:00.619704562Z caller=sync.go:479 method=Sync cmd=delete args= count=20
ts=2019-09-23T14:06:00.634591957Z caller=sync.go:545 method=Sync cmd="kubectl delete -f -" took=14.84856ms err="running kubectl: " output=
ts=2019-09-23T14:06:06.460748151Z caller=loop.go:206 component=sync-loop tag=flux-sync old=d2b2ddf066fcef90b3b9be21bf4d59e756401a54 new=5f086fefbc583298e81b7f88c047cc562a88a4df
W0923 14:09:02.337791       8 reflector.go:289] pkg/mod/k8s.io/client-go@v11.0.0+incompatible/tools/cache/reflector.go:94: watch of *v1beta1.CustomResourceDefinition ended with: The resourceVersion for the provided watch is too old.
ts=2019-09-23T14:13:21.380818655Z caller=loop.go:85 component=sync-loop err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"
ts=2019-09-23T14:13:21.38091123Z caller=loop.go:104 component=sync-loop url=git@github.mygitrepo:openshif-ha/cloud-dev-projects.git err="git repo not ready: git fetch --tags origin []: running git command: git [fetch --tags origin]: context deadline exceeded"
stefanprodan commented 5 years ago

Looks like a connectivity issue between the cluster and your git server. You can increase the git fetch timeout using the --git-timeout flag (by default is set to 20s).

arjunsbabu commented 5 years ago

@stefanprodan i will try that...But is that the reason for error in kubectl apply and kubectl delete command. kubectl delete and kubectl apply is done after fetching the changes only right ?

hiddeco commented 5 years ago

@arjunsbabu with the little information available it is very hard for us to tell what is going wrong at the moment, except that something is not working for two of our users (on totally different Kubernetes setups, which makes it even more complicated). Please try to modify the flag if the timeout issue persist, and see if that resolves the issue.

If this resolves the issue (both for git and for kubectl), it should be possible for us to replicate the issue to see what goes wrong inside Flux, and fix it so it does not happen for others.

arjunsbabu commented 5 years ago

@stefanprodan @hiddeco increased the timeout to 60s. Still same issue. i can see the flux pod get restarted for some reason . I will share the complete log from start.

arjun@DESKTOP-QO0863U:~/workspace/git/ibmclouddev/projects$ oc logs flux-5cccf65fdd-twbtp -f
ts=2019-09-23T19:41:24.482305523Z caller=main.go:225 version=1.13.3
ts=2019-09-23T19:41:24.48238605Z caller=main.go:317 msg="using in cluster config to connect to the cluster"
ts=2019-09-23T19:41:24.538881242Z caller=main.go:396 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-09-23T19:41:24.538929397Z caller=main.go:397 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAUg7kBvHJp+5lo9J2WsGuu9fjFK4RunN5lmHc4qFMWssiLVjhIJpuAXs1Iczo+k955sT5lG0dZbYLVLNJGHdEllYIKHxVi9+n+k+hQq8AlSGP9WzS27UD997s/3CbZyy9mMKOyN9ylCET0feaDK64wZYRpfQ3K44LO8/5tzTGBzP5aDXsl1BnV7EQmZkZ1sp1DSkyXX2krz5fRrCo3uYjJnaKz2uashEBW1IYisEI8keqeW069xc3U0OHhHE2G/bjMHI9utwB1SpoAtI7fYJTYHsNKIF+VCOdl1eDjhY7cXP2G1zg4jwNyP9ZduZDKat"
ts=2019-09-23T19:41:24.538955096Z caller=main.go:402 host=https://172.30.0.1:443 version=kubernetes-v1.11.0+d4cacc0
ts=2019-09-23T19:41:24.53900911Z caller=main.go:414 kubectl=/usr/local/bin/kubectl
ts=2019-09-23T19:41:24.540113031Z caller=main.go:426 ping=true
ts=2019-09-23T19:41:24.542748737Z caller=main.go:562 url=git@github.mygit.com:openshift-deployha/cloud-dev-projects.git user="Weave Flux" email=support@weave.works signing-key= verify-signatures=false sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-09-23T19:41:24.542810482Z caller=main.go:623 upstream="no upstream URL given"
ts=2019-09-23T19:41:24.543493231Z caller=main.go:652 metrics-addr=:3031
ts=2019-09-23T19:41:24.543767363Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-09-23T19:41:24.543802174Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2019-09-23T19:41:24.543873618Z caller=loop.go:85 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-09-23T19:41:24.544954325Z caller=main.go:644 addr=:3030
ts=2019-09-23T19:41:25.561218747Z caller=checkpoint.go:21 component=checkpoint msg="update available" latest=1.14.2 URL=https://github.com/weaveworks/flux/releases/tag/1.14.2
ts=2019-09-23T19:41:30.67791829Z caller=loop.go:111 component=sync-loop event=refreshed url=git@github.ibm.com:openshift-deployment-ha/ibmcloud-dev-projects.git branch=master HEAD=9802f8fc415a20763ffe6478a8bfa5972b5c784e
ts=2019-09-23T19:44:24.767610833Z caller=sync.go:479 method=Sync cmd=apply args= count=208
ts=2019-09-23T19:44:25.137983552Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=370.128585ms err="running kubectl: " output=
ts=2019-09-23T19:44:25.444583128Z caller=sync.go:545 method=Sync cmd="kubectl apply -f -" took=306.52765ms err="running kubectl: " output=
hiddeco commented 5 years ago

@arjunsbabu do you have the logs from the previous pod just before it was killed?

arjunsbabu commented 5 years ago

no @hiddeco I can see restart happening for flux pod. Why 17 restart has happended not sure. Situation seems to be critical now. The change i deployed yesterday night has been deployed before 2 hrs only. i will try to get the log just before its killed.

arjun@DESKTOP-QO0863U:~/workspace/git/ibmclouddev$ oc get po
NAME                         READY     STATUS    RESTARTS   AGE
flux-7d89bb67b4-jsrkj        1/1       Running   17         4h
jenkins-2-7gpng              1/1       Running   0          26d
memcached-56c9fccf5d-fjc94   1/1       Running   0          31d
arjunsbabu commented 5 years ago

i think the pod is killed due to OOMKilled

arjun@DESKTOP-QO0863U:/mnt/c/Windows/System32$ oc get po -w
NAME                         READY     STATUS    RESTARTS   AGE
flux-7d89bb67b4-x9nwc        1/1       Running   8          1h
jenkins-2-7gpng              1/1       Running   0          26d
memcached-56c9fccf5d-fjc94   1/1       Running   0          32d
flux-7d89bb67b4-x9nwc   0/1       OOMKilled   8         1h
flux-7d89bb67b4-x9nwc   0/1       CrashLoopBackOff   8         1h
flux-7d89bb67b4-x9nwc   1/1       Running   9         1h
arjun@DESKTOP-QO0863U:/mnt/c/Windows/System32$

i have the logs also 


arjun@DESKTOP-QO0863U:~/workspace/git$ oc logs  flux-7d89bb67b4-x9nwc -f
ts=2019-09-24T11:07:35.451333425Z caller=main.go:225 version=1.13.3
ts=2019-09-24T11:07:35.45141518Z caller=main.go:317 msg="using in cluster config to connect to the cluster"
ts=2019-09-24T11:07:35.479078098Z caller=main.go:396 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-09-24T11:07:35.479188815Z caller=main.go:397 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkUg7kBvHJp+5lo9J2WsGuu9fjFK4RunN5lmHc4qFMWssiLVjhIJpuAXsRm2ALVLNJGHdEllYIKHxVi9+n+k+hQq8AlSGP9WzS27UD997s/3CbZyy9mMKOyN9ylCET0feaDK64wZYRpfQ3K44LO8/5tzTGBzP5aDXsl1BnV7EQmZkZ1sp1DSkyXX2krz5fRrCo3uYjJnaKz2uashEBW1IYisEI8keqeW069xc3U0OHhHE2G/bjMHI9utwB1SpoAtI7fYJTYHsNKIF+VCOdl1eDjhY7cXP2G1zg4jwNyP9ZduZDKat"
ts=2019-09-24T11:07:35.479214452Z caller=main.go:402 host=https://172.30.0.1:443 version=kubernetes-v1.11.0+d4cacc0
ts=2019-09-24T11:07:35.479275523Z caller=main.go:414 kubectl=/usr/local/bin/kubectl
ts=2019-09-24T11:07:35.480315047Z caller=main.go:426 ping=true
ts=2019-09-24T11:07:35.482722878Z caller=main.go:562 url=git@github.mygit.com:openshift-deployment-ha/ibmcloud-dev-projects.git user="Weave Flux" email=support@weave.works signing-key= verify-signatures=false sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-09-24T11:07:35.482775875Z caller=main.go:623 upstream="no upstream URL given"
ts=2019-09-24T11:07:35.483379079Z caller=loop.go:85 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-09-24T11:07:35.483423567Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-09-24T11:07:35.483438424Z caller=images.go:27 component=sync-loop msg="no automated workloads"
ts=2019-09-24T11:07:35.48350288Z caller=main.go:652 metrics-addr=:3031
ts=2019-09-24T11:07:35.484502608Z caller=main.go:644 addr=:3030
ts=2019-09-24T11:07:36.298528162Z caller=checkpoint.go:21 component=checkpoint msg="update available" latest=1.14.2 URL=https://github.com/weaveworks/flux/releases/tag/1.14.2
ts=2019-09-24T11:07:41.844801669Z caller=loop.go:111 component=sync-loop event=refreshed url=git@github.mygit.com:openshift-deployment-ha/dev-projects.git branch=master HEAD=728f1b2453dbfc480457da7671ba8933bec67602
arjun@DESKTOP-QO0863U:~/workspace/git/clouddev$
arjunsbabu commented 5 years ago

Resources given for flux..Let me increase and check

    Limits:
      cpu:     500m
      memory:  500Mi
    Requests:
      cpu:        50m
      memory:     64Mi
arjunsbabu commented 5 years ago

After increasing the request and limit of flux deployment i am not facing the issue. I think it solved the issue @hiddeco @stefanprodan

hiddeco commented 5 years ago

@arjunsbabu awesome! Given I have been a bit late with my reply and you haven't posted since, I will assume this was indeed the problem and close the issue.

If you are experiencing new issues, or the problem arises again, do not hesitate to either re-open it or open a new one. :tulip: