search

microsoft / bedrock

Automation for Production Kubernetes Clusters with a GitOps Workflow
MIT License
129 stars 12 forks source link

Documentation should reflect gitops repo should not be empty #154

Closed bnookala closed 5 years ago

bnookala commented 5 years ago

Flux will come up, but will error, and go into a state in which it will not pull from the gitops repository unless it is killed (per cluster) and brought up again.

Log lines from flux when trying to pull from an empty repository:

$ KUBECONFIG=eastus2-mikecluster_kube_config k logs flux-898949495-gh9g4 -n flux -f
ts=2019-02-28T02:41:36.799722492Z caller=main.go:156 version=1.10.1
ts=2019-02-28T02:41:36.863215152Z caller=main.go:247 component=cluster identity=/etc/fluxd/ssh/identity
ts=2019-02-28T02:41:36.863317951Z caller=main.go:248 component=cluster identity.pub="ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCr2hbhqyF0moyXdAMk+7swuSY312JIS5q2C8PCH/uMwJqtU4vMeex96VwJ+nJv1Q1eX2qmThBDD1xzA4Kx2OkYSK4blIw7U3CqPypcsf/z/7TNuBSrM6C0VafVNJWO6QcVcaIt1UvfSpPczTabNktsQAMWZ3h2bZi5z1BTiiQJhM9yrciKVTmyeOH+F4lA9Hsil1P7LY/n8pT6VJJSx/LvHW8sNW8MrJPL+p7AmCig60zuzxwFesEE9kiqZpw2znIgJ7u6S7MSxLIhKxg75c77z0L7/0Mq+FlGV+Xt6UuDpVuJ8CcR1DAP7lytn6so0eojeSjW0zmumtuRtplgPleJ"
ts=2019-02-28T02:41:36.863344451Z caller=main.go:249 component=cluster host=https://mikedns-07dd66b5.hcp.eastus2.azmk8s.io:443 version=kubernetes-v1.12.5
ts=2019-02-28T02:41:36.86341215Z caller=main.go:261 component=cluster kubectl=/usr/local/bin/kubectl
ts=2019-02-28T02:41:36.865050236Z caller=main.go:269 component=cluster ping=true
ts=2019-02-28T02:41:36.867767913Z caller=aws.go:69 component=aws warn="no AWS region configured, or detected as cluster region" err="EC2MetadataError: failed to make EC2Metadata request\ncaused by: <!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Strict//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd\">\r\n<html xmlns=\"http://www.w3.org/1999/xhtml\">\r\n<head>\r\n<meta http-equiv=\"Content-Type\" content=\"text/html; charset=iso-8859-1\"/>\r\n<title>404 - File or directory not found.</title>\r\n<style type=\"text/css\">\r\n<!--\r\nbody{margin:0;font-size:.7em;font-family:Verdana, Arial, Helvetica, sans-serif;background:#EEEEEE;}\r\nfieldset{padding:0 15px 10px 15px;} \r\nh1{font-size:2.4em;margin:0;color:#FFF;}\r\nh2{font-size:1.7em;margin:0;color:#CC0000;} \r\nh3{font-size:1.2em;margin:10px 0 0 0;color:#000000;} \r\n#header{width:96%;margin:0 0 0 0;padding:6px 2% 6px 2%;font-family:\"trebuchet MS\", Verdana, sans-serif;color:#FFF;\r\nbackground-color:#555555;}\r\n#content{margin:0 0 0 2%;position:relative;}\r\n.content-container{background:#FFF;width:96%;margin-top:8px;padding:10px;position:relative;}\r\n-->\r\n</style>\r\n</head>\r\n<body>\r\n<div id=\"header\"><h1>Server Error</h1></div>\r\n<div id=\"content\">\r\n <div class=\"content-container\"><fieldset>\r\n  <h2>404 - File or directory not found.</h2>\r\n  <h3>The resource you are looking for might have been removed, had its name changed, or is temporarily unavailable.</h3>\r\n </fieldset></div>\r\n</div>\r\n</body>\r\n</html>\r\n"
ts=2019-02-28T02:41:36.867822913Z caller=main.go:288 warning="AWS authorization not used; pre-flight check failed"
ts=2019-02-28T02:41:36.869197601Z caller=main.go:417 url=git@github.com:bnookala/gitops-manifests.git user="Weave Flux" email=support@weave.works sync-tag=flux-sync notes-ref=flux set-author=false
ts=2019-02-28T02:41:36.869292Z caller=main.go:472 upstream="no upstream URL given"
ts=2019-02-28T02:41:36.869827796Z caller=main.go:493 addr=:3030
ts=2019-02-28T02:41:36.872972769Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:41:36.873065568Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:41:36.873141368Z caller=loop.go:92 component=sync-loop err="git repo not ready: git repo has not been cloned yet"
ts=2019-02-28T02:41:37.316354607Z caller=checkpoint.go:24 component=checkpoint msg="up to date" latest=1.9.0
ts=2019-02-28T02:41:38.195402366Z caller=memcached.go:109 component=memcached err="Fetching tag from memcache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:38.195495065Z caller=warming.go:170 component=warmer canonical_name=gcr.io/google-containers/ip-masq-agent-amd64 auth={map[]} err="fetching previous result from cache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:39.270183408Z caller=memcached.go:109 component=memcached err="Fetching tag from memcache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:39.270312506Z caller=warming.go:170 component=warmer canonical_name=index.docker.io/deis/kube-svc-redirect auth={map[]} err="fetching previous result from cache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:40.270570612Z caller=memcached.go:109 component=memcached err="Fetching tag from memcache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:40.270660411Z caller=warming.go:170 component=warmer canonical_name=index.docker.io/library/nginx auth={map[]} err="fetching previous result from cache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:41.270996048Z caller=memcached.go:109 component=memcached err="Fetching tag from memcache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:41.271061748Z caller=warming.go:170 component=warmer canonical_name=index.docker.io/deis/hcp-tunnel-front auth={map[]} err="fetching previous result from cache: memcache: connect timeout to 10.0.181.178:11211"
ts=2019-02-28T02:41:42.559919123Z caller=warming.go:268 component=warmer info="refreshing image" image=quay.io/weaveworks/flux tag_count=577 to_update=577 of_which_refresh=0 of_which_missing=577
ts=2019-02-28T02:41:45.114308017Z caller=warming.go:364 component=warmer updated=quay.io/weaveworks/flux successful=577 attempted=577
ts=2019-02-28T02:41:45.114663214Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:41:45.114852912Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:41:45.35133127Z caller=warming.go:268 component=warmer info="refreshing image" image=memcached tag_count=66 to_update=66 of_which_refresh=0 of_which_missing=66
ts=2019-02-28T02:41:47.552413951Z caller=warming.go:364 component=warmer updated=memcached successful=66 attempted=66
ts=2019-02-28T02:41:47.553996338Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:41:47.554035738Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:41:47.760332655Z caller=warming.go:268 component=warmer info="refreshing image" image=containernetworking/networkmonitor tag_count=5 to_update=5 of_which_refresh=0 of_which_missing=5
ts=2019-02-28T02:41:48.06879424Z caller=warming.go:364 component=warmer updated=containernetworking/networkmonitor successful=5 attempted=5
ts=2019-02-28T02:41:48.068962539Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:41:48.069019939Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:42:37.363293694Z caller=warming.go:268 component=warmer info="refreshing image" image=gcr.io/google-containers/ip-masq-agent-amd64 tag_count=8 to_update=8 of_which_refresh=0 of_which_missing=8
ts=2019-02-28T02:42:37.779631933Z caller=warming.go:364 component=warmer updated=gcr.io/google-containers/ip-masq-agent-amd64 successful=8 attempted=8
ts=2019-02-28T02:42:37.780028827Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:42:37.780057826Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:42:37.894454569Z caller=warming.go:268 component=warmer info="refreshing image" image=docker.io/deis/kube-svc-redirect tag_count=4 to_update=4 of_which_refresh=0 of_which_missing=4
ts=2019-02-28T02:42:38.257208009Z caller=warming.go:364 component=warmer updated=docker.io/deis/kube-svc-redirect successful=4 attempted=4
ts=2019-02-28T02:42:38.257442005Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:42:38.257479105Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:42:38.497176042Z caller=warming.go:268 component=warmer info="refreshing image" image=nginx tag_count=213 to_update=213 of_which_refresh=0 of_which_missing=213
ts=2019-02-28T02:42:44.664903299Z caller=warming.go:364 component=warmer updated=nginx successful=213 attempted=213
ts=2019-02-28T02:42:44.669110435Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:42:44.669227633Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:42:45.101317984Z caller=warming.go:268 component=warmer info="refreshing image" image=docker.io/deis/hcp-tunnel-front tag_count=40 to_update=40 of_which_refresh=0 of_which_missing=40
ts=2019-02-28T02:42:46.53590039Z caller=warming.go:364 component=warmer updated=docker.io/deis/hcp-tunnel-front successful=40 attempted=40
ts=2019-02-28T02:42:46.537110671Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:42:46.537306168Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:46:36.873475459Z caller=loop.go:92 component=sync-loop err="git repo not ready: tag for write check: fatal: Failed to resolve 'HEAD' as a valid ref."
ts=2019-02-28T02:47:46.53762285Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:47:46.537751048Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:51:36.873829656Z caller=loop.go:92 component=sync-loop err="git repo not ready: tag for write check: fatal: Failed to resolve 'HEAD' as a valid ref."
ts=2019-02-28T02:52:46.537891447Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:52:46.538067545Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T02:56:36.874113041Z caller=loop.go:92 component=sync-loop err="git repo not ready: tag for write check: fatal: Failed to resolve 'HEAD' as a valid ref."
ts=2019-02-28T02:57:38.917028279Z caller=warming.go:268 component=warmer info="refreshing image" image=quay.io/weaveworks/flux tag_count=578 to_update=1 of_which_refresh=0 of_which_missing=1
ts=2019-02-28T02:57:39.039340693Z caller=warming.go:364 component=warmer updated=quay.io/weaveworks/flux successful=1 attempted=1
ts=2019-02-28T02:57:39.04423613Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T02:57:39.044328429Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T03:01:36.874358395Z caller=loop.go:92 component=sync-loop err="git repo not ready: tag for write check: fatal: Failed to resolve 'HEAD' as a valid ref."
ts=2019-02-28T03:02:39.044476543Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T03:02:39.044532042Z caller=images.go:27 component=sync-loop msg="no automated services"
ts=2019-02-28T03:06:36.874674576Z caller=loop.go:92 component=sync-loop err="git repo not ready: tag for write check: fatal: Failed to resolve 'HEAD' as a valid ref."
ts=2019-02-28T03:07:39.044722359Z caller=images.go:17 component=sync-loop msg="polling images"
ts=2019-02-28T03:07:39.044771558Z caller=images.go:27 component=sync-loop msg="no automated services"
timfpark commented 5 years ago

We might also consider raising an issue on this for Flux - it should not go into a permanent failure situation because of this.

timfpark commented 5 years ago

It looks like the repo was created in this situation but had zero commits - is that right? In which case, it looks like our precise guidance should be that the repo has at least one commit - but does not need to have resource manifests yet.

bnookala commented 5 years ago

Yes, exactly - no commits in the repository.

On Mon, Mar 4, 2019 at 9:24 AM Tim Park notifications@github.com wrote:

It looks like the repo was created in this situation but had zero commits

  • is that right? In which case, it looks like our precise guidance should be that the repo has at least one commit.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/Microsoft/bedrock/issues/154#issuecomment-469109998, or mute the thread https://github.com/notifications/unsubscribe-auth/AAFXvOLkIlgKUnMJoc9nxG1Xvr1HVYitks5vTJkCgaJpZM4bV7Zu .

timfpark commented 5 years ago

Addressed in https://github.com/Microsoft/bedrock/commit/c6e70d136a20d33cc3971a47a012aec6f9aa9340