search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.37k stars 685 forks source link

CrashLoopBackoff - Constant restarts #2848

Closed DarkBlaez closed 4 years ago

DarkBlaez commented 4 years ago

I keep getting "Liveness probe failed: HTTP probe failed with statuscode: 503 no matter which way I install as it pertains to the Helm chart or any combo with edgectl install

If I install differently:

kubectl apply --namespace ambassador \ --filename https://getambassador.io/yaml/ambassador/ambassador-rbac.yaml \ --filename https://getambassador.io/yaml/ambassador/ambassador-service.yaml

And proceed with the directions on Knative for Ambassador I see no issues , only exception is I cannot get to the endpoints or admin ui

If I install using the helm per:

https://www.getambassador.io/docs/latest/tutorials/getting-started/

It results in a CrashLoopBackoff with constant restarts below....

2020-07-05T16:49:19.671043674Z 2020-07-05 16:49:19 kubewatch [11 TMainThread] 1.5.5 DEBUG: looking up ID for namespace default 2020-07-05T16:49:22.579732776Z 2020-07-05 16:49:22 kubewatch [11 TMainThread] 1.5.5 DEBUG: cluster ID URL is d6e_id://cb4d673d-88a0-4d2c-ac3f-dd7d2f202257/default 2020-07-05T16:49:25.93385845Z 2020-07-05 16:49:25 kubewatch [11 TMainThread] 1.5.5 DEBUG: IngressClass check got 403 2020-07-05T16:49:26.031122015Z 2020-07-05 16:49:26 kubewatch [11 TMainThread] 1.5.5 DEBUG: Ambassador does not have permission to read IngressClass resources. To enable IngressClass support, configure RBAC to allow Ambassador to read IngressClass resources, then restart the Ambassador pod. 2020-07-05T16:49:30.820492507Z 2020-07-05 16:49:30 kubewatch [11 TMainThread] 1.5.5 DEBUG: CRD type definition not found for ambassadorinstallations.getambassador.io 2020-07-05T16:49:30.867398724Z 2020-07-05 16:49:30 kubewatch [11 TMainThread] 1.5.5 DEBUG: ambassadorinstallations.getambassador.io CRD not available 2020-07-05T16:49:30.867702861Z 2020-07-05 16:49:30 kubewatch [11 TMainThread] 1.5.5 DEBUG: cluster ID is 9e2fd712-5ea6-5938-9496-6460246a5d68 (from namespace default) 2020-07-05T16:49:31.120457185Z 2020-07-05 16:49:31 AMBASSADOR INFO starting with environment: 2020-07-05T16:49:31.122080335Z 2020-07-05 16:49:31 AMBASSADOR INFO ==== 2020-07-05T16:49:31.124629797Z AMBASSADOR_ADMIN_PORT=tcp://10.43.14.99:8877 2020-07-05T16:49:31.124668577Z AMBASSADOR_ADMIN_PORT_8877_TCP=tcp://10.43.14.99:8877 2020-07-05T16:49:31.124675465Z AMBASSADOR_ADMIN_PORT_8877_TCP_ADDR=10.43.14.99 2020-07-05T16:49:31.124680105Z AMBASSADOR_ADMIN_PORT_8877_TCP_PORT=8877 2020-07-05T16:49:31.124685646Z AMBASSADOR_ADMIN_PORT_8877_TCP_PROTO=tcp 2020-07-05T16:49:31.124690164Z AMBASSADOR_ADMIN_SERVICE_HOST=10.43.14.99 2020-07-05T16:49:31.124694293Z AMBASSADOR_ADMIN_SERVICE_PORT=8877 2020-07-05T16:49:31.124699626Z AMBASSADOR_ADMIN_SERVICE_PORT_AMBASSADOR_ADMIN=8877 2020-07-05T16:49:31.124703557Z AMBASSADOR_CLUSTER_ID=9e2fd712-5ea6-5938-9496-6460246a5d68 2020-07-05T16:49:31.124708674Z AMBASSADOR_CONFIG_BASE_DIR=/ambassador 2020-07-05T16:49:31.124712551Z AMBASSADOR_NAMESPACE=ambassador 2020-07-05T16:49:31.124716802Z AMBASSADOR_PORT=tcp://10.43.251.121:80 2020-07-05T16:49:31.124720756Z AMBASSADOR_PORT_443_TCP=tcp://10.43.251.121:443 2020-07-05T16:49:31.124724517Z AMBASSADOR_PORT_443_TCP_ADDR=10.43.251.121 2020-07-05T16:49:31.124728323Z AMBASSADOR_PORT_443_TCP_PORT=443 2020-07-05T16:49:31.124733506Z AMBASSADOR_PORT_443_TCP_PROTO=tcp 2020-07-05T16:49:31.124737478Z AMBASSADOR_PORT_80_TCP=tcp://10.43.251.121:80 2020-07-05T16:49:31.124741156Z AMBASSADOR_PORT_80_TCP_ADDR=10.43.251.121 2020-07-05T16:49:31.124744796Z AMBASSADOR_PORT_80_TCP_PORT=80 2020-07-05T16:49:31.124748889Z AMBASSADOR_PORT_80_TCP_PROTO=tcp 2020-07-05T16:49:31.124752983Z AMBASSADOR_REDIS_PORT=tcp://10.43.66.253:6379 2020-07-05T16:49:31.124757084Z AMBASSADOR_REDIS_PORT_6379_TCP=tcp://10.43.66.253:6379 2020-07-05T16:49:31.124760787Z AMBASSADOR_REDIS_PORT_6379_TCP_ADDR=10.43.66.253 2020-07-05T16:49:31.12476459Z AMBASSADOR_REDIS_PORT_6379_TCP_PORT=6379 2020-07-05T16:49:31.124768329Z AMBASSADOR_REDIS_PORT_6379_TCP_PROTO=tcp 2020-07-05T16:49:31.124772142Z AMBASSADOR_REDIS_SERVICE_HOST=10.43.66.253 2020-07-05T16:49:31.124775643Z AMBASSADOR_REDIS_SERVICE_PORT=6379 2020-07-05T16:49:31.124779342Z AMBASSADOR_SERVICE_HOST=10.43.251.121 2020-07-05T16:49:31.12478293Z AMBASSADOR_SERVICE_PORT=80 2020-07-05T16:49:31.124786686Z AMBASSADOR_SERVICE_PORT_HTTP=80 2020-07-05T16:49:31.124790285Z AMBASSADOR_SERVICE_PORT_HTTPS=443 2020-07-05T16:49:31.126400772Z 2020-07-05 16:49:31 AMBASSADOR INFO ==== 2020-07-05T16:49:31.132667791Z 2020-07-05 16:49:31 AMBASSADOR INFO launching worker process 'ambex': 'ambex' '--ads-listen-address=127.0.0.1:8003' '/ambassador/envoy' 2020-07-05T16:49:31.135500268Z 2020-07-05 16:49:31 AMBASSADOR INFO ambex is PID 56 2020-07-05T16:49:31.1377097Z 2020-07-05 16:49:31 AMBASSADOR INFO launching worker process 'diagd': 'diagd' '/ambassador/snapshots' '/ambassador/bootstrap-ads.json' '/ambassador/envoy/envoy.json' '--notices' '/ambassador/notices.json' '--kick' 'kill -HUP 1' 2020-07-05T16:49:31.141340156Z 2020-07-05 16:49:31 AMBASSADOR INFO diagd is PID 64 2020-07-05T16:49:32.52539851Z 2020-07-05 16:49:32 diagd 1.5.5 [P64TMainThread] INFO: WILL update Mapping status 2020-07-05T16:49:32.525848086Z 2020-07-05 16:49:32 diagd 1.5.5 [P64TMainThread] INFO: thread count 9, listening on 0.0.0.0:8877 2020-07-05T16:49:33.001730991Z 2020-07-05 16:49:32 diagd 1.5.5 [P64TMainThread] INFO: Ambassador 1.5.5 booted 2020-07-05T16:49:33.019579963Z [2020-07-05 16:49:33 +0000] [64] [INFO] Starting gunicorn 19.9.0 2020-07-05T16:49:33.020833661Z [2020-07-05 16:49:33 +0000] [64] [INFO] Listening at: http://0.0.0.0:8877 (64) 2020-07-05T16:49:33.020873051Z [2020-07-05 16:49:33 +0000] [64] [INFO] Using worker: threads 2020-07-05T16:49:33.029013342Z [2020-07-05 16:49:33 +0000] [77] [INFO] Booting worker with pid: 77 2020-07-05T16:49:33.035065982Z 2020-07-05 16:49:33 diagd 1.5.5 [P77TAEW] INFO: starting Scout checker and timer logger 2020-07-05T16:49:33.038145712Z 2020-07-05 16:49:33 diagd 1.5.5 [P77TAEW] INFO: starting event watcher 2020-07-05T16:49:34.213121614Z 2020-07-05 16:49:34 AMBASSADOR INFO diagd running 2020-07-05T16:49:34.224762109Z 2020-07-05 16:49:34 AMBASSADOR INFO launching worker process 'watt': 'watt' '--listen-address=127.0.0.1:8002' '--notify' 'python /ambassador/post_update.py --watt ' '--watch' 'python /ambassador/watch_hook.py' '-s' 'service' '-s' 'ingresses' '-s' 'AuthService' '-s' 'Mapping' '-s' 'Module' '-s' 'RateLimitService' '-s' 'TCPMapping' '-s' 'TLSContext' '-s' 'TracingService' '-s' 'ConsulResolver' '-s' 'KubernetesEndpointResolver' '-s' 'KubernetesServiceResolver' '-s' 'Host' '-s' 'LogService' 2020-07-05T16:49:34.232763955Z 2020-07-05 16:49:34 AMBASSADOR INFO watt is PID 87 2020-07-05T16:49:34.238747152Z 2020-07-05 16:49:34 AMBASSADOR INFO launching worker process 'amb-sidecar': '/ambassador/sidecars/amb-sidecar' 2020-07-05T16:49:34.241482433Z 2020-07-05 16:49:34 AMBASSADOR INFO amb-sidecar is PID 98 2020-07-05T16:49:34.373744379Z time="2020-07-05 16:49:34" level=info msg="Ambassador Edge Stack configuration loaded" func=github.com/datawire/apro/cmd/amb-sidecar/runner.runE file="github.com/datawire/apro@/cmd/amb-sidecar/runner/main.go:187" 2020-07-05T16:49:38.757994896Z time="2020-07-05 16:49:38" level=error msg="Failed to create watch on /home/ambassador/.config/ambassador/: Changes might require a restart: no such file or directory" func=github.com/datawire/apro/cmd/amb-sidecar/runner.triggerOnChange file="github.com/datawire/apro@/cmd/amb-sidecar/runner/files.go:54" 2020-07-05T16:49:39.638677974Z time="2020-07-05 16:49:39" level=warning msg="statsd is not in use" func=github.com/lyft/gostats.NewDefaultStore file="github.com/lyft/gostats@v0.2.6/stats.go:193" 2020-07-05T16:49:39.653146346Z 2020-07-05 16:49:39 diagd 1.5.5 [P77TThreadPoolExecutor-0_2] ERROR: 'NoneType' object has no attribute 'overview' 2020-07-05T16:49:39.653188912Z Traceback (most recent call last): 2020-07-05T16:49:39.653196091Z File "/usr/lib/python3.7/site-packages/ambassador-0.0.0.dev0-py3.7.egg/ambassador_diag/diagd.py", line 292, in wrapper 2020-07-05T16:49:39.653201866Z result = f(*args, reqid=reqid, *kwds) 2020-07-05T16:49:39.653209414Z File "/usr/lib/python3.7/site-packages/ambassador-0.0.0.dev0-py3.7.egg/ambassador_diag/diagd.py", line 578, in show_overview 2020-07-05T16:49:39.653215358Z ov = diag.overview(request, app.estats) 2020-07-05T16:49:39.653220721Z AttributeError: 'NoneType' object has no attribute 'overview' 2020-07-05T16:49:39.653247287Z 2020-07-05 16:49:39 diagd 1.5.5 [P77TThreadPoolExecutor-0_2] ERROR: 3E9C29BC-94A7-45C9-8B84-10230A3F615E: 127.0.0.1 "GET /ambassador/v0/diag/" 2ms 500 server error 2020-07-05T16:49:39.653253179Z time="2020-07-05 16:49:39" level=error msg="Bad HTTP response" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro@/cmd/amb-sidecar/devportal/server/fetcher.go:165" status_code=500 subsystem=fetcher url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true" 2020-07-05T16:49:39.653259294Z time="2020-07-05 16:49:39" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet file="github.com/datawire/apro@/cmd/amb-sidecar/devportal/server/fetcher.go:172" subsystem=fetcher url="http://127.0.0.1:8877/ambassador/v0/diag/?json=true" 2020-07-05T16:49:39.995479936Z time="2020-07-05T16:49:39Z" level=info msg="starting watt..." 2020-07-05T16:49:39.995561235Z 2020/07/05 16:49:39 api: snapshot server listening on: tcp:127.0.0.1:8002 2020-07-05T16:49:41.031586912Z time="2020-07-05 16:49:41" level=warning msg="runtime: error processing /tmp/amb/config/config: lstat /tmp/amb/config/config: no such file or directory" func="github.com/lyft/goruntime/loader.(Loader).walkDirectoryCallback" file="github.com/lyft/goruntime@v0.1.8/loader/loader.go:87" 2020-07-05T16:49:51.160384515Z Calling Metriton 2020-07-05T16:49:54.197463235Z time="2020-07-05 16:49:54" level=error msg="0 filters configured" func="github.com/datawire/apro/cmd/amb-sidecar/filters/controller.(*Controller).Watch.func1" file="github.com/datawire/apro@/cmd/amb-sidecar/filters/controller/controller.go:151" MAIN=auth_controller 2020-07-05T16:50:15.015053789Z 2020/07/05 16:50:15 aggregator: bootstrapped! 2020-07-05T16:50:21.136778514Z 2020-07-05 16:50:21 AMBASSADOR INFO launching worker process 'envoy': 'envoy' '-c' '/ambassador/bootstrap-ads.json' '--base-id' '0' '--drain-time-s' '600' '-l' 'error' 2020-07-05T16:50:21.228965195Z 2020-07-05 16:50:21 diagd 1.5.5 [P77TAEW] INFO: configuration updated from snapshot 1 (S87 L1 G9 C4) 2020-07-05T16:50:21.229009001Z 2020-07-05 16:50:21 AMBASSADOR INFO envoy is PID 149 2020-07-05T16:50:21.336433273Z time="2020-07-05 16:50:21" level=warning msg="license_secret_watch: empty decoded license data" func=github.com/datawire/apro/cmd/amb-sidecar/runner.runE.func4 file="github.com/datawire/apro@/cmd/amb-sidecar/runner/main.go:344" MAIN=license_secret_watch 2020-07-05T16:50:21.435092066Z 2020-07-05 16:50:21 AMBASSADOR INFO KICK: started Envoy as PID 149 2020-07-05T16:50:29.752571047Z 2020-07-05 16:50:29 diagd 1.5.5 [P77TAEW] INFO: configuration updated from snapshot 2 (S87 L1 G9 C4) 2020-07-05T16:50:29.844836775Z time="2020-07-05 16:50:29" level=warning msg="license_secret_watch: empty decoded license data" func=github.com/datawire/apro/cmd/amb-sidecar/runner.runE.func4 file="github.com/datawire/apro@/cmd/amb-sidecar/runner/main.go:344" MAIN=license_secret_watch

DarkBlaez commented 4 years ago

Closing as this seems to happen with replicas and I need to troubleshoot further