search

datawire / ambassador-chart

Helm Chart for Installing Ambassador
42 stars 60 forks source link

Ambassador 1.9.0 fails to start after upgrading #142

Closed vsimon closed 3 years ago

vsimon commented 3 years ago

Do the following:

helm install ambassador datawire/ambassador --version 6.5.11 -n ambassador helm upgrade ambassador datawire/ambassador --version 6.5.12 -n ambassador

Get this in the ambassador-xxxxxxxx pod:

time="2020-11-13 04:05:25" level=error msg="Failed to create watch on /home/ambassador/.config/ambassador/: Changes might require a restart: no such file or directory" func=github.com/datawire/apro/cmd/amb-sidecar/runner.triggerOnChange file="github.com/datawire/apro/cm
 2020-11-13 04:05:25 diagd 1.9.0 [P73TThreadPoolExecutor-0_0] ERROR: CABB27B6-DAA2-4054-938B-098AF84E1AC8: 127.0.0.1 "GET /ambassador/v0/diag/" 0ms 500 failure
 time="2020-11-13 04:05:25" level=error msg="Bad HTTP response" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1.1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:75" component=devportal mhost= status_code=500 url="http
 time="2020-11-13 04:05:25" level=error msg="HTTP error 500 from http://127.0.0.1:8877/ambassador/v0/diag/?json=true" func=github.com/datawire/apro/cmd/amb-sidecar/devportal/server.HTTPGet.func1 file="github.com/datawire/apro/cmd/amb-sidecar/devportal/server/http.go:82"
 2020/11/13 04:05:26 aggregator: watch hook stderr: 2020-11-13 04:05:26 watch-hook INFO: -global-: <RichStatus BAD error='Ambassador could not find the DevPortal CRD definition. Please visit https://www.getambassador.io/reference/core/crds/ for more information. You can
 2020/11/13 04:05:26 aggregator: watch hook stderr: Traceback (most recent call last):
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/ambassador/watch_hook.py", line 299, in <module>
 2020/11/13 04:05:26 aggregator: watch hook stderr:     wh = WatchHook(logger, yaml_stream)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/ambassador/watch_hook.py", line 104, in __init__
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self.load_yaml(yaml_stream)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/ambassador/watch_hook.py", line 126, in load_yaml
 2020/11/13 04:05:26 aggregator: watch hook stderr:     fetcher.parse_watt(yaml_stream.read())
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/fetcher.py", line 242, in parse_watt
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self.handle_k8s(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/fetcher.py", line 303, in handle_k8s
 2020/11/13 04:05:26 aggregator: watch hook stderr:     if self.k8s_processor.try_process(obj):
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 40, in try_process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self._process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 119, in _process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self.delegate.try_process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 40, in try_process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self._process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 91, in _process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     proc.try_process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 40, in try_process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self._process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/service.py", line 209, in _process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self.delegate.try_process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 40, in try_process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self._process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 91, in _process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     proc.try_process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/k8sprocessor.py", line 40, in try_process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self._process(obj)
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/service.py", line 93, in _process
 2020/11/13 04:05:26 aggregator: watch hook stderr:     self.manager.emit_annotated(NormalizedResource.from_kubernetes_object_annotation(obj))
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/fetch/resource.py", line 96, in from_kubernetes_object_annotation
 2020/11/13 04:05:26 aggregator: watch hook stderr:     return [clean_normalize(r) for r in parse_yaml(config) if r]
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/buildroot/ambassador/python/ambassador/utils.py", line 72, in parse_yaml
 2020/11/13 04:05:26 aggregator: watch hook stderr:     return list(yaml.load_all(serialization, Loader=yaml_loader))
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/usr/lib/python3.8/site-packages/yaml/__init__.py", line 130, in load_all
 2020/11/13 04:05:26 aggregator: watch hook stderr:     yield loader.get_data()
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 45, in get_data
 2020/11/13 04:05:26 aggregator: watch hook stderr:     return self.construct_document(self.get_node())
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 699, in _yaml.CParser.get_node
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 725, in _yaml.CParser._compose_document
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 732, in _yaml.CParser._compose_node
 2020/11/13 04:05:26 aggregator: watch hook stderr:   File "ext/_yaml.pyx", line 905, in _yaml.CParser._parse_next_event
 2020/11/13 04:05:26 aggregator: watch hook stderr: yaml.scanner.ScannerError: while scanning an alias
 2020/11/13 04:05:26 aggregator: watch hook stderr:   in "<unicode string>", line 7, column 7
 2020/11/13 04:05:26 aggregator: watch hook stderr: did not find expected alphabetic or numeric character
 2020/11/13 04:05:26 aggregator: watch hook stderr:   in "<unicode string>", line 7, column 8
 2020/11/13 04:05:26 aggregator: watch hook stderr:
 2020/11/13 04:05:26 aggregator: watch hook failed: exit status 1
 2020/11/13 04:05:26 aggregator: bootstrapped!
 2020-11-13 04:05:26 diagd 1.9.0 [P73TAEW] INFO: -global-: <RichStatus BAD error='Ambassador could not find the DevPortal CRD definition. Please visit https://www.getambassador.io/reference/core/crds/ for more information. You can continue using Ambassador via Kubernetes
 2020-11-13 04:05:26 diagd 1.9.0 [P73TAEW] ERROR: could not reconfigure: while scanning an alias
  in "<unicode string>", line 7, column 7
 did not find expected alphabetic or numeric character
   in "<unicode string>", line 7, column 8
 2020-11-13 04:05:26 diagd 1.9.0 [P73TAEW] ERROR: while scanning an alias
   in "<unicode string>", line 7, column 7
 did not find expected alphabetic or numeric character
   in "<unicode string>", line 7, column 8
 Traceback (most recent call last):
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador_diag/diagd.py", line 1298, in run
     self.load_config_watt(rqueue, url)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador_diag/diagd.py", line 1447, in load_config_watt
     fetcher.parse_watt(serialization)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/fetcher.py", line 242, in parse_watt
     self.handle_k8s(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/fetcher.py", line 303, in handle_k8s
     if self.k8s_processor.try_process(obj):
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 40, in try_process
     self._process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 119, in _process
     self.delegate.try_process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 40, in try_process
     self._process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 91, in _process
     proc.try_process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 40, in try_process
     self._process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/service.py", line 209, in _process
     self.delegate.try_process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 40, in try_process
     self._process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 91, in _process
     proc.try_process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/k8sprocessor.py", line 40, in try_process
     self._process(obj)
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/service.py", line 93, in _process
     self.manager.emit_annotated(NormalizedResource.from_kubernetes_object_annotation(obj))
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/fetch/resource.py", line 96, in from_kubernetes_object_annotation
     return [clean_normalize(r) for r in parse_yaml(config) if r]
   File "/usr/lib/python3.8/site-packages/ambassador-0.0.0.dev0-py3.8.egg/ambassador/utils.py", line 72, in parse_yaml
     return list(yaml.load_all(serialization, Loader=yaml_loader))
   File "/usr/lib/python3.8/site-packages/yaml/__init__.py", line 130, in load_all
     yield loader.get_data()
   File "/usr/lib/python3.8/site-packages/yaml/constructor.py", line 45, in get_data
     return self.construct_document(self.get_node())
   File "ext/_yaml.pyx", line 699, in _yaml.CParser.get_node
   File "ext/_yaml.pyx", line 725, in _yaml.CParser._compose_document
   File "ext/_yaml.pyx", line 776, in _yaml.CParser._compose_node
   File "ext/_yaml.pyx", line 890, in _yaml.CParser._compose_mapping_node
   File "ext/_yaml.pyx", line 732, in _yaml.CParser._compose_node
   File "ext/_yaml.pyx", line 905, in _yaml.CParser._parse_next_event
 yaml.scanner.ScannerError: while scanning an alias
   in "<unicode string>", line 7, column 7
 did not find expected alphabetic or numeric character
   in "<unicode string>", line 7, column 8
 2020/11/13 04:05:33 Memory Usage 0.17Gi (28%)
     PID 1, 0.00Gi: bash /buildroot/ambassador/python/entrypoint.sh
     PID 56, 0.04Gi: ambex --ads-listen-address=127.0.0.1:8003 /ambassador/envoy
     PID 59, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --kick kill -HUP 1
     PID 69, 0.00Gi (exited): sleep 1
     PID 73, 0.04Gi: /usr/bin/python /usr/bin/diagd /ambassador/snapshots /ambassador/bootstrap-ads.json /ambassador/envoy/envoy.json --notices /ambassador/notices.json --kick kill -HUP 1
     PID 82, 0.05Gi: watt --listen-address=127.0.0.1:8002 --notify python /ambassador/post_update.py --watt  --watch python /ambassador/watch_hook.py -s service -s ingresses -s AuthService -s Mapping -s Module -s RateLimitService -s TCPMapping -s TLSContext -s TracingSer
     PID 90, 0.11Gi: /ambassador/sidecars/amb-sidecar
 Calling Metriton

Expected behavior:

ambassador 1.9.0 should start by upgrading

vsimon commented 3 years ago

For anyone running into this do prior to upgrading:

kubectl apply -f https://www.getambassador.io/yaml/aes-crds.yaml

Helm doesn't manage CRD's. It’s kinda here in the yaml manifests section but it wasn’t obvious to make the connection

https://www.getambassador.io/docs/latest/topics/install/upgrading/#upgrading-an-installation-with-manifests