search

emissary-ingress / emissary

open source Kubernetes-native API gateway for microservices built on the Envoy Proxy
https://www.getambassador.io
Apache License 2.0
4.37k stars 685 forks source link

ambassador pod in CrashLoopBackOff due to ConnectionRefusedError: [Errno 111] Connection refused when deploying ambassador1.4.2 at side of existing ambassador deploy #2654

Closed lsambolino closed 1 year ago

lsambolino commented 4 years ago

Describe the bug When trying to deploy Ambassador Edge Stack 1.4.2 via Kubernetes YAML on a Kubernetes v1.13, the ambassador pod refuses to start showing __Connection _refused_ error.

To Reproduce

  1. Install a Kubernetes v1.13 cluster
  2. Install Ambassador Edge via YAML following the Manual YAML Installation Page : 2a. Rename the aes-crds.yaml in aes142-crds.yaml and aes.yaml in aes142.yaml and customize the names of namespaces, categories, and serviceaccount with the same criteria (see following). 2b. apply YAML and set timeouts 
    kubectl apply -f https://www.getambassador.io/yaml/aes142-crds.yaml && \
kubectl wait --for condition=established --timeout=90s crd -lproduct=aes && \
kubectl apply -f https://www.getambassador.io/yaml/aes142.yaml && \
kubectl -n ambassador wait --for condition=available --timeout=90s deploy -lproduct=aes
  3. Check if ambassador pods are correctly running.

Expected behavior The ambassador pods should be deployed and go in Running state with no issues.

Versions:

Additional context

YAMLs: aes142-crds.yaml:

# GENERATED FILE: edits made by hand will not be preserved.

apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: authservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: AuthService
    plural: authservices
    singular: authservice
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: consulresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: ConsulResolver
    plural: consulresolvers
    singular: consulresolver
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: hosts.getambassador.io
spec:
  additionalPrinterColumns:
  - JSONPath: .spec.hostname
    name: Hostname
    type: string
  - JSONPath: .status.state
    name: State
    type: string
  - JSONPath: .status.phaseCompleted
    name: Phase Completed
    type: string
  - JSONPath: .status.phasePending
    name: Phase Pending
    type: string
  - JSONPath: .metadata.creationTimestamp
    name: Age
    type: date
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: Host
    plural: hosts
    singular: host
  scope: Namespaced
  subresources:
    status: {}
  versions:
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: kubernetesendpointresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador-crds
    kind: KubernetesEndpointResolver
    plural: kubernetesendpointresolvers
    singular: kubernetesendpointresolver
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: kubernetesserviceresolvers.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: KubernetesServiceResolver
    plural: kubernetesserviceresolvers
    singular: kubernetesserviceresolver
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: logservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: LogService
    plural: logservices
    singular: logservice
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: mappings.getambassador.io
spec:
  additionalPrinterColumns:
  - JSONPath: .spec.prefix
    name: Prefix
    type: string
  - JSONPath: .spec.service
    name: Service
    type: string
  - JSONPath: .status.state
    name: State
    type: string
  - JSONPath: .status.reason
    name: Reason
    type: string
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: Mapping
    plural: mappings
    singular: mapping
  scope: Namespaced
  subresources:
    status: {}
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: modules.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: Module
    plural: modules
    singular: module
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: ratelimitservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: RateLimitService
    plural: ratelimitservices
    singular: ratelimitservice
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: tcpmappings.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: TCPMapping
    plural: tcpmappings
    singular: tcpmapping
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: tlscontexts.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: TLSContext
    plural: tlscontexts
    singular: tlscontext
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: tracingservices.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: TracingService
    plural: tracingservices
    singular: tracingservice
  scope: Namespaced
  versions:
  - name: v1
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: filterpolicies.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: FilterPolicy
    plural: filterpolicies
    shortNames:
    - fp
    singular: filterpolicy
  scope: Namespaced
  versions:
  - name: v1beta1
    served: true
    storage: false
  - name: v1beta2
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: filters.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: Filter
    plural: filters
    shortNames:
    - fil
    singular: filter
  scope: Namespaced
  versions:
  - name: v1beta1
    served: true
    storage: false
  - name: v1beta2
    served: true
    storage: false
  - name: v2
    served: true
    storage: true
---
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
  labels:
    product: aes
  name: ratelimits.getambassador.io
spec:
  group: getambassador.io
  names:
    categories:
    - ambassador142-crds
    kind: RateLimit
    plural: ratelimits
    shortNames:
    - rl
    singular: ratelimit
  scope: Namespaced
  versions:
  - name: v1beta1
    served: true
    storage: false
  - name: v1beta2
    served: true
    storage: false
  - name: v2
    served: true
    storage: true

aes142.yaml:

# GENERATED FILE: edits made by hand will not be preserved.

######################################################################
# 'ambassador' namespace.
---
apiVersion: v1
kind: Namespace
metadata:
  name: ambassador142namespace
  labels:
    product: aes

######################################################################
# RBAC
# NB: ClusterRoles and ClusterRoleBindings are not namespaced
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: ambassador142serviceaccount
  namespace: ambassador142namespace
  labels:
    product: aes
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
  name: ambassador142clusterrole
  labels:
    product: aes
rules:
# Edge Stack mostly needs the same permissions as Ambassador OSS,
# but:
#  1. It needs extra permissions ("update"/"patch"/"create"/"delete")
#     for getambassador.io resources.
#  2. It needs extra permissions ("create"/"update") for secrets.
#  3. It needs permission to "create" events.
#  4. It needs permission to "get"/"create"/"update" Leases (or
#     Endpoints, if on Kubernetes <1.12).
#
# Note: If you edit this, be sure to also edit the pytest
# `manifests/rbac_*_scope.yaml` files.
- apiGroups: [""]
  resources: [ "namespaces", "services", "pods" ]
  verbs: ["get", "list", "watch"]
- apiGroups: [ "getambassador.io" ]
  resources: [ "*" ]
  verbs: ["get", "list", "watch", "update", "patch", "create", "delete" ]
- apiGroups: [ "apiextensions.k8s.io" ]
  resources: [ "customresourcedefinitions" ]
  verbs: ["get", "list", "watch"]
- apiGroups: [ "networking.internal.knative.dev" ]
  resources: [ "clusteringresses", "ingresses" ]
  verbs: ["get", "list", "watch"]
- apiGroups: [ "networking.internal.knative.dev" ]
  resources: [ "ingresses/status", "clusteringresses/status" ]
  verbs: ["update"]
- apiGroups: [ "extensions", "networking.k8s.io" ]
  resources: [ "ingresses" ]
  verbs: ["get", "list", "watch"]
- apiGroups: [ "extensions", "networking.k8s.io" ]
  resources: [ "ingresses/status" ]
  verbs: ["update"]
- apiGroups: [""]
  resources: [ "secrets" ]
  verbs: ["get", "list", "watch", "create", "update"]
- apiGroups: [""]
  resources: [ "events" ]
  verbs: ["get", "list", "watch", "create"]
- apiGroups: ["coordination.k8s.io"]
  resources: [ "leases" ]
  verbs: ["get", "create", "update"]
- apiGroups: [""]
  resources: [ "endpoints" ]
  verbs: ["get", "list", "watch", "create", "update"]
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
  name: ambassador142clusterrolebinding
  labels:
    product: aes
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: ambassador142clusterrole
subjects:
- kind: ServiceAccount
  name: ambassador142serviceaccount
  namespace: ambassador142namespace

######################################################################
# Redis
---
apiVersion: v1
kind: Service
metadata:
  name: ambassador142-redis
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  type: ClusterIP
  ports:
  - port: 6379
    targetPort: 6379
  selector:
    service: ambassador142-redis
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: ambassador142-redis
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  replicas: 1
  selector:
    matchLabels:
      service: ambassador142-redis
  template:
    metadata:
      labels:
        service: ambassador142-redis
    spec:
      containers:
      - name: redis
        image: redis:5.0.1
      restartPolicy: Always

######################################################################
# Configure Ambassador Edge Stack.
---
apiVersion: getambassador.io/v2
kind: RateLimitService
metadata:
  name: ambassador142-edge-stack-ratelimit
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  service: "127.0.0.1:8500"
---
apiVersion: getambassador.io/v2
kind: AuthService
metadata:
  name: ambassador-edge-stack-auth
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  proto: grpc
  status_on_error:
    code: 504
  auth_service: "127.0.0.1:8500"
  allow_request_body: false # setting this to 'true' allows Plugin and External filters to access the body, but has performance overhead
---
apiVersion: v1
kind: Secret
metadata:
  name: ambassador142-edge-stack
  namespace: ambassador142namespace
data:
  license-key: "" # This secret is just a placeholder, it is mounted as a volume and refreshed when changed

######################################################################
# Configure DevPortal
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  # This Mapping name is referenced by convention, it's important to leave as-is.
  name: ambassador142-devportal
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  prefix: /docs/
  rewrite: "/docs/"
  service: "127.0.0.1:8500"
---
apiVersion: getambassador.io/v2
kind: Mapping
metadata:
  # This Mapping name is referenced by convention, it's important to leave as-is.
  name: ambassador142-devportal-api
  namespace: ambassador142namespace
  labels:
    product: aes
spec:
  prefix: /openapi/
  rewrite: ""
  service: "127.0.0.1:8500"

######################################################################
# Create the 'ambassador' and 'ambassador-admin' Services.
---
apiVersion: v1
kind: Service
metadata:
  name: ambassador142
  namespace: ambassador142namespace
  labels:
    product: aes
    app.kubernetes.io/component: ambassador142-service
spec:
  type: LoadBalancer
  ports:
  - name: http
    port: 80
    targetPort: http
  - name: https
    port: 443
    targetPort: https
  selector:
    service: ambassador142
---
apiVersion: v1
kind: Service
metadata:
  labels:
    service: ambassador142-admin
    product: aes
  name: ambassador142-admin
  namespace: ambassador142namespace
spec:
  type: ClusterIP
  ports:
  - name: ambassador142-admin
    port: 8878
    targetPort: admin
  selector:
    service: ambassador142

######################################################################
# Create the Deployment backing the 'ambassador' and
# 'ambassador-admin' Services.
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    product: aes
  name: ambassador142
  namespace: ambassador142namespace
spec:
  replicas: 1
  selector:
    matchLabels:
      service: ambassador142
  template:
    metadata:
      annotations:
        consul.hashicorp.com/connect-inject: 'false'
        sidecar.istio.io/inject: 'false'
      labels:
        app.kubernetes.io/managed-by: getambassador.io
        service: ambassador142
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  service: ambassador142
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - name: AMBASSADOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: REDIS_URL
          value: 10.11.135.8:6379
        - name: AMBASSADOR_URL
          value: https://ambassador.default.svc.cluster.local
        - name: POLL_EVERY_SECS
          value: '60'
        - name: AMBASSADOR_INTERNAL_URL
          value: https://127.0.0.1:8443
        - name: AMBASSADOR_ADMIN_URL
          value: http://127.0.0.1:8878
        image: quay.io/datawire/aes:1.4.2
        imagePullPolicy: IfNotPresent
        livenessProbe:
          httpGet:
            path: /ambassador/v0/check_alive
            port: 8878
          periodSeconds: 3
          initialDelaySeconds: 3
        name: aes
        ports:
        - containerPort: 8080
          name: http
        - containerPort: 8443
          name: https
        - containerPort: 8878
          name: admin
        readinessProbe:
          httpGet:
            path: /ambassador/v0/check_ready
            port: 8878
          periodSeconds: 3
          initialDelaySeconds: 3
        resources:
          limits:
            cpu: 1000m
            memory: 600Mi
          requests:
            cpu: 200m
            memory: 300Mi
        volumeMounts:
        - mountPath: /tmp/ambassador-pod-info
          name: ambassador142-pod-info
        - mountPath: /.config/ambassador
          name: ambassador142-edge-stack-secrets
          readOnly: true
      restartPolicy: Always
      securityContext:
        runAsUser: 8888
      serviceAccountName: ambassador142serviceaccount
      terminationGracePeriodSeconds: 0
      volumes:
      - downwardAPI:
          items:
          - fieldRef:
              fieldPath: metadata.labels
            path: labels
        name: ambassador142-pod-info
      - name: ambassador142-edge-stack-secrets
        secret:
          secretName: ambassador142-edge-stack

The output of applying the original YAMLs is:

namespace/ambassador142namespace configured
serviceaccount/ambassador142serviceaccount configured
clusterrole.rbac.authorization.k8s.io/ambassador142clusterrole configured
clusterrolebinding.rbac.authorization.k8s.io/ambassador142clusterrolebinding configured
service/ambassador142-redis configured
deployment.apps/ambassador142-redis configured
ratelimitservice.getambassador.io/ambassador142-edge-stack-ratelimit configured
authservice.getambassador.io/ambassador-edge-stack-auth configured
secret/ambassador142-edge-stack configured
mapping.getambassador.io/ambassador142-devportal configured
mapping.getambassador.io/ambassador142-devportal-api configured
service/ambassador142 configured
service/ambassador142-admin configured
deployment.apps/ambassador142 configured
deployment.extensions/ambassador142-redis condition met
error: timed out waiting for the condition

The log of ambassador pod is:

2020-04-29 08:43:46 diagd 1.4.2 [P154TAEW] INFO: Scout reports {"latest_version": "1.0.0", "application": "aes", "cached": true, "timestamp": 1588149825.645167}
2020-04-29 08:43:46 diagd 1.4.2 [P154TAEW] INFO: Scout notices: [{"level": "DEBUG", "message": "Returning cached result"}]
time="2020-04-29T08:43:46Z" level=info msg="Loaded file /ambassador/envoy/envoy.json"
2020/04/29 08:43:46 notify: Traceback (most recent call last):
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 159, in _new_conn
2020/04/29 08:43:46 notify:     (self._dns_host, self.port), self.timeout, **extra_kw)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 80, in create_connection
2020/04/29 08:43:46 notify:     raise err
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/util/connection.py", line 70, in create_connection
2020/04/29 08:43:46 notify:     sock.connect(sa)
2020/04/29 08:43:46 notify: ConnectionRefusedError: [Errno 111] Connection refused
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: During handling of the above exception, another exception occurred:
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: Traceback (most recent call last):
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 600, in urlopen
2020/04/29 08:43:46 notify:     chunked=chunked)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 354, in _make_request
2020/04/29 08:43:46 notify:     conn.request(method, url, **httplib_request_kw)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/http/client.py", line 1252, in request
2020/04/29 08:43:46 notify:     self._send_request(method, url, body, headers, encode_chunked)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/http/client.py", line 1298, in _send_request
time="2020-04-29T08:43:46Z" level=info msg="Pushing snapshot v2"
[2020-04-29 08:43:46.846][201][info][upstream] [source/common/upstream/cds_api_impl.cc:67] cds: add 3 cluster(s), remove 1 cluster(s)
[2020-04-29 08:43:46.846][201][warning][misc] [source/common/protobuf/utility.cc:282] Using deprecated option 'envoy.api.v2.listener.FilterChain.tls_context' from file listener.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
[2020-04-29 08:43:46.846][201][warning][misc] [source/common/protobuf/utility.cc:282] Using deprecated option 'envoy.api.v2.listener.ListenerFilter.config' from file listener.proto. This configuration will be removed from Envoy soon. Please see https://www.envoyproxy.io/docs/envoy/latest/intro/deprecated for details.
2020/04/29 08:43:46 notify:     self.endheaders(body, encode_chunked=encode_chunked)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/http/client.py", line 1247, in endheaders
2020/04/29 08:43:46 notify:     self._send_output(message_body, encode_chunked=encode_chunked)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/http/client.py", line 1026, in _send_output
2020/04/29 08:43:46 notify:     self.send(msg)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/http/client.py", line 966, in send
2020/04/29 08:43:46 notify:     self.connect()
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 181, in connect
2020/04/29 08:43:46 notify:     conn = self._new_conn()
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connection.py", line 168, in _new_conn
2020/04/29 08:43:46 notify:     self, "Failed to establish a new connection: %s" % e)
2020/04/29 08:43:46 notify: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7fa619083390>: Failed to establish a new connection: [Errno 111] Connection refused
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: During handling of the above exception, another exception occurred:
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: Traceback (most recent call last):
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
2020/04/29 08:43:46 notify:     timeout=timeout
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/connectionpool.py", line 638, in urlopen
2020/04/29 08:43:46 notify:     _stacktrace=sys.exc_info()[2])
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/urllib3/util/retry.py", line 399, in increment
2020/04/29 08:43:46 notify:     raise MaxRetryError(_pool, url, error or ResponseError(cause))
2020/04/29 08:43:46 notify: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8500): Max retries exceeded with url: /_internal/v0/watt?url=http%3A%2F%2Flocalhost%3A8002%2Fsnapshots%2F2 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa619083390>: Failed to establish a new connection: [Errno 111] Connection refused'))
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: During handling of the above exception, another exception occurred:
2020/04/29 08:43:46 notify: 
2020/04/29 08:43:46 notify: Traceback (most recent call last):
2020/04/29 08:43:46 notify:   File "/ambassador/post_update.py", line 50, in <module>
2020/04/29 08:43:46 notify:     r = requests.post(url, params={ arg_key: args[0] })
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/api.py", line 116, in post
2020/04/29 08:43:46 notify:     return request('post', url, data=data, json=json, **kwargs)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/api.py", line 60, in request
2020/04/29 08:43:46 notify:     return session.request(method=method, url=url, **kwargs)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 533, in request
2020/04/29 08:43:46 notify:     resp = self.send(prep, **send_kwargs)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/sessions.py", line 646, in send
2020/04/29 08:43:46 notify:     r = adapter.send(request, **kwargs)
2020/04/29 08:43:46 notify:   File "/usr/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
2020/04/29 08:43:46 notify:     raise ConnectionError(e, request=request)
2020/04/29 08:43:46 notify: requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8500): Max retries exceeded with url: /_internal/v0/watt?url=http%3A%2F%2Flocalhost%3A8002%2Fsnapshots%2F2 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa619083390>: Failed to establish a new connection: [Errno 111] Connection refused'))
kflynn commented 4 years ago

Is that the entire pod log? It looks like a piece of Ambassador died early on, and that's what would be helpful to see...

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

cindymullins-dw commented 1 year ago

Closing this issue. If it persists on newer versions 2.x or 3.x please feel free to reopen.