Open prasanth-openet opened 1 year ago
Some additional detail on the issue observered here.
This requires a least the following conditions:
gloo-system
Upstream
s defined yetWe use Consul integration as default discovery mechanism. Discovery is turned off, as we don't want those auto-discovered upstreams - the side effect is there are no upstreams, until we add our own later after Gloo chart installation.
From checking the OSS code, we observe that the SDS container seems to no be Ready as the Gloo container has not opened its GRPC port yet. This is why we only see the issue it if mTLS is enabled.
This seems to be due to the startup order where Gloo container only opens the GPRC port after it checks for healthy endpoints. If we set endpointsWarmingTimeout
to 0s
to disable feature, we do not have this problem.
Also, we set singleNamespace
and we install in a custom namespace (not gloo-system
). This seems to be an important part of the problem. We don't want Gloo looking in other namespaces in our case, as install is restricted to single installed namspace. We observed that the this does not occur if installed in gloo-system
namespace, but does if installed in any other namespace.
In summary, installing with the settings in original description, in a namespace other than gloo-system
will observer the problem, and chart installation will fail/timeout.
Currently the only way to workaround this is to either:
Upstream
s before installing Gloo chart
This was not an issue in previous versions of Gloo.
Further analysis, by process of elimination of versions, shows that the bug was introduced in 1.13.0-beta10.
This version introduces some new settings for Consul.
We also now using settings under consulUpstreamDiscovery
, as we have noticed that occasionally Gloo is out of sync with the services in Consul catalog. Changing these settings solve this.
However, these settings do not seem to directly affect the reproducibility of the issue in this ticket. However they may be relate given the features added in that beta release.
settings:
singleNamespace: true
disableKubernetesDestinations: true
integrations:
consul:
httpAddress: http://consul-consul-server.consul.svc:8500
dnsAddress: kube-dns.kube-system.svc:53
serviceDiscovery: {}
consulUpstreamDiscovery:
consistencyMode: ConsistentMode
edsBlockingQueries: true
queryOptions:
useCache: false
This issue has been marked as stale because of no activity in the last 180 days. It will be closed in the next 180 days unless it is tagged "no stalebot" or other activity occurs.
Gloo Edge Version
1.13.x (beta)
Kubernetes Version
v1.24.0
Describe the bug
I am trying to install gloo chart (version v1.13.0) on kubernetes in a namespace other than 'gloo-system'. However I could see the sds scontainer in the gloo pod is not getting ready. In my values.yaml file, I have disabled the service discovery and also i have provided a consul integration details. However, the above issue is not happening when I install version v1.12.56 or previous versions. Seems like it is broken from gloo versions >=1.13.0 onwards.
appreciate if you can help thank you.
Steps to reproduce the bug
1) Install the the chart in a namespace other than gloo-system.
helm upgrade --install gloo -n my-namespace --create-namespace --wait --debug --values values.yaml gloo-1.13.0.tgz
the following is my values.yaml file2) kubectl get pods -n gloo-system you could observe only 2 out 3 containers in the gloo pod is started.
the kubectl output is as follows
3)
glooctl check
gives the following resultsNot the issue is reproducible only if you use a namespace other than 'gloo-system'. In the test i am using 'my-namespace'. if we use the namespace gloo-system , the gloo pod is runs without any issues.
Expected Behavior
i should expect gloo pod should be started.
Additional Context
No response