Open Poweranimal opened 2 years ago
Came across this issue by accident. I'm not very familiar with SDS config either but my understanding is that the config is using static secret instead of SDS resources? According to the doc static secret is used if there's only a "name" field. I would suggest to have a ConfigSource with path pointing to a yaml/json resource file containing the paths to the key/cert, example:
tls_certificate_sds_secret_configs:
- name: zookeeper_cert
- sds_config
- path: /some/resource.json
- resource_api_version: V3
cat /some/resource.json
{
"resources": [
{
"@type": "type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret",
"name": "zookeeper_cert",
"tls_certificate": {
"certificate_chain": {
"filename": "/run/server.pem.crt"
},
"private_key": {
"filename": "/run/server.pem.key"
}
}
}
]
}
@wanlill thanks a lot for your input. I’m going to check this in the upcoming days.
Thanks @wanlill
I tried to set it up. However, envoy constantly crashes with segmentation fault.
I attached the error logs, the config.yaml
and resources.yaml
below.
I look forward to your support.
I added
node:
id: test
cluster: test
to my config.yaml
and now I get a little bit further but it still ends with a segmentation fault.
Ok, I finally was able to get it running.
In order to get it running, I created a resources.yaml
file for each sds type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
.
I attached by final config to the end of this comment.
However, my journey exposed three things in envoy's sds setup process that I think are worth improving:
node.id
and node.cluster
is missing. An error explaining the lack of these configuration parameters would help.sds_config.path_config_source.path
does not exists. An error explaining that the file does not exists would help.type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.Secret
is defined in resources file referenced by sds_config.path_config_source
. An error explaining this circumstance would help.Thanks these are all bugs that should be fixed to obviously not crash. I will mark help wanted. cc @kyessenov @lambdai who might be able to fix.
I don't have any idea why id and cluster is needed..
I couldn't get it to stop segfaulting, but it seems to be a regression in 1.22.x - downgrading to 1.21.4 stopped the crashing with same config (though changed path_config_source
to path
, since that syntax had changed)
Title: SDS file watching action not triggered
Description: The SDS file watch action does not get triggered in my setup.
The issue appeared in a kubernetes pod.
The pod has a container that regularly fetches certificates from vault and stores them in a shared volume attached to my envoy container.
Despite new certificates/keys being fetched, the SDS secrets do not get reloaded.
To verify that the inotify
MOVED_TO
event gets triggered by the certificate fetching container, I installedinotifywait
in the envoy container and watched the shared volume. Doing so I can confirm thatMOVED_TO
are triggered when new certificates get fetched. I can also confirm that the content of the files in the watched directory has changed.I attached my config to the issue,
I look forward to your support.
Config:
envoy.config
```yaml static_resources: secrets: - name: server_cert tls_certificate: certificate_chain: filename: /envoy/certs/public.crt private_key: filename: /envoy/certs/private.key - name: server_ca validation_context: trusted_ca: filename: /envoy/certs/ca.crt - name: zookeeper_cert tls_certificate: certificate_chain: filename: /envoy/certs/zookeeper_public.crt private_key: filename: /envoy/certs/zookeeper_private.key - name: zookeeper_ca validation_context: trusted_ca: filename: /envoy/certs/zookeeper_ca.crt clusters: - name: internal connect_timeout: 0.25s type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: internal endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 9093 - name: client connect_timeout: 0.25s type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: client endpoints: - lb_endpoints: - endpoint: address: socket_address: address: 127.0.0.1 port_value: 9092 - name: zookeeper connect_timeout: 0.25s type: STRICT_DNS lb_policy: ROUND_ROBIN load_assignment: cluster_name: zookeeper endpoints: - lb_endpoints: - endpoint: address: socket_address: address: zookeeper-0.zookeeper-headless.zookeeper.svc.cluster.local port_value: 3181 - endpoint: address: socket_address: address: zookeeper-1.zookeeper-headless.zookeeper.svc.cluster.local port_value: 3181 - endpoint: address: socket_address: address: zookeeper-2.zookeeper-headless.zookeeper.svc.cluster.local port_value: 3181 transport_socket: name: envoy.transport_sockets.tls typed_config: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.UpstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: zookeeper_cert validation_context_sds_secret_config: name: zookeeper_ca listeners: - name: listener_internal address: socket_address: address: 0.0.0.0 port_value: 19093 filter_chains: - filters: - name: envoy.filters.network.kafka_broker typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.kafka_broker.v3.KafkaBroker stat_prefix: kafka_broker - name: envoy.filters.network.tcp_proxy typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: tcp cluster: internal transport_socket: name: envoy.transport_sockets.tls typed_config: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: server_cert validation_context_sds_secret_config: name: server_ca - name: listener_client address: socket_address: address: 0.0.0.0 port_value: 19092 filter_chains: - filters: - name: envoy.filters.network.kafka_broker typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.kafka_broker.v3.KafkaBroker stat_prefix: kafka_broker - name: envoy.filters.network.tcp_proxy typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: tcp cluster: client transport_socket: name: envoy.transport_sockets.tls typed_config: '@type': type.googleapis.com/envoy.extensions.transport_sockets.tls.v3.DownstreamTlsContext common_tls_context: tls_certificate_sds_secret_configs: - name: server_cert validation_context_sds_secret_config: name: server_ca - name: listener_zookeeper address: socket_address: address: 127.0.0.1 port_value: 3181 filter_chains: - filters: - name: envoy.filters.network.tcp_proxy typed_config: '@type': type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy stat_prefix: tcp cluster: zookeeper ```Version: