Open jinjianming opened 1 year ago
Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=1 Should I issue a certificate for the domain name "apisix-control-plane-control-plane"?
You can refer to some test cases in the APISIX repo.
Thank you for the test cases provided. I configure and generate certificates in this way, although I have achieved the effect of test cases;
But 'apisix-data-plane' is still in the CrashLoopBackOff state. I suspect it is a self-signed certificate problem;
Because curl needs to add the - k parameter to get the content.
curl --cert tls.crt --key tls.key https://apisix-control-plane-control-plane:9280/version curl: (60) SSL certificate problem: unable to get local issuer certificate
curl --cert tls.crt --key tls.key -k https://apisix-control-plane-control-plane:9280/version {"etcdserver":"3.5.4","etcdcluster":"3.5.0"}
kubectl logs apisix-data-plane-566cb54f9d-bbjhv -f /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed all etcd nodes are unavailable Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=1 Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=2
deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: "/conf-server-ssl/tls.crt"
cert_key: "/conf-server-ssl/tls.key"
client_ca_cert: "/conf-ca-ssl/tls.crt"
certs:
cert: "/conf-client-ssl/tls.crt"
cert_key: "/conf-client-ssl/tls.key"
trusted_ca_cert: "/conf-ca-ssl/tls.crt"
deployment:
role: data_plane
role_data_plane:
config_provider: control_plane
control_plane:
host:
- "https://apisix-control-plane-control-plane:9280"
prefix: /apisix
timeout: 30
certs:
cert: "/conf-client-ssl/tls.crt"
cert_key: "/conf-client-ssl/tls.key"
trusted_ca_cert: "/conf-ca-ssl/tls.crt"
@jinjianming Did you try to connect to the https://apisix-control-plane-control-plane:9280
manually? With the TLS bundles configured on the APISIX data plane. Make sure:
@tokers Hello I am a certificate generated through openssl genrsa; request will prompt "failed to verify the legitimacy of the server and there could not establish a secure connection to it".
curl -v https://apisix-control-plane-control-plane:9280/version ...
With the curl command, I can get normal results through the "- k" parameter
curl -v -k https://apisix-control-plane-control-plane:9280/version
GET /version HTTP/1.1 Host: apisix-control-plane-control-plane:9280
{"etcdserver":"3.5.4","etcdcluster":"3.5.0"} ...
Whether Apisix has configuration to skip ssl certificate check or use http, because I am an intranet environment.
@tokers In addition“ https://apisix-control-plane-control-plane:9280/version "This address is the SVC address of K8S.
The issuing authority cannot sign for the svc name; Whether the load balancer needs to be deployed separately to achieve dp access to the cp.
@tokers In addition“ https://apisix-control-plane-control-plane:9280/version "This address is the SVC address of K8S.
The issuing authority cannot sign for the svc name; Whether the load balancer needs to be deployed separately to achieve dp access to the cp.
You should add the--cacert option to specify the CA cert.
@tokers At present, I add --cacert can return normal results, but data-plane still CrashLoopBackOff
curl -v --cacert /conf-ca-ssl/tls.crt --cert /conf-client-ssl/tls.crt --key /conf-client-ssl/tls.key https://apisix-control-plane-control-plane:9280/version
< HTTP/1.1 200 OK
{"etcdserver":"3.5.4","etcdcluster":"3.5.0"}
@tokers Hello
I tried to change the command of data-plane to "sleep 360" to make its data-plane PODS in running status;
Then I went into data-plane PODS and executed "curl - v -- cacert/conf-ca-ssl/tls.crt -- cert/conf-client-ssl/tls.crt -- key/conf-client-ssl/tls.key" https://apisix-control-plane-control-plane:9280/version "Normal results can be obtained;
deployment: role: data_plane role_data_plane: config_provider: control_plane control_plane: host:
you can try to use openssl verify
to verify certs.
e.g. for the test case https://github.com/apache/apisix/blob/4ab50da569fb209a91c035a7b951f54fbbc9cd56/t/deployment/mtls.t#L47-L63
➜ certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_client.crt
mtls_client.crt: OK
➜ certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_server.crt
mtls_server.crt: OK
您可以尝试使用
openssl verify
来验证证书。例如,对于测试用例https://github.com/apache/apisix/blob/4ab50da569fb209a91c035a7b951f54fbbc9cd56/t/deployment/mtls.t#L47-L63
➜ certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_client.crt mtls_client.crt: OK ➜ certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_server.crt mtls_server.crt: OK
@tao12345666333
我在control-plane PODS 里面执行openssl verify是ok的.apisix@apisix-control-plane-6d5699b8b-n7l7f:/usr/local/apisix$ openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt /conf-client-ssl/tls.crt: OK apisix@apisix-control-plane-6d5699b8b-n7l7f:/usr/local/apisix$ openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-server-ssl/tls.crt /conf-server-ssl/tls.crt: OK
Can you check what happens in data-plane role APISIX ?
你能检查一下数据平面角色 APISIX 中发生了什么吗?
The test in the data plane also passed
apisix@apisix-data-plane-5c589b88df-v6td9:/usr/local/apisix$ openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt
/conf-client-ssl/tls.crt: OK
你能查一下数据平面角APISIX 中生成了什么吗?
数据面的测试也通过了
apisix@apisix-data-plane-5c589b88df-v6td9:/usr/local/apisix$ openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt /conf-client-ssl/tls.crt: OK
I generated the certificate according to this document (https://apisix.apache.org/docs/apisix/tutorials/client-to-apisix-mtls/#generate-certificates)
@tao12345666333 @tokers Hello, I'm here( https://github.com/apache/apisix/issues/8067#issuecomment-1276976555 ) found a solution and added tls. verify: false;
Is it because the name is not in the domain name format; (apisix-control-plane-control-plane)
Here is my CN information(https://github.com/apache/apisix-helm-chart/issues/479#issuecomment-1442700906)
subject: CN=apisix-control-plane-control-plane
@tao12345666333 @tokers I found the root cause ssl_trusted_certificate set the CA cert that issued the cert of conf_server in the control plane.
apisix:
ssl:
ssl_trusted_certificate: t/certs/mtls_ca.crt
Configure trusted_ ca_ Cert
didn't get the expected effect
@jinjianming could you provide your full configuration of data plane and control plane?
@jinjianming你能提供你的数据平面和控制平面的完整配置吗?
https://github.com/apache/apisix/issues/9038#issuecomment-1461589293
@jinjianming你能提供你的数据平面和控制平面的完整配置吗?
@jinjianming I've seen there're many configurations, which is the correct one?
1、CP-Config
apisix: # universal configurations
enable_heartbeat: true
enable_admin: true
enable_admin_cors: true
enable_debug: false
enable_dev_mode: false # Sets nginx worker_processes to 1 if set to true
enable_reuseport: true # Enable nginx SO_REUSEPORT switch if set to true.
enable_ipv6: true # Enable nginx IPv6 resolver
enable_server_tokens: true # Whether the APISIX version number should be shown in Server header
admin_listen.
# # This port can only receive http request with proxy protocol, but node_listen & admin_listen
# # can only receive http request. If you enable proxy protocol, you must use this port to
option
# enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server
proxy_cache: # Proxy Caching configuration
cache_ttl: 10s # The default caching time if the upstream does not specify the cache time
zones: # The parameters of a cache
- name: disk_cache_one # The name of the cache, administrator can be specify
# which cache to use by name in the admin api
memory_size: 50m # The size of shared memory, it's used to store the cache index
disk_size: 1G # The size of disk, it's used to store the cache data
disk_path: "/tmp/disk_cache_one" # The path to store the cache data
cache_levels: "1:2" # The hierarchy levels of a cache
router:
http: radixtree_uri # radixtree_uri: match route by uri(base on radixtree)
# radixtree_host_uri: match route by host + uri(base on radixtree)
# radixtree_uri_with_parameter: match route by uri with parameters
ssl: 'radixtree_sni' # radixtree_sni: match route by SNI(base on radixtree)
stream_proxy: # TCP/UDP proxy
only: false
tcp: # TCP proxy port list
- 9100
udp: # UDP proxy port list
- 9200
dns_resolver:
- 10.233.0.3
dns_resolver_valid: 30
resolver_timeout: 5
ssl:
enable: true
listen:
- port: 9443
enable_http2: true
ssl_protocols: "TLSv1.2 TLSv1.3"
ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
nginx_config: # config for render the template to genarate nginx.conf
error_log: "/dev/stderr"
error_log_level: "debug" # warn,error
worker_processes: "auto"
enable_cpu_affinity: true
worker_rlimit_nofile: 20480 # the number of files a worker process can open, should be larger than worker_connections
event:
worker_connections: 10620
http:
enable_access_log: true
access_log: "/dev/stdout"
access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
access_log_format_escape: default
keepalive_timeout: 60s # timeout during which a keep-alive client connection will stay open on the server side.
client_header_timeout: 60s # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
client_body_timeout: 60s # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
send_timeout: 10s # timeout for transmitting a response to the client.then the connection is closed
underscores_in_headers: "on" # default enables the use of underscores in client request header fields
real_ip_header: "X-Real-IP" # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
real_ip_from: # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
- 127.0.0.1
- 10.233.0.0/16
- 'unix:'
plugins: # plugin list
- api-breaker
- xxx
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
deployment:
role: control_plane
role_control_plane:
config_provider: etcd
conf_server:
listen: 0.0.0.0:9280
cert: "/conf-server-ssl/tls.crt"
cert_key: "/conf-server-ssl/tls.key"
admin:
allow_admin: # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
- 127.0.0.1/24
- 0.0.0.0/0
# - "::/64"
admin_listen:
ip: 0.0.0.0
port: 9180
# Default token when use API to call for Admin API.
# *NOTE*: Highly recommended to modify this value to protect APISIX's Admin API.
# Disabling this configuration item means that the Admin API does not
# require any authentication.
admin_key:
# admin: can everything for configuration data
- name: "admin"
key: edd1c9f034335f136f87ad84b625c8f1
role: admin
# viewer: only can view configuration data
- name: "viewer"
key: 4054f7cf07e344346cd3f287985e76a2
role: viewer
etcd:
host: # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
- "http://apisix-control-plane-etcd.ingress-apisix.svc.cluster.local:2379"
prefix: "/apisix" # configuration prefix in etcd
timeout: 30 # 30 seconds
certs:
cert: "/conf-client-ssl/tls.crt"
cert_key: "/conf-client-ssl/tls.key"
2、DP-config
apisix: # universal configurations
ssl:
ssl_trusted_certificate: "/conf-ca-ssl/tls.crt"
node_listen: 9080 # APISIX listening port
enable_heartbeat: true
enable_admin: false
enable_admin_cors: true
enable_debug: false
enable_dev_mode: false # Sets nginx worker_processes to 1 if set to true
enable_reuseport: true # Enable nginx SO_REUSEPORT switch if set to true.
enable_ipv6: true # Enable nginx IPv6 resolver
enable_server_tokens: true # Whether the APISIX version number should be shown in Server header
# proxy_protocol: # Proxy Protocol configuration
# listen_http_port: 9181 # The port with proxy protocol for http, it differs from node_listen and admin_listen.
# # This port can only receive http request with proxy protocol, but node_listen & admin_listen
# # can only receive http request. If you enable proxy protocol, you must use this port to
# # receive http request with proxy protocol
# listen_https_port: 9182 # The port with proxy protocol for https
# enable_tcp_pp: true # Enable the proxy protocol for tcp proxy, it works for stream_proxy.tcp option
# enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server
proxy_cache: # Proxy Caching configuration
cache_ttl: 10s # The default caching time if the upstream does not specify the cache time
zones: # The parameters of a cache
- name: disk_cache_one # The name of the cache, administrator can be specify
# which cache to use by name in the admin api
memory_size: 50m # The size of shared memory, it's used to store the cache index
disk_size: 1G # The size of disk, it's used to store the cache data
disk_path: "/tmp/disk_cache_one" # The path to store the cache data
cache_levels: "1:2" # The hierarchy levels of a cache
router:
http: radixtree_uri # radixtree_uri: match route by uri(base on radixtree)
# radixtree_host_uri: match route by host + uri(base on radixtree)
# radixtree_uri_with_parameter: match route by uri with parameters
ssl: 'radixtree_sni' # radixtree_sni: match route by SNI(base on radixtree)
dns_resolver:
- 10.233.0.3
dns_resolver_valid: 30
resolver_timeout: 5
ssl:
enable: true
listen:
- port: 9443
enable_http2: false
ssl_protocols: "TLSv1.2 TLSv1.3"
ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"
nginx_config: # config for render the template to genarate nginx.conf
error_log: "/dev/stderr"
error_log_level: "debug" # warn,error
worker_processes: "auto"
enable_cpu_affinity: true
worker_rlimit_nofile: 20480 # the number of files a worker process can open, should be larger than worker_connections
event:
worker_connections: 10620
http:
enable_access_log: true
access_log: "/dev/stdout"
access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
access_log_format_escape: default
keepalive_timeout: 60s # timeout during which a keep-alive client connection will stay open on the server side.
client_header_timeout: 60s # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
client_body_timeout: 60s # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
send_timeout: 10s # timeout for transmitting a response to the client.then the connection is closed
underscores_in_headers: "on" # default enables the use of underscores in client request header fields
real_ip_header: "X-Real-IP" # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
real_ip_from: # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
- 127.0.0.1
- 10.233.0.0/16
- 'unix:'
plugins: # plugin list
- api-breaker
- xxx
stream_plugins:
- mqtt-proxy
- ip-restriction
- limit-conn
deployment:
role: data_plane
role_data_plane:
config_provider: control_plane
control_plane:
host:
- "https://apisix-control-plane-control-plane:9280"
prefix: "/apisix"
timeout: 30
certs:
cert: "/conf-client-ssl/tls.crt"
cert_key: "/conf-client-ssl/tls.key"
How to generate available apisix certificates used for certificates in decoupled mode.
{"cert":"","cert_key":"","certsSecret":"","mTLSCACert":"","mTLSCACertSecret":""}
At present, I want to pass[ https://github.com/apache/apisix-helm-chart/tree/apisix-1.1.1/charts/apisix ]To deploy the decoupled mode. Now the certificate problem makes me unable to deploy successfully;
I tried some self-signed certificates, but still failed to start.