jinjianming commented 1 year ago

How to generate available apisix certificates used for certificates in decoupled mode.

{"cert":"","cert_key":"","certsSecret":"","mTLSCACert":"","mTLSCACertSecret":""}

At present, I want to pass[ https://github.com/apache/apisix-helm-chart/tree/apisix-1.1.1/charts/apisix ]To deploy the decoupled mode. Now the certificate problem makes me unable to deploy successfully;

I tried some self-signed certificates, but still failed to start.

jinjianming commented 1 year ago

Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=1 Should I issue a certificate for the domain name "apisix-control-plane-control-plane"?

tao12345666333 commented 1 year ago

You can refer to some test cases in the APISIX repo.

https://github.com/apache/apisix/blob/4ab50da569fb209a91c035a7b951f54fbbc9cd56/t/deployment/mtls.t#L47-L63

https://github.com/apache/apisix/tree/master/t/certs

jinjianming commented 1 year ago

Thank you for the test cases provided. I configure and generate certificates in this way, although I have achieved the effect of test cases;

But 'apisix-data-plane' is still in the CrashLoopBackOff state. I suspect it is a self-signed certificate problem;

Because curl needs to add the - k parameter to get the content.

curl --cert tls.crt --key tls.key https://apisix-control-plane-control-plane:9280/version curl: (60) SSL certificate problem: unable to get local issuer certificate

curl --cert tls.crt --key tls.key -k https://apisix-control-plane-control-plane:9280/version {"etcdserver":"3.5.4","etcdcluster":"3.5.0"}

kubectl logs apisix-data-plane-566cb54f9d-bbjhv -f /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init /usr/local/openresty//luajit/bin/luajit ./apisix/cli/apisix.lua init_etcd request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed all etcd nodes are unavailable Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=1 Warning! Request etcd endpoint 'https://apisix-control-plane-control-plane:9280/version' error, certificate verify failed, retry time=2

Does the Apisix data plane have parameters that can use SSL to allow insecure server connections

jinjianming commented 1 year ago

This is my configuration list;

deployment:
  role: control_plane
  role_control_plane:
    config_provider: etcd
    conf_server:
      listen: 0.0.0.0:9280
      cert: "/conf-server-ssl/tls.crt"
      cert_key: "/conf-server-ssl/tls.key"
      client_ca_cert: "/conf-ca-ssl/tls.crt"
  certs:
    cert: "/conf-client-ssl/tls.crt"
    cert_key: "/conf-client-ssl/tls.key"
    trusted_ca_cert: "/conf-ca-ssl/tls.crt"

deployment:
  role: data_plane
  role_data_plane:
    config_provider: control_plane
    control_plane:
      host:
        - "https://apisix-control-plane-control-plane:9280"
      prefix: /apisix
      timeout: 30
  certs:
    cert: "/conf-client-ssl/tls.crt"
    cert_key: "/conf-client-ssl/tls.key"
    trusted_ca_cert: "/conf-ca-ssl/tls.crt"

tokers commented 1 year ago

@jinjianming Did you try to connect to the https://apisix-control-plane-control-plane:9280 manually? With the TLS bundles configured on the APISIX data plane. Make sure:

The control plane server returns the certificate which CN or SANs match the TLS SNI;
The CA cert configured on the APISIX data plane does can verify the control plane certificate;

jinjianming commented 1 year ago

@tokers Hello I am a certificate generated through openssl genrsa; request will prompt "failed to verify the legitimacy of the server and there could not establish a secure connection to it".

curl -v https://apisix-control-plane-control-plane:9280/version ...

TLSv1.2 (OUT), TLS alert, unknown CA (560):
SSL certificate problem: unable to get local issuer certificate
Closing connection 0 curl: (60) SSL certificate problem: unable to get local issuer certificate ...

With the curl command, I can get normal results through the "- k" parameter

curl -v -k https://apisix-control-plane-control-plane:9280/version

SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
ALPN, server accepted to use http/1.1
Server certificate:
subject: CN=apisix-control-plane-control-plane
start date: Feb 23 11:59:51 2023 GMT
expire date: Jan 30 11:59:51 2123 GMT
issuer: CN=Encryption Everywhere DV TLS CA - G1
SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.

GET /version HTTP/1.1 Host: apisix-control-plane-control-plane:9280

{"etcdserver":"3.5.4","etcdcluster":"3.5.0"} ...

Whether Apisix has configuration to skip ssl certificate check or use http, because I am an intranet environment.

jinjianming commented 1 year ago

@tokers In addition“ https://apisix-control-plane-control-plane:9280/version "This address is the SVC address of K8S.

The issuing authority cannot sign for the svc name; Whether the load balancer needs to be deployed separately to achieve dp access to the cp.

tokers commented 1 year ago

@tokers In addition“ https://apisix-control-plane-control-plane:9280/version "This address is the SVC address of K8S.

The issuing authority cannot sign for the svc name; Whether the load balancer needs to be deployed separately to achieve dp access to the cp.

You should add the--cacert option to specify the CA cert.

jinjianming commented 1 year ago

@tokers At present, I add --cacert can return normal results, but data-plane still CrashLoopBackOff

curl -v --cacert /conf-ca-ssl/tls.crt --cert /conf-client-ssl/tls.crt --key /conf-client-ssl/tls.key https://apisix-control-plane-control-plane:9280/version

Server certificate:
subject: CN=apisix-control-plane-control-plane
start date: Feb 24 01:54:18 2023 GMT
expire date: Jan 31 01:54:18 2123 GMT
common name: apisix-control-plane-control-plane (matched)
issuer: CN=ROOTCA
SSL certificate verify ok.

< HTTP/1.1 200 OK

Connection #0 to host apisix-control-plane-control-plane left intact
{"etcdserver":"3.5.4","etcdcluster":"3.5.0"}

jinjianming commented 1 year ago

@tokers Hello

I tried to change the command of data-plane to "sleep 360" to make its data-plane PODS in running status;
Then I went into data-plane PODS and executed "curl - v -- cacert/conf-ca-ssl/tls.crt -- cert/conf-client-ssl/tls.crt -- key/conf-client-ssl/tls.key" https://apisix-control-plane-control-plane:9280/version "Normal results can be obtained;
However, the data-plane itself cannot be started successfully. My data-plane configuration:

deployment: role: data_plane role_data_plane: config_provider: control_plane control_plane: host:
- "https://apisix-control-plane-control-plane:9280" prefix: /apisix timeout: 30 certs: cert: "/conf-client-ssl/tls.crt" cert_key: "/conf-client-ssl/tls.key" trusted_ca_cert: "/conf-ca-ssl/tls.crt"

tao12345666333 commented 1 year ago

you can try to use openssl verify to verify certs.

e.g. for the test case https://github.com/apache/apisix/blob/4ab50da569fb209a91c035a7b951f54fbbc9cd56/t/deployment/mtls.t#L47-L63

➜  certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_client.crt
mtls_client.crt: OK
➜  certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_server.crt
mtls_server.crt: OK

jinjianming commented 1 year ago

您可以尝试使用openssl verify来验证证书。

例如，对于测试用例https://github.com/apache/apisix/blob/4ab50da569fb209a91c035a7b951f54fbbc9cd56/t/deployment/mtls.t#L47-L63
➜  certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_client.crt
mtls_client.crt: OK
➜  certs git:(master) openssl verify -CAfile ./mtls_ca.crt mtls_server.crt
mtls_server.crt: OK
@tao12345666333
我在control-plane PODS 里面执行openssl verify是ok的.
apisix@apisix-control-plane-6d5699b8b-n7l7f:/usr/local/apisix$  openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt 
/conf-client-ssl/tls.crt: OK
apisix@apisix-control-plane-6d5699b8b-n7l7f:/usr/local/apisix$  openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-server-ssl/tls.crt 
/conf-server-ssl/tls.crt: OK

tao12345666333 commented 1 year ago

Can you check what happens in data-plane role APISIX ?

jinjianming commented 1 year ago

你能检查一下数据平面角色 APISIX 中发生了什么吗？

The test in the data plane also passed

apisix@apisix-data-plane-5c589b88df-v6td9:/usr/local/apisix$  openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt 
/conf-client-ssl/tls.crt: OK

jinjianming commented 1 year ago

你能查一下数据平面角APISIX 中生成了什么吗？

数据面的测试也通过了
apisix@apisix-data-plane-5c589b88df-v6td9:/usr/local/apisix$  openssl verify -CAfile /conf-ca-ssl/tls.crt /conf-client-ssl/tls.crt 
/conf-client-ssl/tls.crt: OK

I generated the certificate according to this document (https://apisix.apache.org/docs/apisix/tutorials/client-to-apisix-mtls/#generate-certificates)

jinjianming commented 1 year ago

@tao12345666333 @tokers Hello, I'm here（ https://github.com/apache/apisix/issues/8067#issuecomment-1276976555 ) found a solution and added tls. verify: false;

I think the reason for the problem is that the common name (CN) of the certificate needs to be the same as the apisix-control-plane name;
My certificate is generated according to SVC-Name (openssl req - new - sha256 - key server.key - out server.csr - subj "/CN=apisix-control-plane-control-plane")
Even if my certificate CN information is the same as the apisix-control-plane name, it cannot be authenticated successfully;

Is it because the name is not in the domain name format；（apisix-control-plane-control-plane）


Here is my CN information（https://github.com/apache/apisix-helm-chart/issues/479#issuecomment-1442700906）

subject: CN=apisix-control-plane-control-plane

jinjianming commented 1 year ago

@tao12345666333 @tokers I found the root cause ssl_trusted_certificate set the CA cert that issued the cert of conf_server in the control plane.

apisix:
  ssl:
    ssl_trusted_certificate: t/certs/mtls_ca.crt

Configure trusted_ ca_ Cert didn't get the expected effect

PhanPirang commented 1 year ago

@jinjianming could you provide your full configuration of data plane and control plane?

jinjianming commented 1 year ago

@jinjianming你能提供你的数据平面和控制平面的完整配置吗？

https://github.com/apache/apisix/issues/9038#issuecomment-1461589293

PhanPirang commented 1 year ago

@jinjianming你能提供你的数据平面和控制平面的完整配置吗？

apache/apisix#9038 (comment)

@jinjianming I've seen there're many configurations, which is the correct one?

jinjianming commented 1 year ago

1、CP-Config


    apisix:    # universal configurations
      enable_heartbeat: true
      enable_admin: true
      enable_admin_cors: true
      enable_debug: false

      enable_dev_mode: false                       # Sets nginx worker_processes to 1 if set to true
      enable_reuseport: true                       # Enable nginx SO_REUSEPORT switch if set to true.
      enable_ipv6: true # Enable nginx IPv6 resolver
      enable_server_tokens: true # Whether the APISIX version number should be shown in Server header

      admin_listen.
      #                                   # This port can only receive http request with proxy protocol, but node_listen & admin_listen
      #                                   # can only receive http request. If you enable proxy protocol, you must use this port to
      option
      #   enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server

      proxy_cache:                         # Proxy Caching configuration
        cache_ttl: 10s                     # The default caching time if the upstream does not specify the cache time
        zones:                             # The parameters of a cache
        - name: disk_cache_one             # The name of the cache, administrator can be specify
                                           # which cache to use by name in the admin api
          memory_size: 50m                 # The size of shared memory, it's used to store the cache index
          disk_size: 1G                    # The size of disk, it's used to store the cache data
          disk_path: "/tmp/disk_cache_one" # The path to store the cache data
          cache_levels: "1:2"              # The hierarchy levels of a cache

      router:
        http: radixtree_uri  # radixtree_uri: match route by uri(base on radixtree)
                                    # radixtree_host_uri: match route by host + uri(base on radixtree)
                                    # radixtree_uri_with_parameter: match route by uri with parameters
        ssl: 'radixtree_sni'        # radixtree_sni: match route by SNI(base on radixtree)
      stream_proxy:                 # TCP/UDP proxy
        only: false
        tcp:                        # TCP proxy port list
          - 9100
        udp:                        # UDP proxy port list
          - 9200
      dns_resolver:
        - 10.233.0.3
      dns_resolver_valid: 30
      resolver_timeout: 5
      ssl:
        enable: true
        listen:
          - port: 9443
            enable_http2: true
        ssl_protocols: "TLSv1.2 TLSv1.3"
        ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"

    nginx_config:    # config for render the template to genarate nginx.conf
      error_log: "/dev/stderr"
      error_log_level: "debug"    # warn,error
      worker_processes: "auto"
      enable_cpu_affinity: true
      worker_rlimit_nofile: 20480  # the number of files a worker process can open, should be larger than worker_connections
      event:
        worker_connections: 10620
      http:
        enable_access_log: true
        access_log: "/dev/stdout"
        access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
        access_log_format_escape: default
        keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
        client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
        client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
        send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
        underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
        real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
        real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
          - 127.0.0.1
          - 10.233.0.0/16
          - 'unix:'
    plugins:    # plugin list
      - api-breaker
      - xxx
    stream_plugins:
      - mqtt-proxy
      - ip-restriction
      - limit-conn

    deployment:
      role: control_plane
      role_control_plane:
        config_provider: etcd
        conf_server:
          listen: 0.0.0.0:9280
          cert: "/conf-server-ssl/tls.crt"
          cert_key: "/conf-server-ssl/tls.key"

      admin:
        allow_admin:    # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
          - 127.0.0.1/24
          - 0.0.0.0/0
        #   - "::/64"
        admin_listen:
          ip: 0.0.0.0
          port: 9180
        # Default token when use API to call for Admin API.
        # *NOTE*: Highly recommended to modify this value to protect APISIX's Admin API.
        # Disabling this configuration item means that the Admin API does not
        # require any authentication.
        admin_key:
          # admin: can everything for configuration data
          - name: "admin"
            key: edd1c9f034335f136f87ad84b625c8f1
            role: admin
          # viewer: only can view configuration data
          - name: "viewer"
            key: 4054f7cf07e344346cd3f287985e76a2
            role: viewer
      etcd:
        host:                          # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
          - "http://apisix-control-plane-etcd.ingress-apisix.svc.cluster.local:2379"
        prefix: "/apisix"    # configuration prefix in etcd
        timeout: 30    # 30 seconds
      certs:
        cert: "/conf-client-ssl/tls.crt"
        cert_key: "/conf-client-ssl/tls.key"

2、DP-config


    apisix:    # universal configurations
      ssl:
        ssl_trusted_certificate: "/conf-ca-ssl/tls.crt"
      node_listen: 9080    # APISIX listening port
      enable_heartbeat: true
      enable_admin: false
      enable_admin_cors: true
      enable_debug: false

      enable_dev_mode: false                       # Sets nginx worker_processes to 1 if set to true
      enable_reuseport: true                       # Enable nginx SO_REUSEPORT switch if set to true.
      enable_ipv6: true # Enable nginx IPv6 resolver
      enable_server_tokens: true # Whether the APISIX version number should be shown in Server header

      # proxy_protocol:                   # Proxy Protocol configuration
      #   listen_http_port: 9181          # The port with proxy protocol for http, it differs from node_listen and admin_listen.
      #                                   # This port can only receive http request with proxy protocol, but node_listen & admin_listen
      #                                   # can only receive http request. If you enable proxy protocol, you must use this port to
      #                                   # receive http request with proxy protocol
      #   listen_https_port: 9182         # The port with proxy protocol for https
      #   enable_tcp_pp: true             # Enable the proxy protocol for tcp proxy, it works for stream_proxy.tcp option
      #   enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server

      proxy_cache:                         # Proxy Caching configuration
        cache_ttl: 10s                     # The default caching time if the upstream does not specify the cache time
        zones:                             # The parameters of a cache
        - name: disk_cache_one             # The name of the cache, administrator can be specify
                                           # which cache to use by name in the admin api
          memory_size: 50m                 # The size of shared memory, it's used to store the cache index
          disk_size: 1G                    # The size of disk, it's used to store the cache data
          disk_path: "/tmp/disk_cache_one" # The path to store the cache data
          cache_levels: "1:2"              # The hierarchy levels of a cache

      router:
        http: radixtree_uri  # radixtree_uri: match route by uri(base on radixtree)
                                    # radixtree_host_uri: match route by host + uri(base on radixtree)
                                    # radixtree_uri_with_parameter: match route by uri with parameters
        ssl: 'radixtree_sni'        # radixtree_sni: match route by SNI(base on radixtree)
      dns_resolver:
        - 10.233.0.3
      dns_resolver_valid: 30
      resolver_timeout: 5
      ssl:
        enable: true
        listen:
          - port: 9443
            enable_http2: false
        ssl_protocols: "TLSv1.2 TLSv1.3"
        ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA"

    nginx_config:    # config for render the template to genarate nginx.conf
      error_log: "/dev/stderr"
      error_log_level: "debug"    # warn,error
      worker_processes: "auto"
      enable_cpu_affinity: true
      worker_rlimit_nofile: 20480  # the number of files a worker process can open, should be larger than worker_connections
      event:
        worker_connections: 10620
      http:
        enable_access_log: true
        access_log: "/dev/stdout"
        access_log_format: '$remote_addr - $remote_user [$time_local] $http_host \"$request\" $status $body_bytes_sent $request_time \"$http_referer\" \"$http_user_agent\" $upstream_addr $upstream_status $upstream_response_time \"$upstream_scheme://$upstream_host$upstream_uri\"'
        access_log_format_escape: default
        keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
        client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned to the client
        client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to the client
        send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
        underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
        real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
        real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
          - 127.0.0.1
          - 10.233.0.0/16
          - 'unix:'
    plugins:    # plugin list
      - api-breaker
      - xxx
    stream_plugins:
      - mqtt-proxy
      - ip-restriction
      - limit-conn

    deployment:
      role: data_plane
      role_data_plane:
        config_provider: control_plane
        control_plane:
          host:
            - "https://apisix-control-plane-control-plane:9280"
          prefix: "/apisix"
          timeout: 30
      certs:
        cert: "/conf-client-ssl/tls.crt"
        cert_key: "/conf-client-ssl/tls.key"

apache / apisix-helm-chart

request help: apisix certs used for certificates in decoupled mode #479

Does the Apisix data plane have parameters that can use SSL to allow insecure server connections

This is my configuration list;

However, the data-plane itself cannot be started successfully. My data-plane configuration: