How to connect etcd cluster with tls

[huangyutongs]

Environmental description: rke version: v1.2.1 kubernetes: v1.18.3 I use the kubernetes and etcd cluster (3 nodes) built by rke。They are https://10.1.7.51:2379, https://10.1.7.52:2379, https://10.1.7.53:2379

Use: curl -k --cert /etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem --key /etc/kubernetes/ssl/kube-etcd-10-1-7-51 -key.pem https://10.1.7.51:2379/version Output: {"etcdserver":"3.4.3","etcdcluster":"3.4.0"}

start up：docker-compose up -d

Output:

got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd

This is my complete config.yaml,

apisix:
  node_listen: 80              # APISIX listening port
  enable_heartbeat: true
  enable_admin: true
  enable_admin_cors: true         # Admin API support CORS response headers.
  enable_debug: false
  enable_dev_mode: false          # Sets nginx worker_processes to 1 if set to true
  enable_reuseport: true          # Enable nginx SO_REUSEPORT switch if set to true.
  enable_ipv6: true
  config_center: etcd             # etcd: use etcd to store the config value
                                  # yaml: fetch the config value from local yaml file `/your_path/conf/apisix.yaml`

  #proxy_protocol:                 # Proxy Protocol configuration
  #  listen_http_port: 9181        # The port with proxy protocol for http, it differs from node_listen and port_admin.
                                   # This port can only receive http request with proxy protocol, but node_listen & port_admin
                                   # can only receive http request. If you enable proxy protocol, you must use this port to
                                   # receive http request with proxy protocol
  #  listen_https_port: 9182       # The port with proxy protocol for https
  #  enable_tcp_pp: true           # Enable the proxy protocol for tcp proxy, it works for stream_proxy.tcp option
  #  enable_tcp_pp_to_upstream: true # Enables the proxy protocol to the upstream server

  proxy_cache:                     # Proxy Caching configuration
    cache_ttl: 10s                 # The default caching time if the upstream does not specify the cache time
    zones:                         # The parameters of a cache
    - name: disk_cache_one         # The name of the cache, administrator can be specify
                                   # which cache to use by name in the admin api
      memory_size: 50m             # The size of shared memory, it's used to store the cache index
      disk_size: 1G                # The size of disk, it's used to store the cache data
      disk_path: "/tmp/disk_cache_one" # The path to store the cache data
      cache_levels: "1:2"           # The hierarchy levels of a cache
  #  - name: disk_cache_two
  #    memory_size: 50m
  #    disk_size: 1G
  #    disk_path: "/tmp/disk_cache_two"
  #    cache_levels: "1:2"

#  allow_admin:                  # http://nginx.org/en/docs/http/ngx_http_access_module.html#allow
#    - 127.0.0.0/24              # If we don't set any IP list, then any IP access is allowed by default.
#    - 172.17.0.0/24
  #   - "::/64"
  # port_admin: 9180              # use a separate port

  # Default token when use API to call for Admin API.
  # *NOTE*: Highly recommended to modify this value to protect APISIX's Admin API.
  # Disabling this configuration item means that the Admin API does not
  # require any authentication.
  admin_key:
    -
      name: "admin"
      key: edd1c9f034335f136f87ad84b625c8f1
      role: admin                 # admin: manage all configuration data
                                  # viewer: only can view configuration data
    -
      name: "viewer"
      key: 4054f7cf07e344346cd3f287985e76a2
      role: viewer
  router:
    http: 'radixtree_uri'         # radixtree_uri: match route by uri(base on radixtree)
                                  # radixtree_host_uri: match route by host + uri(base on radixtree)
    ssl: 'radixtree_sni'          # radixtree_sni: match route by SNI(base on radixtree)
  # stream_proxy:                 # TCP/UDP proxy
  #   tcp:                        # TCP proxy port list
  #     - 9100
  #     - 9101
  #   udp:                        # UDP proxy port list
  #     - 9200
  #     - 9211
  dns_resolver:                   # default DNS resolver, with disable IPv6 and enable local DNS
    - 10.43.0.10
    - 223.5.5.5
    - 1.1.1.1
    - 8.8.8.8
  dns_resolver_valid: 30          # valid time for dns result 30 seconds
  resolver_timeout: 5             # resolver timeout
  ssl:
    enable: true
    enable_http2: true
    listen_port: 443
    ssl_protocols: "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3"
    ssl_ciphers: "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM--SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES12ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES256-SHA256:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SH

nginx_config:                     # config for render the template to genarate nginx.conf
  error_log: "logs/error.log"
  error_log_level: "warn"         # warn,error
  worker_rlimit_nofile: 20480     # the number of files a worker process can open, should be larger than worker_connections
  event:
    worker_connections: 10620
  http:
    access_log: "logs/access.log"
    keepalive_timeout: 60s         # timeout during which a keep-alive client connection will stay open on the server side.
    client_header_timeout: 60s     # timeout for reading client request header, then 408 (Request Time-out) error is returned t
    client_body_timeout: 60s       # timeout for reading client request body, then 408 (Request Time-out) error is returned to 
    send_timeout: 10s              # timeout for transmitting a response to the client.then the connection is closed
    underscores_in_headers: "on"   # default enables the use of underscores in client request header fields
    real_ip_header: "X-Real-IP"    # http://nginx.org/en/docs/http/ngx_http_realip_module.html#real_ip_header
    real_ip_from:                  # http://nginx.org/en/docs/http/ngx_http_realip_module.html#set_real_ip_from
      - 127.0.0.1
      - 'unix:'
    #lua_shared_dicts:              # add custom shared cache to nginx.conf
    #  ipc_shared_dict: 100m        # custom shared cache, format: `cache-key: cache-size`

etcd:
  host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
#    - "http://172.18.5.10:2379"     # multiple etcd address
    - "https://10.1.7.51:2379"     # multiple etcd address
  prefix: "/apisix"               # apisix configurations prefix
  timeout: 3                      # 3 seconds

plugins:                          # plugin list
  - example-plugin
  - limit-req
  - limit-count
  - limit-conn
  - key-auth
  - basic-auth
  - prometheus
  - node-status
  - jwt-auth
  - zipkin
  - ip-restriction
  - grpc-transcode
  - serverless-pre-function
  - serverless-post-function
  - openid-connect
  - proxy-rewrite
  - redirect
  - response-rewrite
  - fault-injection
  - udp-logger
  - wolf-rbac
  - proxy-cache
  - tcp-logger
  - proxy-mirror
  - kafka-logger
  - cors
stream_plugins:
  - mqtt-proxy

[huangyutongs]

I used this example: https://github.com/apache/apisix-docker/tree/master/example

[huangyutongs]

I set etcd host as:

etcd:
  host:                           # it's possible to define multiple etcd hosts addresses of the same etcd cluster.
    - "https://10.1.7.51:2379,https://10.1.7.52:2379,https://10.1.7.53:2379"

appear：

/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
    /usr/bin/apisix:619: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:660: in function 'read_yaml_conf'
    /usr/bin/apisix:826: in function </usr/bin/apisix:819>
    /usr/bin/apisix:1166: in main chunk
    [C]: at 0x00404c00
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
    /usr/bin/apisix:619: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:660: in function 'read_yaml_conf'
    /usr/bin/apisix:826: in function </usr/bin/apisix:819>
    /usr/bin/apisix:1166: in main chunk
    [C]: at 0x00404c00
/usr/local/openresty/luajit/bin/luajit: /usr/bin/apisix:619: attempt to index local 'base' (a nil value)
stack traceback:
    /usr/bin/apisix:619: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:616: in function 'merge_conf'
    /usr/bin/apisix:660: in function 'read_yaml_conf'
    /usr/bin/apisix:826: in function </usr/bin/apisix:819>
    /usr/bin/apisix:1166: in main chunk
    [C]: at 0x00404c00

[membphis]

Here is the latest conf/config.yaml file from APISIX project:

https://github.com/apache/apisix/blob/master/conf/config.yaml

Needs to update the current version

[huangyutongs]

@membphis Now this is my configuration file， It still appears

got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd

cat apisix_conf/config.yaml

apisix:
  admin_key:
    -
      name: "admin"
      key: edd1c9f034335f136f87ad84b625c8f1 
      role: admin
etcd:
  host:
    - "https://10.1.7.51:2379"

[tokers]

This is a bug, we don't respect the scheme when calling bin/apisix init_etcd.

[gxthrj]

@hyt05 I have two questions:

1. https://github.com/apache/apisix-docker/tree/master/example use docker-compose with network as below, make sure the etcd address can be reached.

   networks:
   apisix:
   driver: bridge
   ipam:
     config:
     - subnet: 172.18.0.0/16

2. multi-etcd IPs can not be configured in a line , need to configure like this:

   etcd:
   host:
   - "https://10.1.7.51:2379"
   - "https://10.1.7.52:2379"
   - "https://10.1.7.53:2379"

[huangyutongs]

@tokers Do you have a good solution

[tokers]

This is a bug, we don't respect the scheme when calling bin/apisix init_etcd.

Sorry, i misunderstood this issue, just ignore it.

[huangyutongs]

@gxthrj In fact, I didn't use docker- compose.yml Etcd in, I just use the example docker- compose.yml To start apifix, the etcd connection address used in my configuration file is the etcd cluster deployed with RKE,

Just tested and used the configuration

etcd:
  host:
    - "https://10.1.7.51:2379"
    - "https://10.1.7.52:2379"
    - "https://10.1.7.53:2379"

Still appear：

got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd
got malformed version message: "" from etcd

[gxthrj]

The error messsage got malformed version message: "" from etcd means APISIX can not connect to etcd. In most cases, the network cannot be connected between APISIX and etcd.

We need to check if etcd can be reached in APISIX container.

A feasible method is to use docker exec to enter a container which on the same network as the APISIX container, and then telnet 10.1.7.51 2379.

[huangyutongs]

@gxthrj I am sure that I can communicate with 10.1.7.51 2379 in the same container

This is the startup parameter of my etcd cluster. I wonder if it is related to the "--enable-v2=true" parameter.

root     11701  2.4  3.1 10888476 322624 ?     Ssl  11月22 321:14 /usr/local/bin/etcd --listen-client-urls=https://0.0.0.0:2379 \
--initial-advertise-peer-urls=https://10.1.7.51:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem \ 
--peer-key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem --peer-client-cert-auth=true \ 
--initial-cluster-token=etcd-cluster-1 --name=etcd-master1 --cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem \
--advertise-client-urls=https://10.1.7.51:2379,https://10.1.7.51:4001 --enable-v2=true --data-dir=/var/lib/rancher/etcd/ \ 
--listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem \ 
--peer-cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem --initial-cluster=etcd-master1=https://10.1.7.51:2380,etcd-worker1=https://10.1.7.52:2380,etcd-worker2=https://10.1.7.53:2380 \ 
--trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --election-timeout=5000 --heartbeat-interval=500

[membphis]

@hyt05 what are the apisix and etcd versions?

[huangyutongs]

@membphis apisix version：2.1 、{"etcdserver":"3.4.3","etcdcluster":"3.4.0"}

[tokers]

@gxthrj I am sure that I can communicate with 10.1.7.51 2379 in the same container

This is the startup parameter of my etcd cluster. I wonder if it is related to the "--enable-v2=true" parameter.

root     11701  2.4  3.1 10888476 322624 ?     Ssl  11月22 321:14 /usr/local/bin/etcd --listen-client-urls=https://0.0.0.0:2379 \
--initial-advertise-peer-urls=https://10.1.7.51:2380 --key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem \ 
--peer-key-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51-key.pem --peer-client-cert-auth=true \ 
--initial-cluster-token=etcd-cluster-1 --name=etcd-master1 --cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem \
--advertise-client-urls=https://10.1.7.51:2379,https://10.1.7.51:4001 --enable-v2=true --data-dir=/var/lib/rancher/etcd/ \ 
--listen-peer-urls=https://0.0.0.0:2380 --initial-cluster-state=new --peer-trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem \ 
--peer-cert-file=/etc/kubernetes/ssl/kube-etcd-10-1-7-51.pem --initial-cluster=etcd-master1=https://10.1.7.51:2380,etcd-worker1=https://10.1.7.52:2380,etcd-worker2=https://10.1.7.53:2380 \ 
--trusted-ca-file=/etc/kubernetes/ssl/kube-ca.pem --client-cert-auth=true --election-timeout=5000 --heartbeat-interval=500

The option --client-cert-auth=true is used in your etcd server, which forces the clients offer their own certificate to meet the mTLS, which is not supported by APISIX.

If the mLTS is not required necessarily, just remove it.

[huangyutongs]

@tokers Thanks for your answer, I will try to remove this parameter

[membphis]

@hyt05 https://github.com/apache/apisix/pull/2940/files here is the related PR

feel free to reopen this if you still have an issue.