search

Kong / kubernetes-ingress-controller

:gorilla: Kong for Kubernetes: The official Ingress Controller for Kubernetes.
https://docs.konghq.com/kubernetes-ingress-controller/
Apache License 2.0
2.2k stars 590 forks source link

Kong Docker postgres migration job cannot login into Azure Postgres service #919

Closed david-freistrom closed 3 years ago

david-freistrom commented 3 years ago

Summary

Kong could not migrate Postgres Database placed in Azure as a service because of a invalid authentication format The Problem here seems like the mandatory format of the username which includes a @ symbol which is maybe not parsed well from the scripts.

Kong Ingress controller version

2.1

Kong or Kong Enterprise version

Kubernetes version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.8", GitCommit:"73ec19bdfc6008cd3ce6de96c663f70a69e2b8fc", GitTreeState:"clean", BuildDate:"2020-09-17T04:17:08Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}

Environment

What happened

2020-10-24T04:05:16.080913975Z Error: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: FATAL: Invalid Username specified. Please check the Username and retry connection. The Username should be in username@hostname format. 2020-10-24T04:05:16.080959375Z 2020-10-24T04:05:16.080963275Z Run with --v (verbose) or --vv (debug) for more details

Expected behavior

Migrating the Database - Login access

Steps To Reproduce

  1. Create Azure Postgresql Database
  2. Run Kong Helm chart against it
  3. Check out the migration job which produce the error
env:
  database: postgres
  pg_username: "kong%40kong-db" # I tried to mask the '@' 
  pg_database: kong-db
  # pg_host: SET BY ANSIBLE AT RUNTIME
  pg_password: !vault |
    $ANSIBLE_VAULT;1.1;AES256
    35643832376630643032616566313165643039643238366639646234363836643533333031363531
    6466386566383537613930376438666337333037646435330a616339313233393064353831613838
    38383533386134356438353362663262623264643431653461376133396162623664303661636433
    3161303636623461330a323662333038353237633734623033393761313966306635386434333935
    66303264356330346338303730386263333566303963306535613134353231643138363036383036
    65666231643637373663373864363531386431343332663766313165623237363034623938336535
    373030343661343064376339373139303635
  nginx_worker_processes: "1"
  proxy_access_log: /dev/stdout
  admin_access_log: /dev/stdout
  admin_gui_access_log: /dev/stdout
  portal_api_access_log: /dev/stdout
  proxy_error_log: /dev/stderr
  admin_error_log: /dev/stderr
  admin_gui_error_log: /dev/stderr
  portal_api_error_log: /dev/stderr
  prefix: /kong_prefix/
david-freistrom commented 3 years ago

I found out that the Problem lies in 'kong start'

bash-5.0$ env
KONG_PG_DATABASE=kong-db
KONG_PG_USERNAME=kong@euris-dbms.postgres.database.azure.com
HOSTNAME=4561803c9d1c
KONG_NGINX_DAEMON=off
ASSET=ce
PWD=/
KONG_VERSION=2.1.4
KONG_PG_PASSWORD=*******
HOME=/home/kong
TERM=xterm
SHLVL=1
KONG_PG_HOST=euris-dbms.postgres.database.azure.com
KONG_DATABASE=postgres
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
_=/usr/bin/env
bash-5.0$ kong start --vv
2020/10/26 12:56:58 [verbose] Kong: 2.1.4
2020/10/26 12:56:58 [debug] ngx_lua: 10015
2020/10/26 12:56:58 [debug] nginx: 1015008
2020/10/26 12:56:58 [debug] Lua: LuaJIT 2.1.0-beta3
2020/10/26 12:56:58 [verbose] no config file found at /etc/kong/kong.conf
2020/10/26 12:56:58 [verbose] no config file found at /etc/kong.conf
2020/10/26 12:56:58 [verbose] no config file, skip loading
2020/10/26 12:56:58 [debug] reading environment variables
2020/10/26 12:56:58 [debug] KONG_PG_DATABASE ENV found with "kong-db"
2020/10/26 12:56:58 [debug] KONG_PG_HOST ENV found with "euris-dbms.postgres.database.azure.com"
2020/10/26 12:56:58 [debug] KONG_DATABASE ENV found with "postgres"
2020/10/26 12:56:58 [debug] KONG_PG_PASSWORD ENV found with "******"
2020/10/26 12:56:58 [debug] KONG_NGINX_DAEMON ENV found with "off"
2020/10/26 12:56:58 [debug] admin_access_log = "logs/admin_access.log"
2020/10/26 12:56:58 [debug] admin_error_log = "logs/error.log"
2020/10/26 12:56:58 [debug] admin_listen = {"127.0.0.1:8001 reuseport backlog=16384","127.0.0.1:8444 http2 ssl reuseport backlog=16384"}
2020/10/26 12:56:58 [debug] anonymous_reports = true
2020/10/26 12:56:58 [debug] cassandra_contact_points = {"127.0.0.1"}
2020/10/26 12:56:58 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2020/10/26 12:56:58 [debug] cassandra_keyspace = "kong"
2020/10/26 12:56:58 [debug] cassandra_lb_policy = "RequestRoundRobin"
2020/10/26 12:56:58 [debug] cassandra_port = 9042
2020/10/26 12:56:58 [debug] cassandra_read_consistency = "ONE"
2020/10/26 12:56:58 [debug] cassandra_refresh_frequency = 60
2020/10/26 12:56:58 [debug] cassandra_repl_factor = 1
2020/10/26 12:56:58 [debug] cassandra_repl_strategy = "SimpleStrategy"
2020/10/26 12:56:58 [debug] cassandra_schema_consensus_timeout = 10000
2020/10/26 12:56:58 [debug] cassandra_ssl = false
2020/10/26 12:56:58 [debug] cassandra_ssl_verify = false
2020/10/26 12:56:58 [debug] cassandra_timeout = 5000
2020/10/26 12:56:58 [debug] cassandra_username = "kong"
2020/10/26 12:56:58 [debug] cassandra_write_consistency = "ONE"
2020/10/26 12:56:58 [debug] client_body_buffer_size = "8k"
2020/10/26 12:56:58 [debug] client_max_body_size = "0"
2020/10/26 12:56:58 [debug] client_ssl = false
2020/10/26 12:56:58 [debug] cluster_control_plane = "127.0.0.1:8005"
2020/10/26 12:56:58 [debug] cluster_listen = {"0.0.0.0:8005"}
2020/10/26 12:56:58 [debug] cluster_mtls = "shared"
2020/10/26 12:56:58 [debug] database = "postgres"
2020/10/26 12:56:58 [debug] db_cache_ttl = 0
2020/10/26 12:56:58 [debug] db_cache_warmup_entities = {"services","plugins"}
2020/10/26 12:56:58 [debug] db_resurrect_ttl = 30
2020/10/26 12:56:58 [debug] db_update_frequency = 5
2020/10/26 12:56:58 [debug] db_update_propagation = 0
2020/10/26 12:56:58 [debug] dns_error_ttl = 1
2020/10/26 12:56:58 [debug] dns_hostsfile = "/etc/hosts"
2020/10/26 12:56:58 [debug] dns_no_sync = false
2020/10/26 12:56:58 [debug] dns_not_found_ttl = 30
2020/10/26 12:56:58 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2020/10/26 12:56:58 [debug] dns_resolver = {}
2020/10/26 12:56:58 [debug] dns_stale_ttl = 4
2020/10/26 12:56:58 [debug] error_default_type = "text/plain"
2020/10/26 12:56:58 [debug] go_plugins_dir = "off"
2020/10/26 12:56:58 [debug] go_pluginserver_exe = "/usr/local/bin/go-pluginserver"
2020/10/26 12:56:58 [debug] headers = {"server_tokens","latency_tokens"}
2020/10/26 12:56:58 [debug] host_ports = {}
2020/10/26 12:56:58 [debug] kic = false
2020/10/26 12:56:58 [debug] log_level = "notice"
2020/10/26 12:56:58 [debug] lua_package_cpath = ""
2020/10/26 12:56:58 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2020/10/26 12:56:58 [debug] lua_socket_pool_size = 30
2020/10/26 12:56:58 [debug] lua_ssl_verify_depth = 1
2020/10/26 12:56:58 [debug] mem_cache_size = "128m"
2020/10/26 12:56:58 [debug] nginx_admin_directives = {}
2020/10/26 12:56:58 [debug] nginx_daemon = "off"
2020/10/26 12:56:58 [debug] nginx_events_directives = {{name="worker_connections",value="auto"},{name="multi_accept",value="on"}}
2020/10/26 12:56:58 [debug] nginx_events_multi_accept = "on"
2020/10/26 12:56:58 [debug] nginx_events_worker_connections = "auto"
2020/10/26 12:56:58 [debug] nginx_http_client_body_buffer_size = "8k"
2020/10/26 12:56:58 [debug] nginx_http_client_max_body_size = "0"
2020/10/26 12:56:58 [debug] nginx_http_directives = {{name="client_max_body_size",value="0"},{name="ssl_prefer_server_ciphers",value="off"},{name="client_body_buffer_size",value="8k"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_tickets",value="on"},{name="ssl_session_timeout",value="1d"}}
2020/10/26 12:56:58 [debug] nginx_http_ssl_prefer_server_ciphers = "off"
2020/10/26 12:56:58 [debug] nginx_http_ssl_protocols = "TLSv1.2 TLSv1.3"
2020/10/26 12:56:58 [debug] nginx_http_ssl_session_tickets = "on"
2020/10/26 12:56:58 [debug] nginx_http_ssl_session_timeout = "1d"
2020/10/26 12:56:58 [debug] nginx_http_status_directives = {}
2020/10/26 12:56:58 [debug] nginx_http_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
2020/10/26 12:56:58 [debug] nginx_http_upstream_keepalive = "60"
2020/10/26 12:56:58 [debug] nginx_http_upstream_keepalive_requests = "100"
2020/10/26 12:56:58 [debug] nginx_http_upstream_keepalive_timeout = "60s"
2020/10/26 12:56:58 [debug] nginx_main_daemon = "off"
2020/10/26 12:56:58 [debug] nginx_main_directives = {{name="daemon",value="off"},{name="worker_processes",value="auto"},{name="worker_rlimit_nofile",value="auto"}}
2020/10/26 12:56:58 [debug] nginx_main_worker_processes = "auto"
2020/10/26 12:56:58 [debug] nginx_main_worker_rlimit_nofile = "auto"
2020/10/26 12:56:58 [debug] nginx_optimizations = true
2020/10/26 12:56:58 [debug] nginx_proxy_directives = {{name="real_ip_header",value="X-Real-IP"},{name="real_ip_recursive",value="off"}}
2020/10/26 12:56:58 [debug] nginx_proxy_real_ip_header = "X-Real-IP"
2020/10/26 12:56:58 [debug] nginx_proxy_real_ip_recursive = "off"
2020/10/26 12:56:58 [debug] nginx_sproxy_directives = {}
2020/10/26 12:56:58 [debug] nginx_status_directives = {}
2020/10/26 12:56:58 [debug] nginx_stream_directives = {{name="ssl_session_timeout",value="1d"},{name="ssl_prefer_server_ciphers",value="off"},{name="ssl_protocols",value="TLSv1.2 TLSv1.3"},{name="ssl_session_tickets",value="on"}}
2020/10/26 12:56:58 [debug] nginx_stream_ssl_prefer_server_ciphers = "off"
2020/10/26 12:56:58 [debug] nginx_stream_ssl_protocols = "TLSv1.2 TLSv1.3"
2020/10/26 12:56:58 [debug] nginx_stream_ssl_session_tickets = "on"
2020/10/26 12:56:58 [debug] nginx_stream_ssl_session_timeout = "1d"
2020/10/26 12:56:58 [debug] nginx_supstream_directives = {}
2020/10/26 12:56:58 [debug] nginx_upstream_directives = {{name="keepalive_requests",value="100"},{name="keepalive_timeout",value="60s"},{name="keepalive",value="60"}}
2020/10/26 12:56:58 [debug] nginx_upstream_keepalive = "60"
2020/10/26 12:56:58 [debug] nginx_upstream_keepalive_requests = "100"
2020/10/26 12:56:58 [debug] nginx_upstream_keepalive_timeout = "60s"
2020/10/26 12:56:58 [debug] nginx_worker_processes = "auto"
2020/10/26 12:56:58 [debug] pg_database = "kong-db"
2020/10/26 12:56:58 [debug] pg_host = "euris-dbms.postgres.database.azure.com"
2020/10/26 12:56:58 [debug] pg_max_concurrent_queries = 0
2020/10/26 12:56:58 [debug] pg_password = "******"
2020/10/26 12:56:58 [debug] pg_port = 5432
2020/10/26 12:56:58 [debug] pg_ro_ssl = false
2020/10/26 12:56:58 [debug] pg_ro_ssl_verify = false
2020/10/26 12:56:58 [debug] pg_semaphore_timeout = 60000
2020/10/26 12:56:58 [debug] pg_ssl = false
2020/10/26 12:56:58 [debug] pg_ssl_verify = false
2020/10/26 12:56:58 [debug] pg_timeout = 5000
2020/10/26 12:56:58 [debug] pg_user = "kong"
2020/10/26 12:56:58 [debug] plugins = {"bundled"}
2020/10/26 12:56:58 [debug] port_maps = {}
2020/10/26 12:56:58 [debug] prefix = "/usr/local/kong/"
2020/10/26 12:56:58 [debug] proxy_access_log = "logs/access.log"
2020/10/26 12:56:58 [debug] proxy_error_log = "logs/error.log"
2020/10/26 12:56:58 [debug] proxy_listen = {"0.0.0.0:8000 reuseport backlog=16384","0.0.0.0:8443 http2 ssl reuseport backlog=16384"}
2020/10/26 12:56:58 [debug] real_ip_header = "X-Real-IP"
2020/10/26 12:56:58 [debug] real_ip_recursive = "off"
2020/10/26 12:56:58 [debug] role = "traditional"
2020/10/26 12:56:58 [debug] ssl_cipher_suite = "intermediate"
2020/10/26 12:56:58 [debug] ssl_ciphers = "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384"
2020/10/26 12:56:58 [debug] ssl_prefer_server_ciphers = "on"
2020/10/26 12:56:58 [debug] ssl_protocols = "TLSv1.1 TLSv1.2 TLSv1.3"
2020/10/26 12:56:58 [debug] ssl_session_tickets = "on"
2020/10/26 12:56:58 [debug] ssl_session_timeout = "1d"
2020/10/26 12:56:58 [debug] status_access_log = "off"
2020/10/26 12:56:58 [debug] status_error_log = "logs/status_error.log"
2020/10/26 12:56:58 [debug] status_listen = {"off"}
2020/10/26 12:56:58 [debug] stream_listen = {"off"}
2020/10/26 12:56:58 [debug] trusted_ips = {}
2020/10/26 12:56:58 [debug] upstream_keepalive = 60
2020/10/26 12:56:58 [debug] upstream_keepalive_idle_timeout = 60
2020/10/26 12:56:58 [debug] upstream_keepalive_max_requests = 100
2020/10/26 12:56:58 [debug] upstream_keepalive_pool_size = 60
2020/10/26 12:56:58 [debug] worker_consistency = "strict"
2020/10/26 12:56:58 [debug] worker_state_update_frequency = 5
2020/10/26 12:56:58 [verbose] prefix in use: /usr/local/kong
Error: 
/usr/local/share/lua/5.1/kong/cmd/start.lua:31: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: FATAL: Invalid Username specified. Please check the Username and retry connection. The Username should be in <username@hostname> format.
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong/cmd/start.lua:31: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:88>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:88: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:45>
    /usr/local/bin/kong:9: in function 'file_gen'
    init_worker_by_lua:47: in function <init_worker_by_lua:45>
    [C]: in function 'xpcall'
    init_worker_by_lua:54: in function <init_worker_by_lua:52>
rainest commented 3 years ago

Does the instance connect successfully if you set pg_user in .Values.env, rather than pg_username? That's the kong.conf setting name.

In the startup log, you can see that it's reporting the default user:

2020/10/26 12:56:58 [debug] pg_user = "kong"

I think it's likely just that: even though the escaped @ doesn't appear to show up correctly in env output, I wasn't able to find other issues similar to this that indicated they were using pg_user, and know other users have successfully used Azure Postgres with Kong in the past, so that's hopefully just a rendering quirk.