search

Kong / kong

🦍 The Cloud-Native API Gateway and AI Gateway.
https://konghq.com/install/#kong-community
Apache License 2.0
38.83k stars 4.77k forks source link

bootstrap failure inside the container: migrations.lua:212: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused #13485

Closed lgoswami closed 2 hours ago

lgoswami commented 1 month ago

Is there an existing issue for this?

Kong version ($ kong version)

Kong Enterprise 3.7.1.2

Current Behavior

this issue I am seeing using local docker setup kong migrations bootstrap --vv

Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
    (command line -e):7: in function 'inline_gen'
    init_worker_by_lua(nginx.conf:122):44: in function <init_worker_by_lua(nginx.conf:122):43>
    [C]: in function 'xpcall'
    init_worker_by_lua(nginx.conf:122):52: in function <init_worker_by_lua(nginx.conf:122):50>
$

Expected Behavior

this should success the migration

Steps To Reproduce

just follow the main docker install doc https://docs.konghq.com/gateway/latest/install/docker/

Anything else?

nothing this is the issue

bungle commented 1 month ago

Probably some kind of networking issue (perhaps because of Docker or OS changes, firewall?). Seems like Kong cannot connect to PG.

lgoswami commented 1 month ago

I am able to connect the pg but when run kong migrations bootstrap --vv it fails 

$ psql -h kong-database -U kong -d kong
Password for user kong:
psql (14.12 (Ubuntu 14.12-0ubuntu0.22.04.1), server 13.15 (Debian 13.15-1.pgdg120+1))
Type "help" for help.

kong=# \q
could not save history to file "/home/kong/.psql_history": No such file or directory
$ kong migrations bootstrap --vv
2024/08/09 12:59:05 [verbose] Kong: 3.7.1.2-enterprise-edition
2024/08/09 12:59:05 [debug] ngx_lua: 10026
2024/08/09 12:59:05 [debug] nginx: 1025003
bungle commented 1 month ago

@lgoswami,

So it looks like you can connect from host machine, but you cannot connect from kong container.

lgoswami commented 1 month ago

this is from kong container @bungle

╰─❯ docker run -it --rm --network=kong-net custom-kong-gateway  /bin/sh
$ vim /usr/local/share/lua/5.1/kong/cmd/migrations.lua
$ kong migrations bootstrap --vv
2024/08/09 12:57:32 [verbose] Kong: 3.7.1.2-enterprise-edition
2024/08/09 12:57:32 [debug] ngx_lua: 10026
2024/08/09 12:57:32 [debug] nginx: 1025003

2024/08/09 12:57:33 [debug] Connecting to PostgreSQL with host: 127.0.0.1, port: 5432, user: kong
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
    [C]: in function 'assert'
    /usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: in function 'cmd_exec'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
    [C]: in function 'xpcall'
    /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
    (command line -e):7: in function 'inline_gen'
    init_worker_by_lua(nginx.conf:122):44: in function <init_worker_by_lua(nginx.conf:122):43>
    [C]: in function 'xpcall'
    init_worker_by_lua(nginx.conf:122):52: in function <init_worker_by_lua(nginx.conf:122):50>
$ tc
/bin/sh: 3: tc: not found
$ nc
/bin/sh: 4: nc: not found
$ telnet
/bin/sh: 5: telnet: not found
$
$
$ host
/bin/sh: 8: host: not found
$
$
$ psql -h kong-database -U kong -d kong
Password for user kong:
psql (14.12 (Ubuntu 14.12-0ubuntu0.22.04.1), server 13.15 (Debian 13.15-1.pgdg120+1))
Type "help" for help.

kong=# \q
could not save history to file "/home/kong/.psql_history": No such file or directory
$ kong migrations bootstrap --vv
2024/08/09 12:59:05 [verbose] Kong: 3.7.1.2-enterprise-edition
2024/08/09 12:59:05 [debug] ngx_lua: 10026

it is failing on migration command 

kong migrations bootstrap --vv

this is the step-3 from kong document https://docs.konghq.com/gateway/latest/install/docker/

I followed the steps and it is failing then I tried logging the the container to verify I am able to connect DB it seem I can connect to DB it't just mailing on kong migrations bootstrap --vv

bungle commented 1 month ago

Interesting. It looks like you had to enter password. Is that also configured when running kong migrations bootstrap -vv? E.g. kong.conf or environment variables? I assume it is. Then we might need to check if there is some problem with Postgres version or has it changed some defaults (e.g. password hashing, TLS or something).

lgoswami commented 1 month ago

if you follow the steps as mentioned here it will fail for you also if not then this might be something to do with my mac m1 or something I am missing https://docs.konghq.com/gateway/latest/install/docker/

when I run this I also get the same error 

docker run --rm --network=kong-net \
-e "KONG_DATABASE=postgres" \
-e "KONG_PG_HOST=kong-database" \
-e "KONG_PG_PASSWORD=kongpass" \
-e "KONG_PASSWORD=test" \
kong/kong-gateway:3.7.1.2 kong migrations bootstrap
chobits commented 3 weeks ago

Interesting. I followed steps from this installation page: https://docs.konghq.com/gateway/latest/install/docker/#prepare-the-database, but not finding some error. It worked for me on my Apple M1 Max (MacOS 14.2.1 (23C71)).

The following commands are what I attempted.

 $ cat db.sh

docker run -d --name kong-database \
 --network=kong-net \
 -p 5432:5432 \
 -e "POSTGRES_USER=kong" \
 -e "POSTGRES_DB=kong" \
 -e "POSTGRES_PASSWORD=kongpass" \
 postgres:13

$ sh db.sh
cf04db87ea92146d14f85056eb1b8dc67a6e7a582740912584c6ae98f2749ae8

 $ cat boot.sh
docker run --rm --network=kong-net \
  -e "KONG_DATABASE=postgres" \
  -e "KONG_PG_HOST=kong-database" \
  -e "KONG_PG_PASSWORD=kongpass" \
  -e "KONG_PASSWORD=test" \
  kong/kong-gateway:3.7.1.2 kong migrations bootstrap

$ sh boot.sh
Unable to find image 'kong/kong-gateway:3.7.1.2' locally
3.7.1.2: Pulling from kong/kong-gateway
4ce000a43472: Already exists
...
....
migrating enterprise.response-transformer-advanced on database 'kong'...
enterprise.response-transformer-advanced migrated up to: 001_1500_to_2100 (executed)
127 migrations processed
127 executed
Database is up-to-date

⚠️ But when I went into the docker container and ran bootstrap again, it failed as following:

$ docker run -it --rm --network=kong-net kong/kong-gateway:3.7.1.2  /bin/sh
$ kong migrations bootstrap --vv
2024/08/19 03:58:20 [verbose] Kong: 3.7.1.2-enterprise-edition
2024/08/19 03:58:20 [debug] ngx_lua: 10026
2024/08/19 03:58:20 [debug] nginx: 1025003
...
sl/admin-gui-kong-default.key) for listener
2024/08/19 03:58:20 [verbose] generating admin_gui SSL certificate (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.crt) and key (/usr/local/kong/ssl/admin-gui-kong-default-ecdsa.key) for listener
2024/08/19 03:58:20 [verbose] generating trusted certs combined file in /usr/local/kong/.ca_combined
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused
stack traceback:
        [C]: in function 'assert'
        /usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: in function 'cmd_exec'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:38>
        [C]: in function 'xpcall'
        /usr/local/share/lua/5.1/kong/cmd/init.lua:38: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:22>
        (command line -e):7: in function 'inline_gen'
        init_worker_by_lua(nginx.conf:122):44: in function <init_worker_by_lua(nginx.conf:122):43>
        [C]: in function 'xpcall'
        init_worker_by_lua(nginx.conf:122):52: in function <init_worker_by_lua(nginx.conf:122):50>

THen if I ran the boot.sh outside of the container, it showed it worked.

 $ sh boot.sh
Database already bootstrapped
chobits commented 3 weeks ago

psql -h kong-database -U kong -d kong

this is from kong container @bungle


╰─❯ docker run -it --rm --network=kong-net custom-kong-gateway  /bin/sh
$ vim /usr/local/share/lua/5.1/kong/cmd/migrations.lua
$ kong migrations bootstrap --vv
2024/08/09 12:57:32 [verbose] Kong: 3.7.1.2-enterprise-edition
2024/08/09 12:57:32 [debug] ngx_lua: 10026
2024/08/09 12:57:32 [debug] nginx: 1025003

2024/08/09 12:57:33 [debug] Connecting to PostgreSQL with host: 127.0.0.1, port: 5432, user: kong
Error:

-> 127.0.0.1

$ tc /bin/sh: 3: tc: not found $ nc /bin/sh: 4: nc: not found $ telnet /bin/sh: 5: telnet: not found $ $ $ host /bin/sh: 8: host: not found $ $ $ psql -h kong-database -U kong -d kong

-> kong-database

Password for user kong: psql (14.12 (Ubuntu 14.12-0ubuntu0.22.04.1), server 13.15 (Debian 13.15-1.pgdg120+1)) Type "help" for help.

kong=# \q could not save history to file "/home/kong/.psql_history": No such file or directory $ kong migrations bootstrap --vv 2024/08/09 12:59:05 [verbose] Kong: 3.7.1.2-enterprise-edition 2024/08/09 12:59:05 [debug] ngx_lua: 10026


it is failing on migration command

kong migrations bootstrap --vv



this is the step-3 from kong document https://docs.konghq.com/gateway/latest/install/docker/

I followed the steps and it is failing then I tried logging the the container to verify I am able to connect DB it seem I can connect to DB it't just mailing on `kong migrations bootstrap --vv`

I find that your psql command tried to connect the domain kong-database, and your kong command tried the Host 127.0.0.1.

I'm guessing whether these two command used different destination IP addresses. could you try ur psql command using 127.0.0.1 address?

lgoswami commented 3 weeks ago

it works Mine is Apple M1 Pro 

╰─❯ psql -h 127.0.0.1 -U kong -d kong
Password for user kong:
psql (16.2, server 13.15 (Debian 13.15-1.pgdg120+1))
Type "help" for help.

kong=#
chobits commented 3 weeks ago

it works Mine is Apple M1 Pro

╰─❯ psql -h 127.0.0.1 -U kong -d kong
Password for user kong:
psql (16.2, server 13.15 (Debian 13.15-1.pgdg120+1))
Type "help" for help.

kong=#

So cool!

lgoswami commented 3 weeks ago

but I am still facing the issue , not sure why I am using https://github.com/abiosoft/colima as docker runtime this might be the issue

chobits commented 3 weeks ago

I'm guessing that running kong command directly in docker container might not setup right ENV variables or IP destination.

and I also have a idea to add some failed ip addresses into the message /usr/local/share/lua/5.1/kong/cmd/migrations.lua:212: [PostgreSQL error] failed to retrieve PostgreSQL server_version_num: connection refused to help locate the problem

chobits commented 3 weeks ago

I know why. It's indeed because running the command directly set up the wrong PostgreSQL server IP address.

See the different debug info between running bootstrap command between outside and inside of the container.

@@ -190,10 +187,9 @@
 [debug] opentelemetry_tracing = {"off"}
 [debug] opentelemetry_tracing_sampling_rate = 0.01
 [debug] pg_database = "kong"
-[debug] pg_host = "kong-database"
+[debug] pg_host = "127.0.0.1"
 [debug] pg_iam_auth = false
 [debug] pg_max_concurrent_queries = 0
-[debug] pg_password = "******"
 [debug] pg_port = 5432
 [debug] pg_ro_iam_auth = false
 [debug] pg_ro_ssl = false
chobits commented 3 weeks ago

So the resolution is that you need to set the right KONG ENV variables before running bootstrap command, you could bootstrap kong with appropriate postgres env vars inside the container, like this:

(outside the container) $ docker run -it --rm --network=kong-net kong/kong-gateway:3.7.1.2  /bin/sh
(inside the container) $ KONG_DATABASE=postgres KONG_PG_HOST=kong-database KONG_PG_PASSWORD=kongpass KONG_PASSWORD=test kong migrations bootstrap
Database already bootstrapped

Or you can set the pg ENV vars before you enter the container inside, like this


$ docker run -it --rm \
  -e "KONG_DATABASE=postgres" \
  -e "KONG_PG_HOST=kong-database" \
  -e "KONG_PG_PASSWORD=kongpass" \
  -e "KONG_PASSWORD=test" \
--network=kong-net kong/kong-gateway:3.7.1.2  /bin/sh

(inside the container) $ kong migrations bootstrap
Database already bootstrapped
github-actions[bot] commented 1 week ago

This issue is marked as stale because it has been open for 14 days with no activity.

github-actions[bot] commented 2 hours ago

Dear contributor,

We are automatically closing this issue because it has not seen any activity for three weeks. We're sorry that your issue could not be resolved. If any new information comes up that could help resolving it, please feel free to reopen it.

Your contribution is greatly appreciated!

Please have a look our pledge to the community for more information.

Sincerely, Your Kong Gateway team