search

pact-foundation / pact-broker-docker

Dockerized Pact Broker
https://pact.io
MIT License
107 stars 109 forks source link

Error database connection since version 2.87.0.1 #126

Closed RyanCDS closed 1 year ago

RyanCDS commented 1 year ago

Pre issue-raising checklist

I have already (please mark the applicable with an x):

Software versions

Expected behaviour

The pact broker should start correctly

Actual behaviour

I got an error during kubernetes deployment.

For information, i did some test with different version and the changeset who broke the deployment appears with version 2.87.0.1

The logs below come from version 2.107.0.1 (it's similar issue with version 2.87.0.1).

I [7:3120] pact-broker -- Connecting to database: -- {:adapter=>"postgres", :user=>"pact_broker", :password=>"*****", :host=>"ad06pg-pactbroker.cnx.cdbdx.biz", :database=>"pact_broker", :encoding=>"utf8", :sslmode=>"require", :sql_log_level=>:debug, :enable_caller_logging=>false, :log_warn_duration=>5.0, :pool_timeout=>5, :driver_options=>{:options=>"-c statement_timeout=15s"}, :connect_max_retries=>5}
I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 5 tries to go.
I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 4 tries to go.
I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 3 tries to go.
I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 2 tries to go.
I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 1 tries to go.
! Unable to load application: Sequel::DatabaseConnectionError: PG::ConnectionBad: connection to server at "10.10.241.33", port 5432 failed: ERROR:  Unsupported startup parameter: options

Steps to reproduce

Deploy image docker v 2.107.0.1 with config below (for information, i use postgre in version 11) :

apiVersion: apps/v1
kind: Deployment
metadata:
  name: pact-broker
  namespace: pact-broker
  labels:
    app: pact-broker
spec:
  replicas: 1
  selector:
    matchLabels:
      app: pact-broker
  template:
    metadata:
      labels:
        app: pact-broker
    spec:
      containers:
      - name: pact-broker
        image: 2.107.0.1
        resources:
          limits:
              cpu: "1"
              memory: 2048
        livenessProbe:
          httpGet:
            path: /hal-browser/browser.html
        readinessProbe:
          httpGet:
            path: /hal-browser/browser.html
        volumeMounts:
          - name: configmap
            mountPath: /run/dotnet/config
          - name: secrets
            mountPath: /run/dotnet/secrets
        envs: >-
            "PLATFORM": "$(platform)",
            "PERIMETER":"$(namespace)",
            "PACT_BROKER_PORT": "9292",
            "PACT_BROKER_DATABASE_URL":"postgres://user:password@host/name",
            "PACT_BROKER_DATABASE_CONNECT_MAX_RETRIES": "5",
            "PACT_BROKER_DATABASE_SSLMODE": "require",
            "PACT_BROKER_PUBLIC_HEARTBEAT": "true",
            "PACT_BROKER_LOG_LEVEL": "DEBUG",
            "PACT_BROKER_SQL_LOG_LEVEL": "DEBUG"
  service:
    ports: '[9292]'
  secretFilename: 'secret.yml'

Relevent log files

Here the log file from Kubernetes during deployment

Puma starting in single mode...
* Puma version: 5.6.5 (ruby 2.7.7-p221) ("Birdie's Version")
*  Min threads: 0
*  Max threads: 5
*  Environment: production
*          PID: 7
2023-04-13 09:37:47.111467 I [7:3120] pact-broker -- Connecting to database: -- {:adapter=>"postgres", :user=>"user", :password=>"*****", :host=>"host", :database=>"name", :encoding=>"utf8", :sslmode=>"require", :sql_log_level=>:debug, :enable_caller_logging=>false, :log_warn_duration=>5.0, :pool_timeout=>5, :driver_options=>{:options=>"-c statement_timeout=15s"}, :connect_max_retries=>5}
2023-04-13 09:37:47.218007 I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 5 tries to go.
2023-04-13 09:37:50.266645 I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 4 tries to go.
2023-04-13 09:37:53.316706 I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 3 tries to go.
2023-04-13 09:37:56.364629 I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 2 tries to go.
2023-04-13 09:37:59.412661 I [7:3120] pact-broker -- Error connecting to database (Sequel::DatabaseConnectionError). Waiting 3 seconds and trying again. 1 tries to go.
! Unable to load application: Sequel::DatabaseConnectionError: PG::ConnectionBad: connection to server at "20.20.20.42", port 5432 failed: ERROR:  Unsupported startup parameter: options
bundler: failed to load command: puma (/pact_broker/vendor/bundle/ruby/2.7.0/bin/puma)
/pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:633:in `async_connect_or_reset': PG::ConnectionBad: connection to server at "20.20.20.42", port 5432 failed: ERROR:  Unsupported startup parameter: options (Sequel::DatabaseConnectionError)
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:761:in `connect_to_hosts'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:696:in `new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/adapters/postgres.rb:231:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:248:in `new_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool.rb:154:in `make_new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:209:in `assign_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:139:in `acquire'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:293:in `synchronize'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:302:in `test_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/misc.rb:188:in `initialize'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:57:in `new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:57:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/core.rb:124:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:17:in `block in create_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:31:in `with_retries'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:16:in `create_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:135:in `configure_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:99:in `post_configure'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:50:in `initialize'
    from config.ru:4:in `new'
    from config.ru:4:in `block in <main>'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:116:in `eval'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:116:in `new_from_string'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:105:in `load_file'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:66:in `parse_file'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:348:in `load_rackup'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:270:in `app'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/runner.rb:150:in `load_and_bind'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/single.rb:44:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
    from /pact_broker/vendor/bundle/ruby/2.7.0/bin/puma:23:in `load'
    from /pact_broker/vendor/bundle/ruby/2.7.0/bin/puma:23:in `<top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `load'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in `run'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in `exec'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in `start'
    from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in `block in <top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in `<top (required)>'
    from /usr/local/bundle/bin/bundle:23:in `load'
    from /usr/local/bundle/bin/bundle:23:in `<main>'
/pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:633:in `async_connect_or_reset': connection to server at "20.20.20.42", port 5432 failed: ERROR:  Unsupported startup parameter: options (PG::ConnectionBad)
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:761:in `connect_to_hosts'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pg-1.4.6/lib/pg/connection.rb:696:in `new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/adapters/postgres.rb:231:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:248:in `new_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool.rb:154:in `make_new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:209:in `assign_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:139:in `acquire'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/connection_pool/threaded.rb:91:in `hold'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:293:in `synchronize'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:302:in `test_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/misc.rb:188:in `initialize'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:57:in `new'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/database/connecting.rb:57:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/sequel-5.66.0/lib/sequel/core.rb:124:in `connect'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:17:in `block in create_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:31:in `with_retries'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/initializers/database_connection.rb:16:in `create_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:135:in `configure_database_connection'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:99:in `post_configure'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/pact_broker-2.106.0/lib/pact_broker/app.rb:50:in `initialize'
    from config.ru:4:in `new'
    from config.ru:4:in `block in <main>'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:116:in `eval'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:116:in `new_from_string'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:105:in `load_file'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/rack-2.2.6.4/lib/rack/builder.rb:66:in `parse_file'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:348:in `load_rackup'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/configuration.rb:270:in `app'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/runner.rb:150:in `load_and_bind'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/single.rb:44:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/launcher.rb:193:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/lib/puma/cli.rb:81:in `run'
    from /pact_broker/vendor/bundle/ruby/2.7.0/gems/puma-5.6.5/bin/puma:10:in `<top (required)>'
    from /pact_broker/vendor/bundle/ruby/2.7.0/bin/puma:23:in `load'
    from /pact_broker/vendor/bundle/ruby/2.7.0/bin/puma:23:in `<top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `load'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:58:in `kernel_load'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli/exec.rb:23:in `run'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:492:in `exec'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/invocation.rb:127:in `invoke_command'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor.rb:392:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:34:in `dispatch'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/vendor/thor/lib/thor/base.rb:485:in `start'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/cli.rb:28:in `start'
    from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:45:in `block in <top (required)>'
    from /usr/local/bundle/gems/bundler-2.4.7/lib/bundler/friendly_errors.rb:117:in `with_friendly_errors'
    from /usr/local/bundle/gems/bundler-2.4.7/exe/bundle:33:in `<top (required)>'
    from /usr/local/bundle/bin/bundle:23:in `load'
    from /usr/local/bundle/bin/bundle:23:in `<main>'
TimothyJones commented 1 year ago

How is your database configured? Is it fronted by PgBouncer?

RyanCDS commented 1 year ago

Yes, is it fronted by PgBouncer. Which configuration information may be relevant for helping ?

TimothyJones commented 1 year ago

This error isn’t coming from the pact broker, it’s from pgbouncer. You can correct it by telling pgbouncer to ignore the options parameter:

https://github.com/Athou/commafeed/issues/559#issuecomment-169551299

RyanCDS commented 1 year ago

Ok, so the options "-c statement_timeout=15s" will be manage directly by postgre and we will not override this with pact-broker docker ? I thought that i had an issue on the format from "driver_options" who needs the options directly for pgbouncer and not with the format "{:options=>"-c statement_timeout=15s"}".

I ask to my database admin to try your solution and let you know if it works.

TimothyJones commented 1 year ago

The error isn’t coming from the pact broker (or the way it behaves), it’s because pgbouncer doesn’t support the startup parameter options, so yes, your db admin will need to check that everything is in order

mefellows commented 1 year ago

Thanks Tim, great spot.

The logs below come from version 2.107.0.1 (it's similar issue with version 2.87.0.1).

This seems to imply that it previous did work. Is that right and if so, which version previously worked?

RyanCDS commented 1 year ago

Thanks Tim, great spot.

The logs below come from version 2.107.0.1 (it's similar issue with version 2.87.0.1).

This seems to imply that it previous did work. Is that right and if so, which version previously worked?

This is right, the version 2.87.0.0 work without errors.

TimothyJones commented 1 year ago

Have you tried it recently, or was that when you introduced pgbouncer?

RyanCDS commented 1 year ago

Nothing change on my configuration between each test, only the version change.

RyanCDS commented 1 year ago

Hello, for the record i successfully deploy the pact broker docker after update pgbouncer options startup. It's still weird that i have to change that since version 2.87.0.1.

I mark the issue as completed.