filipcynarski commented 3 years ago

Pre issue-raising checklist

I have already (please mark the applicable with an x):

[x ] Confirmed this is the right place to raise the issue - only issues related to the Dockerization of the Pact Broker should be raised here. Issues related to the Pact Broker application itself should be raised in the Pact Broker project.
[x ] Upgraded to the latest Pact Broker Docker image OR
[ x] Checked the CHANGELOG to see if the issue I am about to raise has been fixed
[ x] Read the Troubleshooting page

Software versions

pact-broker gem version: eg 2.3.1
pact-broker docker version: 2.81.0.2
OS: Linux 4.4.8-20.46.amzn1.x86_64 #1 SMP Wed Apr 27 19:28:52 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
pact broker client details: eg. pact-ruby-standalone CLI v 1.23.0

Expected behaviour

I should be able to update docker image

Actual behaviour

The docker container is restarting all the time and in the logs I can find following details:

bundler: command not found: puma
Install missing gem executables with `bundle install`
`/pact_broker` is not writable.
Bundler will use `/tmp/bundler20210907-7-6mh4z57' as your home directory temporarily.
bundler: command not found: puma
Install missing gem executables with `bundle install`
`/pact_broker` is not writable.
Bundler will use `/tmp/bundler20210907-7-12gtzxt7' as your home directory temporarily.
bundler: command not found: puma
Install missing gem executables with `bundle install`

Steps to reproduce

The latest working version for me is 2.81.0.2

Relevent log files

bundler: command not found: puma
Install missing gem executables with `bundle install`
`/pact_broker` is not writable.
Bundler will use `/tmp/bundler20210907-7-6mh4z57' as your home directory temporarily.
bundler: command not found: puma
Install missing gem executables with `bundle install`
`/pact_broker` is not writable.
Bundler will use `/tmp/bundler20210907-7-12gtzxt7' as your home directory temporarily.
bundler: command not found: puma
Install missing gem executables with `bundle install`

bethesque commented 3 years ago

Can you provide a docker-compose file that reproduces the issue please? I've just tried running this one, and it's working fine for me, so there must be something different in our setups. It's using pactfoundation/pact-broker:2.83.0.0.

schowave commented 3 years ago

I have the same problem with 2.81.0.2 and 2.83.0.0

postgres_1    |
postgres_1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1    |
postgres_1    | 2021-09-08 09:59:16.623 UTC [1] LOG:  starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1    | 2021-09-08 09:59:16.623 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1    | 2021-09-08 09:59:16.623 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1    | 2021-09-08 09:59:16.646 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2021-09-08 09:59:16.697 UTC [25] LOG:  database system was shut down at 2021-09-08 09:56:28 UTC
postgres_1    | 2021-09-08 09:59:16.720 UTC [1] LOG:  database system is ready to accept connections
broker_app_1  | `/pact_broker` is not writable.
broker_app_1  | Bundler will use `/tmp/bundler20210908-7-o16wo37' as your home directory temporarily.
broker_app_1  | bundler: command not found: puma
broker_app_1  | Install missing gem executables with `bundle install`

DockerCompose file:

version: "3"

services:
  postgres:
    image: postgres
    healthcheck:
      test: psql postgres --command "select 1" -U postgres
    ports:
      - "5433:5432"
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD: password
      POSTGRES_DB: postgres
    volumes:
      - pgdata:/var/lib/postgresql/data

  broker_app:
    image: pactfoundation/pact-broker:2.81.0.2
    depends_on:
      - postgres
    ports:
      - 9292:9292
    environment:
      PACT_BROKER_BASIC_AUTH_USERNAME: pact_user
      PACT_BROKER_BASIC_AUTH_PASSWORD: pact_password
      PACT_BROKER_DATABASE_URL: "postgres://postgres:password@postgres/postgres"
      PACT_BROKER_DISABLE_SSL_VERIFICATION: "true"
      PACT_BROKER_BASE_URL: "http://my-broker:9292/"

volumes:
  pgdata:

With 2.81.0.1 everything works fine.

On my local computer windows this works also with 2.83.0.0, but not on the remote server (SLES)...

bethesque commented 3 years ago

Strange. I have no problems running the docker-compose sample you posted. See the output below. You said "On my local computer windows...". I'm using a mac. Maybe it is a windows specific issue.

The only change I can see that might have caused the problem is this base image upgrade. You could test this by checking out this git repository on you machine, and setting the base image in the Dockefile back to ruby:2.6.7-alpine on you local machine, then running docker-compose -f docker-compose-dev.yml up --build

I also wonder if it's something to do with the permissions you're running the docker image with.

Status: Downloaded newer image for pactfoundation/pact-broker:2.81.0.2
Creating pact-broker-docker_postgres_1 ... done
Creating pact-broker-docker_broker_app_1 ... done
Attaching to pact-broker-docker_postgres_1, pact-broker-docker_broker_app_1
postgres_1    | The files belonging to this database system will be owned by user "postgres".
postgres_1    | This user must also own the server process.
postgres_1    |
postgres_1    | The database cluster will be initialized with locale "en_US.utf8".
postgres_1    | The default database encoding has accordingly been set to "UTF8".
postgres_1    | The default text search configuration will be set to "english".
postgres_1    |
postgres_1    | Data page checksums are disabled.
postgres_1    |
postgres_1    | fixing permissions on existing directory /var/lib/postgresql/data ... ok
postgres_1    | creating subdirectories ... ok
postgres_1    | selecting dynamic shared memory implementation ... posix
postgres_1    | selecting default max_connections ... 100
postgres_1    | selecting default shared_buffers ... 128MB
postgres_1    | selecting default time zone ... Etc/UTC
postgres_1    | creating configuration files ... ok
postgres_1    | running bootstrap script ... ok
postgres_1    | performing post-bootstrap initialization ... ok
postgres_1    | syncing data to disk ... ok
postgres_1    |
postgres_1    | initdb: warning: enabling "trust" authentication for local connections
postgres_1    | You can change this by editing pg_hba.conf or using the option -A, or
postgres_1    | --auth-local and --auth-host, the next time you run initdb.
postgres_1    |
postgres_1    | Success. You can now start the database server using:
postgres_1    |
postgres_1    |     pg_ctl -D /var/lib/postgresql/data -l logfile start
postgres_1    |
postgres_1    | waiting for server to start....2021-09-08 23:33:39.763 UTC [48] LOG:  starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1    | 2021-09-08 23:33:39.766 UTC [48] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2021-09-08 23:33:39.775 UTC [49] LOG:  database system was shut down at 2021-09-08 23:33:39 UTC
postgres_1    | 2021-09-08 23:33:39.781 UTC [48] LOG:  database system is ready to accept connections
postgres_1    |  done
postgres_1    | server started
postgres_1    |
postgres_1    | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
postgres_1    |
postgres_1    | waiting for server to shut down....2021-09-08 23:33:39.923 UTC [48] LOG:  received fast shutdown request
postgres_1    | 2021-09-08 23:33:39.931 UTC [48] LOG:  aborting any active transactions
postgres_1    | 2021-09-08 23:33:39.933 UTC [48] LOG:  background worker "logical replication launcher" (PID 55) exited with exit code 1
postgres_1    | 2021-09-08 23:33:39.934 UTC [50] LOG:  shutting down
postgres_1    | 2021-09-08 23:33:39.953 UTC [48] LOG:  database system is shut down
postgres_1    |  done
postgres_1    | server stopped
postgres_1    |
postgres_1    | PostgreSQL init process complete; ready for start up.
postgres_1    |
postgres_1    | 2021-09-08 23:33:40.048 UTC [1] LOG:  starting PostgreSQL 13.2 (Debian 13.2-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1    | 2021-09-08 23:33:40.048 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1    | 2021-09-08 23:33:40.048 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1    | 2021-09-08 23:33:40.054 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2021-09-08 23:33:40.063 UTC [67] LOG:  database system was shut down at 2021-09-08 23:33:39 UTC
postgres_1    | 2021-09-08 23:33:40.072 UTC [1] LOG:  database system is ready to accept connections
broker_app_1  | Puma starting in single mode...
broker_app_1  | * Puma version: 5.3.2 (ruby 2.6.8-p205) ("Sweetnighter")
broker_app_1  | *  Min threads: 0
broker_app_1  | *  Max threads: 5
broker_app_1  | *  Environment: production
broker_app_1  | *          PID: 8
postgres_1    | 2021-09-08 23:33:42.760 UTC [74] ERROR:  relation "schema_migrations" does not exist at character 27
postgres_1    | 2021-09-08 23:33:42.760 UTC [74] STATEMENT:  SELECT NULL AS "nil" FROM "schema_migrations" LIMIT 1
postgres_1    | 2021-09-08 23:33:42.777 UTC [74] ERROR:  relation "schema_info" does not exist at character 27
postgres_1    | 2021-09-08 23:33:42.777 UTC [74] STATEMENT:  SELECT NULL AS "nil" FROM "schema_info" LIMIT 1
broker_app_1  | * Listening on http://0.0.0.0:9292
broker_app_1  | Use Ctrl-C to stop

bethesque commented 3 years ago

One of my workmates was able to reproduce the issue on his linux machine. I've reverted the base image from ruby:2.6-alpine to ruby:2.6.7-alpine, and released it as 2.83.0.1. That fixed it for my workmate, so hopefully that also fixes it for you.

schowave commented 3 years ago

Thank you @bethesque for the fix! Now it is working for me with 2.83.0.3

bethesque commented 3 years ago

Could you try out pactfoundation/pact-broker:edge for me? I've upgraded from Ruby 2.6.7 to Ruby 2.7 in that image, and I want to know if it will have the same problem. We'll have a big issue if it's everything after 2.6.7 😱

schowave commented 3 years ago

pactfoundation/pact-broker:edge does not work for me. It is the same error as described above:

postgres_1    |
postgres_1    | PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres_1    |
postgres_1    | 2021-09-09 10:45:55.103 UTC [1] LOG:  starting PostgreSQL 12.3 (Debian 12.3-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
postgres_1    | 2021-09-09 10:45:55.112 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
postgres_1    | 2021-09-09 10:45:55.112 UTC [1] LOG:  listening on IPv6 address "::", port 5432
postgres_1    | 2021-09-09 10:45:55.144 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
postgres_1    | 2021-09-09 10:45:55.197 UTC [26] LOG:  database system was shut down at 2021-09-09 10:43:50 UTC
postgres_1    | 2021-09-09 10:45:55.222 UTC [1] LOG:  database system is ready to accept connections
broker_app_1  | `/pact_broker` is not writable.
broker_app_1  | Bundler will use `/tmp/bundler20210909-7-1kdrt5o7' as your home directory temporarily.
broker_app_1  | bundler: command not found: puma
broker_app_1  | Install missing gem executables with `bundle install`

Do you maybe have to install puma in the Dockerfile where you reference 2.7-alpine3.14 image?

bethesque commented 3 years ago

Oh horror 😱 There is something about the more recent alpine base images that is different. They're a good 40MB bigger too, which is a pain.

Do you maybe have to install puma in the Dockerfile where you reference 2.7-alpine3.14 image?

Puma is installed, exactly the same as the other images. It comes in with the rest of the gems when the bundle install is run.

bethesque commented 3 years ago

This seems to be relevant. https://github.com/docker-library/ruby/issues/351

It seems that alpine3.14 requires a newer version of Docker. I'll try using alpine3.13 for as long as possible.

bethesque commented 3 years ago

From scrolling through that linked issue, it looks like it needs at least 20.10 (which is what I happen to be running).

schowave commented 3 years ago

And we have Docker on SLES in Version 19.03.5_ce-6.31.1

bethesque commented 3 years ago

I've been trying to set up a github action to recreate the Docker 19 issue, and I cannot make the build fail! Not sure what I'm doing differently to the set up that causes the issue.

https://github.com/pact-foundation/pact-broker-docker/runs/3785514654?check_suite_focus=true#step:11:1

I've installed 19.03.9, as that's the oldest version I can find in the repository. You'd think it would have the same issues as 19.03.5.

To recreate the situation as closely as possible, I'm building the image using ruby:2.7.4-alpine3.14 as the base on Docker 20, then installing Docker 19, and then running the integration tests. I cannot make it fail though!

bethesque commented 2 years ago

@schowave and @filipcynarski we'll be dropping support for Docker 19.03 soon. See https://github.com/pact-foundation/pact-broker-docker/issues/76

pact-foundation / pact-broker-docker

Upgrade from version 2.81.0.2 (inclusive) till the most recent release is failing #62

Pre issue-raising checklist

Software versions

Expected behaviour

Actual behaviour

Steps to reproduce

Relevent log files