openfrontier / gerrit-docker

Operational scripts for docker-gerrit project.
Apache License 2.0
16 stars 19 forks source link

gerrit container not starting #1

Open s-friedman opened 8 years ago

s-friedman commented 8 years ago

I've verified that gerrit-docker is up to date and has had no modifications from the stock repo. Trying to start the gerrit container yields:

FATA[0001] Error response from daemon: Cannot start container 936f9931e733f30457269055785653bda80b4dd1d9c650d771d117c688e11a2f: Cannot link to a non running container: /gerrit AS /proxy/gerrit

The gerrit docker container shows that it couldn't contact the pg-gerrit on TCP:

steve@klaatu:~$ sudo docker logs gerrit First time initialize gerrit... [2016-02-18 20:57:27,566] [main] INFO com.google.gerrit.server.config.GerritServerConfigProvider : No /var/gerrit/review_site/etc/gerrit.config; assuming defaults Generating SSH host key ... rsa(simple)... done Initialized /var/gerrit/review_site

/gerrit-entrypoint.sh: ignoring /docker-entrypoint-init.d/*

Upgrading gerrit... fatal: DbInjector failed fatal: Unable to determine SqlDialect fatal: caused by org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections. fatal: caused by java.net.ConnectException: Connection refused

pg-gerrit does have an open port on 5432: steve@klaatu:~$ sudo docker exec -it pg-gerrit bash root@3d9c7c520637:/# cat /proc/net/tcp sl local_address rem_address st tx_queue rx_queue tr tm->when retrnsmt uid timeout inode
0: 00000000:1538 00000000:0000 0A 00000000:00000000 00:00000000 00000000 999 0 2067181 1 0000000000000000 100 0 0 10 0

I noticed that pg-gerrit's log shows that it bounces the database during initialization, but this change to start up script has no effect on the ultimate outcome.

steve@klaatu:~/gerrit-docker (master)$ git diff diff --git a/createGerrit.sh b/createGerrit.sh index 879ae9e..574d135 100755 --- a/createGerrit.sh +++ b/createGerrit.sh @@ -23,7 +23,7 @@ docker run \ -e POSTGRES_DB=reviewdb \ -d ${POSTGRES_IMAGE}

-while [ -z "$(docker logs ${PG_GERRIT_NAME} 2>&1 | grep 'autovacuum launcher started')" ]; do +while [ "$(docker logs ${PG_GERRIT_NAME} 2>&1 | grep 'autovacuum launcher started' | wc -l )" -ge 2 ]; do echo "Waiting postgres ready." sleep 1 done

Not sure how to proceed.

thinkernel commented 8 years ago

FATA[0001] Error response from daemon: Cannot start container 936f9931e733f30457269055785653bda80b4dd1d9c650d771d117c688e11a2f: Cannot link to a non running container: /gerrit AS /proxy/gerrit

It looks like you were using openfrontier/ci project to create the whole ci system, because this error shows proxy container cannot find runing gerrit container. As you showed in sudo docker logs gerrit, gerrit container seems couldn't connect to the pg-gerrit container. If your postgres container is running properly, the 5432 port maybe blocked by your firewall. Can you put the output of docker logs pg-gerrit here?

btw. I update the gerrit & postgres image to the latest and chekout the latest source of ci project and gerrit-docker project. I cannot reproduce this issue in my environment.

s-friedman commented 8 years ago

Yes, I did a ~/ci/createContainer.sh. (Note that "~/ci/createContainer.sh; ~/ci/start.sh" succeeds as pg-gerrit is ready.)

As requested, here are the pg-gerrit logs:

steve@klaatu:~$ sudo docker logs pg-gerrit The files belonging to this database system will be owned by user "postgres". This user must also own the server process.

The database cluster will be initialized with locale "en_US.utf8". The default database encoding has accordingly been set to "UTF8". The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /var/lib/postgresql/data ... ok creating subdirectories ... ok selecting default max_connections ... 100 selecting default shared_buffers ... 128MB selecting dynamic shared memory implementation ... posix creating configuration files ... ok creating template1 database in /var/lib/postgresql/data/base/1 ... ok initializing pg_authid ... ok initializing dependencies ... ok creating system views ... ok loading system objects' descriptions ... ok creating collations ... ok creating conversions ... ok creating dictionaries ... ok setting privileges on built-in objects ... ok creating information schema ... ok loading PL/pgSQL server-side language ... ok vacuuming database template1 ... ok copying template1 to template0 ... ok copying template1 to postgres ... ok syncing data to disk ... ok

Success. You can now start the database server using:

pg_ctl -D /var/lib/postgresql/data -l logfile start

WARNING: enabling "trust" authentication for local connections You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb. waiting for server to start....LOG: database system was shut down at 2016-02-19 15:49:06 UTC LOG: MultiXact member wraparound protections are now enabled LOG: autovacuum launcher started LOG: database system is ready to accept connections done server started CREATE DATABASE

CREATE ROLE

/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*

LOG: received fast shutdown request LOG: aborting any active transactions waiting for server to shut down....LOG: autovacuum launcher shutting down LOG: shutting down LOG: database system is shut down done server stopped

PostgreSQL init process complete; ready for start up.

LOG: database system was shut down at 2016-02-19 15:49:34 UTC LOG: MultiXact member wraparound protections are now enabled LOG: database system is ready to accept connections LOG: autovacuum launcher started

thinkernel commented 8 years ago

Did you figure out this problem? The postgres container looks ok. You can start another postgres container as a client to connect this container. You can try command line like this one: docker run -it --link pg-gerrit:postgres --rm postgres sh -c 'exec psql -h "$POSTGRES_PORT_5432_TCP_ADDR" -p "$POSTGRES_PORT_5432_TCP_PORT" -d reviewdb -U gerrit2' The password is gerrit. If this works, I really have no idea why the gerrit couldn't connect to the pg-gerrit.

s-friedman commented 8 years ago

Neither do I. Strangely, running "docker start gerrit" a couple of seconds later works -- it looks like a race condition in determining when pg-gerrit is up.

s-friedman commented 8 years ago

(I meant ~/ci/run.sh)

thinkernel commented 8 years ago

Did you change the values of HOST_NAME & LDAP_SERVER in the ci/config file to your actual host IP or FQDN?

JackTan-TT commented 4 years ago

Dear thinkernel,

I've encoutered the same issue while runing ~/ci/run.sh. It stucks in "Waiting postgres ready.".

bc2aaed03f2011d2716ce4c6024f6ff139e15a5bbddce3b113d2acdb7b75f61f Waiting postgres ready. Waiting postgres ready. Waiting postgres ready. Waiting postgres ready.

And I have change the config file as below, can you please share some suggestions on it?

!/bin/bash

Hostname, IP or FQDN.

HOST_NAME=192.168.199.101

LDAP server configuration

Comment these variables out if you already have a running ldap server.

If SLAPD_DOMAIN and SLAPD_PASSWORD is set,

LDAP_SERVER and LDAP_ACCOUNTBASE will be overwritten automatically.

SLAPD_DOMAIN=example.com SLAPD_PASSWORD=secret

LDAP_SERVER and LDAP_ACCOUNTBASE become unnecessary

if SLAPD_DOMAIN and SLAPD_PASSWORD are specified.

LDAP Server IP or FQDN.

LDAP_SERVER=ldap://192.168.199.101

LDAP account baseDN

LDAP_ACCOUNTBASE=ou=accounts,dc=example,dc=com

Gerrit administrator's uid in LDAP

GERRIT_ADMIN_UID=admin GERRIT_ADMIN_PWD=passwd GERRIT_ADMIN_EMAIL=admin@${SLAPD_DOMAIN}

A Nexus server will be started by default.

Specify this variable only if you want to use

an existing nexus server instead.

Jenkins uses NEXUS_WEBURL to setup maven repository mirror.

NEXUS_WEBURL=http://192.168.199.101/nexus

JackTan-TT commented 4 years ago

After viewing the logs of pg-gerrit container, I found the below errors, my I have some suggestions?

2020-04-18 02:54:00.655 UTC [1] LOG: starting PostgreSQL 12.2 (Debian 12.2-2.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit server stopped

PostgreSQL init process complete; ready for start up.

2020-04-18 02:54:00.667 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432 2020-04-18 02:54:00.667 UTC [1] LOG: listening on IPv6 address "::", port 5432 2020-04-18 02:54:00.990 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432" 2020-04-18 02:54:01.294 UTC [66] LOG: database system was shut down at 2020-04-18 02:54:00 UTC 2020-04-18 02:54:01.428 UTC [1] LOG: database system is ready to accept connections 2020-04-18 02:56:22.709 UTC [75] LOG: invalid length of startup packet 2020-04-18 02:56:22.712 UTC [76] LOG: invalid length of startup packet 2020-04-18 02:56:22.716 UTC [77] LOG: invalid length of startup packet