RabbitMQ not start

[polcape]

Hi,

unms not work anymore. I think it's not start and when check compose log I see many unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-08-19T21:40:56+02:00","severity":"INFO","level":200}

How can I check RabbitMQ logs or service?

Thanks

[Nico640]

Sorry for the late response, if this is still a problem, can you try to restart the container and post the contents of the compose log after the first couple of Waiting for RabbitMQ messages appear? The log probably contains RabbitMQs attempts to start (should be at the beginning of the log, before the Waiting for RabbitMQ messages appear).

You can also try to rename the /config/rabbitmq directory to something else like rabbitmq_old, restart the container and then see if RabbitMQ starts.

[polcape]

Hi @Nico640 ,

issue already solved, I had to wait a couple of hours and all was started.

Thanks for response.

[polcape]

Hi @Nico640 , I reopen the issue because the start of the container take too long (about 30-40 min) also if I start from scratch.

In the log I have this

unms | 2023/11/13 18:13:07 [error] 829#829: 1399 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.122, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms." unms | 2023/11/13 18:13:08 [error] 829#829: 1401 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.121, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms." unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:09+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:12+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:15+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:18+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:21+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:25+01:00","severity":"INFO","level":200} unms | 2023/11/13 18:13:26 [error] 829#829: 1403 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.122, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms." unms | 2023/11/13 18:13:27 [error] 829#829: 1405 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.121, server: , request: "GET // HTTP/1.1", upstream: "http://127.0.0.1:8082//", host: "unms." unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:28+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:31+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:34+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:37+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-13T18:13:40+01:00","severity":"INFO","level":200}

[Nico640]

Hmm, weird, RabbitMQ is definitely not supposed to take that long to start. Can you upload / post a full log after you start seeing the "Waiting for RabbitMQ" message looping in the log?

Directly after starting the container, RabbitMQ will try to start and log some messages that could tell us why it isn't starting, so it should be pretty early in the log.

[polcape]

Hi @Nico640 , here the full log

unms | s6-rc: info: service s6rc-oneshot-runner: starting unms | s6-rc: info: service s6rc-oneshot-runner successfully started unms | s6-rc: info: service fix-attrs: starting unms | s6-rc: info: service fix-attrs successfully started unms | s6-rc: info: service legacy-cont-init: starting unms | cont-init: info: running /etc/cont-init.d/10-set-timezone unms | cont-init: info: /etc/cont-init.d/10-set-timezone exited 0 unms | s6-rc: info: service legacy-cont-init successfully started unms | s6-rc: info: service init-adduser: starting unms | unms | ------------------------------------- unms | GID/UID unms | ------------------------------------- unms | User uid: 911 unms | User gid: 911 unms | ------------------------------------- unms | unms | s6-rc: info: service init-adduser successfully started unms | s6-rc: info: service init-prepare: starting unms | s6-rc: info: service init-prepare successfully started unms | s6-rc: info: service init-permissions: starting unms | s6-rc: info: service init-permissions successfully started unms | s6-rc: info: service init-postgres: starting unms | s6-rc: info: service siridb: starting unms | s6-rc: info: service rabbitmq: starting unms | s6-rc: info: service nginx: starting unms | s6-rc: info: service siridb successfully started unms | s6-rc: info: service rabbitmq successfully started unms | s6-rc: info: service nginx successfully started unms | Starting siridb-server... unms | Starting nginx... unms | Starting rabbitmq-server... unms | Database already configured unms | s6-rc: info: service init-postgres successfully started unms | Running entrypoint.sh unms | s6-rc: info: service postgres: starting unms | s6-rc: info: service postgres successfully started unms | s6-rc: info: service unms: starting unms | s6-rc: info: service ucrm: starting unms | s6-rc: info: service netflow: starting unms | Starting postgres... unms | [W 2023-11-14 18:43:46] Update server 'c40d4f67c853:9010' to '72b03ab87af5:9010' unms | Creating nginx configuration unms | Waiting for rabbitmq to start... unms | s6-rc: info: service unms successfully started unms | s6-rc: info: service ucrm successfully started unms | s6-rc: info: service netflow successfully started unms | s6-rc: info: service legacy-services: starting unms | /run/postgresql:5432 - no response unms | Waiting for postgres to come up... unms | /run/postgresql:5432 - no response unms | Waiting for postgres to come up... unms | s6-rc: info: service legacy-services successfully started unms | Enabling UNMS https connections on port 443 unms | Enabling UNMS wss connections on port 8444 unms | 2023-11-14 19:43:46.774 CET [191] LOG: starting PostgreSQL 13.12 on x86_64-alpine-linux-musl, compiled by gcc (Alpine 12.2.1_git20220924-r4) 12.2.1 20220924, 64-bit unms | 2023-11-14 19:43:46.774 CET [191] LOG: listening on IPv4 address "127.0.0.1", port 5432 unms | 2023-11-14 19:43:46.775 CET [191] LOG: could not bind IPv6 address "::1": Address not available unms | 2023-11-14 19:43:46.775 CET [191] HINT: Is another postmaster already running on port 5432? If not, wait a few seconds and retry. unms | 2023-11-14 19:43:46.777 CET [191] LOG: listening on Unix socket "/run/postgresql/.s.PGSQL.5432" unms | 2023-11-14 19:43:46.780 CET [191] LOG: listening on Unix socket "/tmp/.s.PGSQL.5432" unms | Will use existing SSL certificate unms | 2023-11-14 19:43:46.785 CET [349] LOG: database system was shut down at 2023-11-14 17:17:30 CET unms | Entrypoint finished unms | Calling exec unms | 2023-11-14 19:43:46.799 CET [191] LOG: database system is ready to accept connections unms | 2023-11-14 19:43:49.861 CET [408] FATAL: role "root" does not exist unms | 2023-11-14 19:43:49.866 CET [409] FATAL: role "root" does not exist unms | /run/postgresql:5432 - accepting connections unms | /run/postgresql:5432 - accepting connections unms | /usr/src/ucrm/scripts/init_log.sh unms | /usr/src/ucrm/scripts/dirs.sh unms | {"message":"Creating directories.","channel":"dirs.sh","datetime":"2023-11-14T19:43:49+01:00","severity":"INFO","level":200} unms | {"message":"Creating /var/log/nginx.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200} unms | {"message":"Creating /run/nginx.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200} unms | {"message":"Creating /etc/nginx/enabled-servers.","channel":"dirs.sh","datetime":"2023-11-14T19:43:50+01:00","severity":"INFO","level":200} unms | Running docker-entrypoint /home/app/unms/index.js unms | Sentry release: unms | Version: 2.3.57+a484f0ef73.2023-10-05T18:06:06+02:00 unms | Waiting for database containers unms | 2023/11/14 19:43:51 [error] 398#398: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.180.238, server: , request: "GET /nms/ HTTP/1.1", upstream: "http://127.0.0.1:8081/nms/", host: "unms.xxx.xx" unms | {"message":"Creating /tmp/plugins.","channel":"dirs.sh","datetime":"2023-11-14T19:43:51+01:00","severity":"INFO","level":200} unms | {"message":"Done creating directories.","channel":"dirs.sh","datetime":"2023-11-14T19:44:00+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic links.","channel":"dirs.sh","datetime":"2023-11-14T19:44:01+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/data.","channel":"dirs.sh","datetime":"2023-11-14T19:44:01+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/EmailQueue.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic link for /usr/src/ucrm/app/logs.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic link for /usr/src/ucrm/web/uploads.","channel":"dirs.sh","datetime":"2023-11-14T19:44:02+01:00","severity":"INFO","level":200} unms | {"message":"Creating symbolic links for /data/updates.","channel":"dirs.sh","datetime":"2023-11-14T19:44:03+01:00","severity":"INFO","level":200} unms | {"message":"Publishing current /usr/src/ucrm/app/config/version.yml.","channel":"dirs.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200} unms | {"message":"Done creating symbolic links.","channel":"dirs.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/parameters.sh unms | {"message":"Replacing configuration parameters.","channel":"parameters.sh","datetime":"2023-11-14T19:44:04+01:00","severity":"INFO","level":200} unms | {"message":"Done replacing configuration parameters.","channel":"parameters.sh","datetime":"2023-11-14T19:44:26+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/cron_jobs_disable.sh unms | crontab /tmp/crontabs/server unms | su-exec unms /usr/src/ucrm/scripts/database_ready.sh unms | {"message":"Waiting for database.","channel":"database_ready.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/database_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied unms | {"message":"Database ready.","channel":"database_ready.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200} unms | su-exec unms /usr/src/ucrm/scripts/database_create_extensions.sh unms | {"message":"Creating database extensions.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200} unms | {"message":"Extension \"citext\" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:27+01:00","severity":"INFO","level":200} unms | {"message":"Extension \"unaccent\" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200} unms | {"message":"Extension \"uuid-ossp\" exists.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200} unms | {"message":"Finished creating database extensions.","channel":"database_create_extensions.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/backup_control_do.sh unms | su-exec unms /usr/src/ucrm/scripts/verify_database_migrations_integrity.sh unms | {"message":"Verifying database migrations integrity.","channel":"verify_database_migrations_integrity.sh","datetime":"2023-11-14T19:44:28+01:00","severity":"INFO","level":200} unms | {"message":"Database migrations integrity verified.","channel":"verify_database_migrations_integrity.sh","datetime":"2023-11-14T19:44:30+01:00","severity":"INFO","level":200} unms | su-exec unms /usr/src/ucrm/scripts/migrate.sh unms | {"message":"Start replace shared views.","channel":"migrate.sh","datetime":"2023-11-14T19:44:30+01:00","severity":"INFO","level":200} unms | {"message":"Done replace shared views.","channel":"migrate.sh","datetime":"2023-11-14T19:44:31+01:00","severity":"INFO","level":200} unms | {"message":"Start database migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:31+01:00","severity":"INFO","level":200} unms | {"message":"Done database migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200} unms | {"message":"Start shared views migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200} unms | {"message":"Done shared views migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200} unms | {"message":"Start notification templates migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:34+01:00","severity":"INFO","level":200} unms | {"message":"Done notification templates migration.","channel":"migrate.sh","datetime":"2023-11-14T19:44:35+01:00","severity":"INFO","level":200} unms | {"message":"Bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2023-11-14T19:44:35+01:00","severity":"INFO","level":200} unms | {"message":"Done bumping UNMS version from ENV variable to database.","channel":"migrate.sh","datetime":"2023-11-14T19:44:36+01:00","severity":"INFO","level":200} unms | su-exec unms /usr/src/ucrm/scripts/database_migrations_ready.sh unms | {"message":"Waiting for migrations.","channel":"database_migrations_ready.sh","datetime":"2023-11-14T19:44:36+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/database_migrations_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied unms | {"message":"Migrations ready.","channel":"database_migrations_ready.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200} unms | /usr/src/ucrm/scripts/web.sh unms | {"message":"Preparing server configuration.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200} unms | cp: cannot stat '/etc/nginx/available-servers/ucrm.conf': No such file or directory unms | {"message":"Done preparing server configuration.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200} unms | {"message":"Updating environment variables.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200} unms | {"message":"Done updating environment variables.","channel":"web.sh","datetime":"2023-11-14T19:44:37+01:00","severity":"INFO","level":200} unms | {"message":"Updating UCRM version.","channel":"web.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200} unms | {"message":"Done UCRM version.","channel":"web.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:38+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:42+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:45+01:00","severity":"INFO","level":200} unms | {"message":"Waiting for RabbitMQ.","channel":"rabbitmq_ready.sh","datetime":"2023-11-14T19:44:48+01:00","severity":"INFO","level":200}

[Nico640]

RabbitMQs doesn't seem to start at all, can you try to start it manually by opening a shell in the container and running rabbitmq-server? Also, check if there are any RabbitMQ logs in /var/log/rabbitmq/

[polcape]

I running the command but logs are the same, No log in /var/log/rabbitmq/

[Nico640]

The command should output something in the console, not in the logs. Does anything happen after manually running rabbitmq-server? Does it hang or immediately return to the shell?

By the way, on what system are you running the container? Which OS / distro and which Docker version? Is there anything else noteworthy about your setup (e.g. is it running on a virtual machine (which virtualization platform?) or bare metal?). I haven't been able to reproduce this yet.

[polcape]

it doesn't appear nothing on shell, still blank. Not quit the process.

I am on RockyLinux 9 last updates with docker in VM

[stephan-schwarz]

Subscribing to this topic.

I've had this same issue months ago and never managed to get it to work due to the endless loop of "Waiting for RabbitMQ" messages.

Today I figured I'd give it another attempt, but turns out it still doesn't work for me.

When I try to start rabbitmq-server from the shell it "hangs", there's no output or what so ever. The var/log/rabbitmq directory is empty as well.

I'm also running this in a VM, different distro though. I'm running on Fedora 38.

[aacebedo]

I have the same issue. I tried to start from scratch. I am running it in a Promox VM with NixOS as a guest system.

[Nico640]

Should be fixed in the latest image.

[danny-waite]

I'm seeing this same issue, tested on M1 Mac and also Ubuntu 23 with AMD64. tried latest and 2.3.57.1

Just keeps looping on: Setting up the Rabbit MQ fabric

and I see the following errors: FATAL: role "root" does not exist

/usr/src/ucrm/scripts/database_migrations_ready.sh: line 7: /tmp/UCRM_init.log: Permission denied

cp: cannot stat '/etc/nginx/available-servers/ucrm.conf': No such file or directory