Unable to start rabbitmq

Do you want to request a feature or report a bug? A bug

What is the current behavior? I am trying to install onlyoffice using docker. I tested it on my laptop, that worked perfectly.

When I installed it on my server

sudo docker run -i -t -d -p 8080:80 onlyoffice/documentserver

I was not getting any response on the port I assigned.

wget localhost:8080/welcome

I checked htop, and saw "erl_child_setup" was still running. Also my 4gb memory was full. I waited until there was no process running on 100%, that took 20 minutes. But still no response from the localhost at 8080.

sudo docker run -i -t -d -p 8080:80 -v /app/onlyoffice/DocumentServer/logs:/var/log/onlyoffice -v /app/onlyoffice/DocumentServer/data:/var/www/onlyoffice/Data -v /app/onlyoffice/DocumentServer/lib:/var/lib/onlyoffice -v /app/onlyoffice/DocumentServer/rabbitmq:/var/lib/rabbitmq -v /app/onlyoffice/DocumentServer/redis:/var/lib/redis -v /app/onlyoffice/DocumentServer/db:/var/lib/postgresql onlyoffice/documentserver

I checked the logs folder:

cat logs/documentserver/docservice/out.log

Where I saw this error: error: password authentication failed for user "onlyoffice"

When I ran:

docker run -it -m 2000M --memory-swap -1 onlyoffice/documentserver

I got the output:

 fhfs@  ~  docker stop c404fb9a1f90
c404fb9a1f90
 fhfs@  ~  docker run -it -m 2000M --memory-swap -1 onlyoffice/documentserver
sed: -e expression #1, char 25: unknown option to `s'
 * Starting PostgreSQL 12 database server              [ OK ] 
 * Starting RabbitMQ Messaging Server rabbitmq-server          * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
                                                       [fail]
Waiting for connection to the localhost host on port 5672
Waiting.... **(removed duplicates)**
Waiting for connection to the localhost host on port 5672
psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:4: NOTICE:  table "doc_callbacks" does not exist, skipping
psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:9: NOTICE:  function merge_db(pg_catalog.varchar,int2,int8,pg_catalog.timestamp,pg_catalog.varchar,int8,int8) does not exist, skipping
psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:10: NOTICE:  function merge_db(pg_catalog.varchar,int2,int4,pg_catalog.timestamp,pg_catalog.varchar,int4,int4) does not exist, skipping
Starting supervisor: supervisord.
 * Starting periodic command scheduler cron            [ OK ] 
 * Starting nginx nginx                                [ OK ] 
Generating AllFonts.js, please wait...Done
Generating presentation themes, please wait...Done
ds:docservice: stopped
ds:docservice: started
ds:converter: stopped
ds:converter: started
 * Reloading nginx configuration nginx                 [ OK ] 
==> /var/log/onlyoffice/documentserver/converter/err.log <==

==> /var/log/onlyoffice/documentserver/converter/out.log <==
[2021-10-05T11:14:24.041] [WARN] nodeJS - update cluster with 1 workers
[2021-10-05T11:14:24.059] [WARN] nodeJS - worker 1108 started.
[2021-10-05T11:14:24.063] [WARN] nodeJS - update cluster with 1 workers
[2021-10-05T11:15:29.281] [WARN] nodeJS - update cluster with 1 workers
[2021-10-05T11:15:29.624] [WARN] nodeJS - worker 1204 started.
[2021-10-05T11:15:29.923] [WARN] nodeJS - update cluster with 1 workers

==> /var/log/onlyoffice/documentserver/docservice/err.log <==

==> /var/log/onlyoffice/documentserver/docservice/out.log <==
[2021-10-05T11:14:25.553] [WARN] nodeJS - Express server starting...
[2021-10-05T11:14:25.558] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
[2021-10-05T11:14:25.664] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode. Version: 6.4.1. Build: 45
[2021-10-05T11:15:32.754] [WARN] nodeJS - Express server starting...
[2021-10-05T11:15:32.757] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
[2021-10-05T11:15:32.809] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode. Version: 6.4.1. Build: 45

==> /var/log/onlyoffice/documentserver/metrics/err.log <==

==> /var/log/onlyoffice/documentserver/metrics/out.log <==
5 Oct 11:14:24 - [1054] reading config file: ./config/config.js
5 Oct 11:14:24 - server is up INFO

==> /var/log/onlyoffice/documentserver/nginx.error.log <==

I also checked the startup log file mentioned in the output.

root@c5c45057cb4c:/var/log/rabbitmq# cat startup_log 

  ##  ##      RabbitMQ 3.8.2
  ##  ##
  ##########  Copyright (c) 2007-2019 Pivotal Software, Inc.
  ######  ##
  ##########  Licensed under the MPL 1.1. Website: https://rabbitmq.com

  Doc guides: https://rabbitmq.com/documentation.html
  Support:    https://rabbitmq.com/contact.html
  Tutorials:  https://rabbitmq.com/getstarted.html
  Monitoring: https://rabbitmq.com/monitoring.html

  Logs: /var/log/rabbitmq/rabbit@c5c45057cb4c.log
        /var/log/rabbitmq/rabbit@c5c45057cb4c_upgrade.log

  Config file(s): (none)

  Starting broker... completed with 0 plugins.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem.

The problem is that everything worked perfectly on my laptop. I have no idea how to reproduce this on another machine.

My docker version:

docker -v
Docker version 20.10.8, build 3967b7d28e

What is the expected behavior?

I'd like to get the welcome page from localhost.

Did this work in previous versions of DocumentServer?

I've tried multiple versions of the docker images for onlyoffice documentserver, all have the same result on my server.

DocumentServer Docker tag:

Host Operating System: Linux 5.14.8-arch1-1 #1 SMP PREEMPT Sun, 26 Sep 2021 19:36:15 +0000 x86_64 GNU/Linux

If you run DocumentServer on your server with command

sudo docker run -i -t -d -p 8080:80 -v /app/onlyoffice/DocumentServer/logs:/var/log/onlyoffice -v /app/onlyoffice/DocumentServer/data:/var/www/onlyoffice/Data -v /app/onlyoffice/DocumentServer/lib:/var/lib/onlyoffice -v /app/onlyoffice/DocumentServer/rabbitmq:/var/lib/rabbitmq -v /app/onlyoffice/DocumentServer/redis:/var/lib/redis -v /app/onlyoffice/DocumentServer/db:/var/lib/postgresql onlyoffice/documentserver

I'm sure that problem somewhere in mounts, I think you can safely remove mount of rabbitmq and redis and remove all other mounts if necceserry

I've tried running onlyoffice/documentserver with and without mounts. I dont understand what you want me to do exactly.

Got it, so running DocumentServer container without any mount cause troubles also

One thing I've remembered - that I think some of our services (Maybe Redis, not Rabbit) do not like then ipv6 is enabled

Could you check it?

Some info I've googled - https://docs.docker.com/config/daemon/ipv6/ and this https://stackoverflow.com/a/41497555/820501

I've started the container with a specific address and now it shows no ipv6 address for the container.

I've started the image with: docker run -i -t -d -p 127.0.0.1:8080:80 -m 2000M --memory-swap -1 --dns-opt='options single-request' onlyoffice/documentserver

From inside the container:

root@f074e7787fbe:/var/log/rabbitmq# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
25: eth0@if26: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

From docker container ls:

CONTAINER ID   IMAGE                            COMMAND                  CREATED         STATUS         PORTS    NAMES
f074e7787fbe   onlyoffice/documentserver        "/app/ds/run-documen…"   9 minutes ago   Up 9 minutes   443/tcp, 127.0.0.1:8080->80/tcp    wonderful_rosalind

Yeah, I've got similar output with a completely fine server

Does disabling ipv6 help?

If not currently I've out of ideas. To help I think we need a way to reproduce your high memory load on any server, not just on your server

ipv6 disabled has no effect. Any logs you want me to post?

I've no idea, because if this problem is actual only for one of your server - until you have a way to reproduce the same problem at least on your laptop (which working fine as I understand) - I can only assume this problem is somehow related is how docker service (not DocumentServer, but docker service) is set up on your server - and I'm not very good at understanding docker internal mechanics

So I think I have the same issue, never used onlyoffice before I just tried installing/running for first time today and when trying to run it, first I got a bunch of

Waiting for connection to the localhost host on port 5672

and then i tried restarting the container and it seems to be stuck on

sed: -e expression #1, char 25: unknown option to `s'
 * Starting PostgreSQL 12 database server
   ...done.
 * Starting RabbitMQ Messaging Server rabbitmq-server

Any ideas?

Seems this is same error

sed: -e expression #1, char 25: unknown option to `s'

And I cannot reproduce it I've used the same command as OP and it's running without any error:

docker run -it -m 2000M --memory-swap -1 onlyoffice/documentserver
 * Starting PostgreSQL 12 database server                      [ OK ]
 ....

@RT-Tap Several questions

Are you sure you've using latest image of onlyoffice/documentserver? What's image hash, is it Digest: sha256:ba4d4bbdc0eef02e3706a039b757b3e8c512f5c741299ffaf0a1dc8c2421be2f Do you have some other docker run options? Could you reproduce this error on different server?

using : onlyoffice/documentserver@sha256:3fd7bf47c2dfaa2ab5204fdc731d0186007be0807590fe91efb267bd19212f1c I thought it was latest image as i just pulled it literally about 8 hours ago.

After some time it is now showing Waiting for connection to the localhost host on port 5672 again. I can't seem to get the image running at all. I'll try pulling latest image again and retry.

BTW: using docker compose with the following:

version: "3.8"
services:
----snip----
  onlyoffice_document_server:
    container_name: onlyofficeDocumentServer
    image: onlyoffice/documentserver
    restart: always
    expose:
      - "80"
    volumes:
      - "../onlyoffice_DocServer/data:/var/www/onlyoffice/Data"
      - "../onlyoffice_DocServer/logs:/var/log/onlyoffice"
    networks:
      default:
        ipv4_address: 172.16.0.5
    depends_on:
      - nextcloud
---snip----

@RT-Tap This is strange

I can pull my image (latest) by sha like these:

docker pull onlyoffice/documentserver@sha256:ba4d4bbdc0eef02e3706a039b757b3e8c512f5c741299ffaf0a1dc8c2421be2f

But yours hash give me 500 error

docker pull onlyoffice/documentserver@sha256:3fd7bf47c2dfaa2ab5204fdc731d0186007be0807590fe91efb267bd19212f1c
Error response from daemon: received unexpected HTTP status: 500 Internal Server Error

So I do not know what's inside Could you attach to running container and see which version of onlyoffice-documentserver package installed

You can do it by

docker exec -it container_id bash
dpkg -l | grep only

sooo that gives me : ii onlyoffice-documentserver 6.4.1-45 amd64 Online editors for text documents, spreadsheets, and presentations and I just pulled this image literally right now by deleting the old one and using docker pull onlyoffice/documentserver:latest and started it with docker-compose up -d from the docker docker compose file i posted above

ii onlyoffice-documentserver 6.4.1-45 amd64 Online editors for text documents, spreadsheets, and presentations

This is correct, latest version of DocumentServer

But image id is different

Could you try to run

docker pull onlyoffice/documentserver@sha256:ba4d4bbdc0eef02e3706a039b757b3e8c512f5c741299ffaf0a1dc8c2421be2f

This is our image, but your is I don't know what it is, are you sure you didn't setup any proxy or local docker hub and it's replacing images I'm not sure what it is, but 3fd7bf47c2dfaa2ab5204fdc731d0186007be0807590fe91efb267bd19212f1c not our image hash of onlyoffice/documentserver:latest

no proxy or docker hub i know of and i'm actually already in the process of doing that right now, its extracting already so ill report back in a bit

UPDATE:

Unfortunately not much has changed, it seems to take some time sitting on

sed: -e expression #1, char 25: unknown option to `s'
 * Starting PostgreSQL 12 database server
   ...done.
 * Starting RabbitMQ Messaging Server rabbitmq-server

after which (~5 min or so) it will most likely go back to Waiting for connection to the localhost host on port 5672

its pretty late here im going to take a quick nap and be back, hopefully theres something missing in this equation

its pretty late here im going to take a quick nap and be back, hopefully theres something missing in this equation

Yeah, ok

The only advice I can give - found out which option results sed: -e expression... error, since I cannot reproduce it, but it may happen because of incorrect mounts volumes or incorrect env variables send to docker run

I found a onlyoffice issue that also had the same error on dev.onlyoffice.com cached from google But it seems they just ignore the sed issue

Yeah, don't think it's ignoring, just missed

But as I see the result is the same - our teams cannot reproduce this problem on our hardware. Same as current issue

BTW - the old forum is available at https://dev-old.onlyoffice.org/viewtopic.php?t=35037&p=44120 right now, without google cache

Actually, the error sed: -e expression #1, char 25: unknown option to s also occurs locally on my laptop. The onlyoffice/documentserver image runs perfectly though.

@FHFS This is strange, but this error is not good either.

So back to step one

We got two situation

sed error
Failing rabbitmq

Both of them we cannot reproduce correctly in 100% situation and we do not understand why

I've checked the status of rabbitmq-server inside the container:

root@2058e5b3864d:/var/log/rabbitmq# service rabbitmq-server status
Status of node rabbit@2058e5b3864d ...
Runtime

OS PID: 631
OS: Linux
Uptime (seconds): 66565
RabbitMQ version: 3.8.2
Node name: rabbit@2058e5b3864d
Erlang configuration: Erlang/OTP 22 [erts-10.6.4] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:64]
Erlang processes: 328 used, 1048576 limit
Scheduler run queue: 1
Cluster heartbeat timeout (net_ticktime): 60

Plugins

Enabled plugin file: /etc/rabbitmq/enabled_plugins
Enabled plugins:

Data directory

Node data directory: /var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d

Config files

Log file(s)

 * /var/log/rabbitmq/rabbit@2058e5b3864d.log
 * /var/log/rabbitmq/rabbit@2058e5b3864d_upgrade.log

Alarms

(none)

Memory

Calculation strategy: rss
Memory high watermark setting: 0.4 of available memory, computed to: 1.6442 gb
other_system: 1.6206 gb (72.5 %)
allocated_unused: 0.5508 gb (24.64 %)
code: 0.0268 gb (1.2 %)
other_proc: 0.0239 gb (1.07 %)
binary: 0.0081 gb (0.36 %)
other_ets: 0.0026 gb (0.12 %)
atom: 0.0014 gb (0.06 %)
connection_other: 0.0005 gb (0.02 %)
queue_procs: 0.0001 gb (0.01 %)
mnesia: 0.0001 gb (0.0 %)
metrics: 0.0001 gb (0.0 %)
connection_channels: 0.0 gb (0.0 %)
connection_readers: 0.0 gb (0.0 %)
msg_index: 0.0 gb (0.0 %)
connection_writers: 0.0 gb (0.0 %)
plugins: 0.0 gb (0.0 %)
quorum_ets: 0.0 gb (0.0 %)
mgmt_db: 0.0 gb (0.0 %)
queue_slave_procs: 0.0 gb (0.0 %)
quorum_queue_procs: 0.0 gb (0.0 %)
reserved_unallocated: 0.0 gb (0.0 %)

File Descriptors

Total: 5, limit: 1073741719
Sockets: 3, limit: 966367545

Free Disk Space

Low free disk space watermark: 0.05 gb
Free disk space: 12.6334 gb

Totals

Connection count: 3
Queue count: 5
Virtual host count: 1

Listeners

Interface: [::], port: 25672, protocol: clustering, purpose: inter-node and CLI tool communication
Interface: [::], port: 5672, protocol: amqp, purpose: AMQP 0-9-1 and AMQP 1.0

I have no idea if this running ok or not.

Event it's running correctly

Waiting for connection to the localhost host on port 5672

from DocumentServer means that DocumentServer cannot connect by any reason This not mean that rabbitmq is not running, but means that DocumentServer cannot connect to it for any reason

I dont see any errors in the logs at /var/log/onlyoffice/documentserver Do you know where I should look for errors?

Do you know why rabbitmq is using so much memory? I've checked the logs for rabbitmq and it is doing something:

root@2058e5b3864d:/# cat /var/log/rabbitmq/rabbit@2058e5b3864d.log
2021-10-05 15:16:50.893 [info] <0.8.0> Log file opened with Lager
2021-10-05 15:16:54.334 [info] <0.8.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.335 [info] <0.8.0> Feature flags:   [ ] implicit_default_bindings
2021-10-05 15:16:54.335 [info] <0.8.0> Feature flags:   [ ] quorum_queue
2021-10-05 15:16:54.335 [info] <0.8.0> Feature flags:   [ ] virtual_host_metadata
2021-10-05 15:16:54.335 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.372 [info] <0.258.0> ra: meta data store initialised. 0 record(s) recovered
2021-10-05 15:16:54.372 [info] <0.263.0> WAL: recovering []
2021-10-05 15:16:54.373 [info] <0.267.0> 
 Starting RabbitMQ 3.8.2 on Erlang 22.2.7
 Copyright (c) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com
2021-10-05 15:16:54.374 [info] <0.267.0> 
 node           : rabbit@2058e5b3864d
 home dir       : /var/lib/rabbitmq
 config file(s) : (none)
 cookie hash    : lPuRN8ZUrJEcO7Y6Ae/DhA==
 log(s)         : /var/log/rabbitmq/rabbit@2058e5b3864d.log
                : /var/log/rabbitmq/rabbit@2058e5b3864d_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d
2021-10-05 15:16:54.381 [info] <0.267.0> Running boot step pre_boot defined by app rabbit
2021-10-05 15:16:54.382 [info] <0.267.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-10-05 15:16:54.382 [info] <0.267.0> Running boot step rabbit_alarm defined by app rabbit
2021-10-05 15:16:54.387 [info] <0.273.0> Memory high watermark set to 1568 MiB (1644219596 bytes) of 3920 MiB (4110548992 bytes) total
2021-10-05 15:16:54.388 [info] <0.273.0> vm_memory_high_watermark set. Memory used:1687019520 allowed:1644219596
2021-10-05 15:16:54.388 [warning] <0.271.0> memory resource limit alarm set on node rabbit@2058e5b3864d.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
2021-10-05 15:16:54.392 [info] <0.275.0> Enabling free disk space monitoring
2021-10-05 15:16:54.392 [info] <0.275.0> Disk free limit set to 50MB
2021-10-05 15:16:54.395 [info] <0.267.0> Running boot step code_server_cache defined by app rabbit
2021-10-05 15:16:54.396 [info] <0.267.0> Running boot step file_handle_cache defined by app rabbit
2021-10-05 15:16:54.396 [info] <0.278.0> Limiting to approx 1073741719 file handles (966367545 sockets)
2021-10-05 15:16:54.396 [info] <0.279.0> FHC read buffering:  OFF
2021-10-05 15:16:54.396 [info] <0.279.0> FHC write buffering: ON
2021-10-05 15:16:54.398 [info] <0.267.0> Running boot step worker_pool defined by app rabbit
2021-10-05 15:16:54.398 [info] <0.268.0> Will use 1 processes for default worker pool
2021-10-05 15:16:54.398 [info] <0.268.0> Starting worker pool 'worker_pool' with 1 processes in it
2021-10-05 15:16:54.398 [info] <0.267.0> Running boot step database defined by app rabbit
2021-10-05 15:16:54.398 [info] <0.267.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d is empty. Assuming we need to join an existing cluster or initialise from scratch...
2021-10-05 15:16:54.399 [info] <0.267.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2021-10-05 15:16:54.399 [info] <0.267.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2021-10-05 15:16:54.399 [info] <0.267.0> Peer discovery backend does not support locking, falling back to randomized delay
2021-10-05 15:16:54.399 [info] <0.267.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2021-10-05 15:16:54.399 [info] <0.267.0> All discovered existing cluster peers: 
2021-10-05 15:16:54.399 [info] <0.267.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
2021-10-05 15:16:54.406 [info] <0.43.0> Application mnesia exited with reason: stopped
2021-10-05 15:16:54.614 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.633 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.634 [info] <0.267.0> Feature flag `implicit_default_bindings`: supported, attempt to enable...
2021-10-05 15:16:54.634 [info] <0.267.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing
2021-10-05 15:16:54.639 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.639 [info] <0.267.0> Feature flags:   [~] implicit_default_bindings
2021-10-05 15:16:54.639 [info] <0.267.0> Feature flags:   [ ] quorum_queue
2021-10-05 15:16:54.640 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-10-05 15:16:54.640 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.650 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-10-05 15:16:54.652 [info] <0.267.0> Feature flag `implicit_default_bindings`: mark as enabled=true
2021-10-05 15:16:54.658 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.658 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-10-05 15:16:54.658 [info] <0.267.0> Feature flags:   [ ] quorum_queue
2021-10-05 15:16:54.658 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-10-05 15:16:54.658 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.669 [info] <0.267.0> Feature flag `quorum_queue`: supported, attempt to enable...
2021-10-05 15:16:54.670 [info] <0.267.0> Feature flag `quorum_queue`: mark as enabled=state_changing
2021-10-05 15:16:54.675 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.675 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-10-05 15:16:54.675 [info] <0.267.0> Feature flags:   [~] quorum_queue
2021-10-05 15:16:54.676 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-10-05 15:16:54.676 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.686 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.687 [info] <0.267.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_queue...
2021-10-05 15:16:54.711 [info] <0.267.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_durable_queue...
2021-10-05 15:16:54.741 [info] <0.267.0> Feature flag `quorum_queue`:   Mnesia tables migration done
2021-10-05 15:16:54.741 [info] <0.267.0> Feature flag `quorum_queue`: mark as enabled=true
2021-10-05 15:16:54.748 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.748 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-10-05 15:16:54.748 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-10-05 15:16:54.748 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-10-05 15:16:54.748 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.759 [info] <0.267.0> Feature flag `virtual_host_metadata`: supported, attempt to enable...
2021-10-05 15:16:54.759 [info] <0.267.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing
2021-10-05 15:16:54.765 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.766 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-10-05 15:16:54.766 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-10-05 15:16:54.766 [info] <0.267.0> Feature flags:   [~] virtual_host_metadata
2021-10-05 15:16:54.766 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.775 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.812 [info] <0.267.0> Feature flag `virtual_host_metadata`: mark as enabled=true
2021-10-05 15:16:54.819 [info] <0.267.0> Feature flags: list of feature flags found:
2021-10-05 15:16:54.819 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-10-05 15:16:54.819 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-10-05 15:16:54.819 [info] <0.267.0> Feature flags:   [x] virtual_host_metadata
2021-10-05 15:16:54.819 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-10-05 15:16:54.830 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.858 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-05 15:16:54.858 [info] <0.267.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step database_sync defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step feature_flags defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step codec_correctness_check defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step external_infrastructure defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step rabbit_registry defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step rabbit_event defined by app rabbit
2021-10-05 15:16:54.858 [info] <0.267.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-10-05 15:16:54.859 [info] <0.267.0> Running boot step kernel_ready defined by app rabbit
2021-10-05 15:16:54.860 [info] <0.267.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-10-05 15:16:54.860 [info] <0.267.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-10-05 15:16:54.862 [info] <0.500.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-10-05 15:16:54.862 [info] <0.267.0> Running boot step guid_generator defined by app rabbit
2021-10-05 15:16:54.870 [info] <0.267.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-10-05 15:16:54.870 [info] <0.506.0> Starting rabbit_node_monitor
2021-10-05 15:16:54.870 [info] <0.267.0> Running boot step delegate_sup defined by app rabbit
2021-10-05 15:16:54.870 [info] <0.267.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-10-05 15:16:54.871 [info] <0.267.0> Running boot step core_initialized defined by app rabbit
2021-10-05 15:16:54.871 [info] <0.267.0> Running boot step upgrade_queues defined by app rabbit
2021-10-05 15:16:54.884 [info] <0.267.0> message_store upgrades: 1 to apply
2021-10-05 15:16:54.884 [info] <0.267.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2021-10-05 15:16:54.884 [info] <0.267.0> message_store upgrades: No durable queues found. Skipping message store migration
2021-10-05 15:16:54.884 [info] <0.267.0> message_store upgrades: Removing the old message store data
2021-10-05 15:16:54.884 [info] <0.267.0> message_store upgrades: All upgrades applied successfully
2021-10-05 15:16:54.904 [info] <0.267.0> Running boot step rabbit_connection_tracking defined by app rabbit
2021-10-05 15:16:54.904 [info] <0.267.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-10-05 15:16:54.905 [info] <0.267.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-10-05 15:16:54.905 [info] <0.267.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-10-05 15:16:54.905 [info] <0.267.0> Running boot step rabbit_policies defined by app rabbit
2021-10-05 15:16:54.906 [info] <0.267.0> Running boot step rabbit_policy defined by app rabbit
2021-10-05 15:16:54.906 [info] <0.267.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-10-05 15:16:54.906 [info] <0.267.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-10-05 15:16:54.906 [info] <0.267.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-10-05 15:16:54.906 [info] <0.267.0> Running boot step recovery defined by app rabbit
2021-10-05 15:16:54.907 [info] <0.267.0> Running boot step load_core_definitions defined by app rabbit
2021-10-05 15:16:54.907 [info] <0.267.0> Running boot step empty_db_check defined by app rabbit
2021-10-05 15:16:54.907 [info] <0.267.0> Adding vhost '/' (description: 'Default virtual host')
2021-10-05 15:16:54.964 [info] <0.543.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-10-05 15:16:54.972 [info] <0.543.0> Starting message stores for vhost '/'
2021-10-05 15:16:54.973 [info] <0.547.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-10-05 15:16:54.974 [info] <0.543.0> Started message store of type transient for vhost '/'
2021-10-05 15:16:54.974 [info] <0.550.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-10-05 15:16:54.975 [warning] <0.550.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-10-05 15:16:54.975 [info] <0.543.0> Started message store of type persistent for vhost '/'
2021-10-05 15:16:54.977 [info] <0.267.0> Creating user 'guest'
2021-10-05 15:16:54.983 [info] <0.267.0> Setting user tags for user 'guest' to [administrator]
2021-10-05 15:16:54.989 [info] <0.267.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2021-10-05 15:16:54.996 [info] <0.267.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-10-05 15:16:54.996 [info] <0.267.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-10-05 15:16:54.996 [info] <0.267.0> Running boot step background_gc defined by app rabbit
2021-10-05 15:16:54.996 [info] <0.267.0> Running boot step connection_tracking defined by app rabbit
2021-10-05 15:16:55.005 [info] <0.267.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@2058e5b3864d
2021-10-05 15:16:55.020 [info] <0.267.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@2058e5b3864d
2021-10-05 15:16:55.020 [info] <0.267.0> Running boot step routing_ready defined by app rabbit
2021-10-05 15:16:55.020 [info] <0.267.0> Running boot step pre_flight defined by app rabbit
2021-10-05 15:16:55.020 [info] <0.267.0> Running boot step notify_cluster defined by app rabbit
2021-10-05 15:16:55.020 [info] <0.267.0> Running boot step networking defined by app rabbit
2021-10-05 15:16:55.022 [info] <0.596.0> started TCP listener on [::]:5672
2021-10-05 15:16:55.022 [info] <0.267.0> Running boot step cluster_name defined by app rabbit
2021-10-05 15:16:55.022 [info] <0.267.0> Running boot step direct_client defined by app rabbit
2021-10-05 15:16:55.022 [notice] <0.104.0> Changed loghwm of /var/log/rabbitmq/rabbit@2058e5b3864d.log to 50
2021-10-05 15:16:55.319 [info] <0.8.0> Server startup complete; 0 plugins started.
2021-10-05 15:16:58.916 [info] <0.609.0> accepting AMQP connection <0.609.0> (127.0.0.1:56888 -> 127.0.0.1:5672)
2021-10-05 15:16:58.934 [info] <0.609.0> connection <0.609.0> (127.0.0.1:56888 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-05 15:16:59.424 [info] <0.638.0> accepting AMQP connection <0.638.0> (127.0.0.1:56892 -> 127.0.0.1:5672)
2021-10-05 15:16:59.428 [info] <0.638.0> connection <0.638.0> (127.0.0.1:56892 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-05 15:16:59.450 [info] <0.659.0> accepting AMQP connection <0.659.0> (127.0.0.1:56894 -> 127.0.0.1:5672)
2021-10-05 15:16:59.454 [info] <0.659.0> connection <0.659.0> (127.0.0.1:56894 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-05 15:17:02.391 [info] <0.273.0> vm_memory_high_watermark clear. Memory used:1638010880 allowed:1644219596
2021-10-05 15:17:02.391 [warning] <0.271.0> memory resource limit alarm cleared on node rabbit@2058e5b3864d
2021-10-05 15:17:02.391 [warning] <0.271.0> memory resource limit alarm cleared across the cluster
2021-10-05 15:17:54.525 [warning] <0.659.0> closing AMQP connection <0.659.0> (127.0.0.1:56894 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-05 15:17:54.525 [warning] <0.638.0> closing AMQP connection <0.638.0> (127.0.0.1:56892 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-05 15:17:55.853 [warning] <0.609.0> closing AMQP connection <0.609.0> (127.0.0.1:56888 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-05 15:17:56.055 [info] <0.773.0> accepting AMQP connection <0.773.0> (127.0.0.1:56896 -> 127.0.0.1:5672)
2021-10-05 15:17:56.060 [info] <0.773.0> connection <0.773.0> (127.0.0.1:56896 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-05 15:17:56.076 [info] <0.793.0> accepting AMQP connection <0.793.0> (127.0.0.1:56898 -> 127.0.0.1:5672)
2021-10-05 15:17:56.081 [info] <0.793.0> connection <0.793.0> (127.0.0.1:56898 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-05 15:18:06.345 [info] <0.831.0> accepting AMQP connection <0.831.0> (127.0.0.1:56900 -> 127.0.0.1:5672)
2021-10-05 15:18:06.350 [info] <0.831.0> connection <0.831.0> (127.0.0.1:56900 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:16:00.270 [info] <0.8148.2> accepting AMQP connection <0.8148.2> (127.0.0.1:56904 -> 127.0.0.1:5672)
2021-10-06 11:16:00.279 [info] <0.8148.2> connection <0.8148.2> (127.0.0.1:56904 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:30:23.944 [info] <0.8.0> Log file opened with Lager
2021-10-06 11:30:27.524 [info] <0.8.0> Feature flags: list of feature flags found:
2021-10-06 11:30:27.524 [info] <0.8.0> Feature flags:   [x] implicit_default_bindings
2021-10-06 11:30:27.524 [info] <0.8.0> Feature flags:   [x] quorum_queue
2021-10-06 11:30:27.524 [info] <0.8.0> Feature flags:   [x] virtual_host_metadata
2021-10-06 11:30:27.524 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2021-10-06 11:30:27.634 [info] <0.284.0> ra: meta data store initialised. 0 record(s) recovered
2021-10-06 11:30:27.635 [info] <0.291.0> WAL: recovering ["/var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d/quorum/rabbit@2058e5b3864d/00000001.wal"]
2021-10-06 11:30:27.637 [info] <0.297.0> 
 Starting RabbitMQ 3.8.2 on Erlang 22.2.7
 Copyright (c) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com
2021-10-06 11:30:27.638 [info] <0.297.0> 
 node           : rabbit@2058e5b3864d
 home dir       : /var/lib/rabbitmq
 config file(s) : (none)
 cookie hash    : lPuRN8ZUrJEcO7Y6Ae/DhA==
 log(s)         : /var/log/rabbitmq/rabbit@2058e5b3864d.log
                : /var/log/rabbitmq/rabbit@2058e5b3864d_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d
2021-10-06 11:30:27.646 [info] <0.297.0> Running boot step pre_boot defined by app rabbit
2021-10-06 11:30:27.646 [info] <0.297.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-10-06 11:30:27.646 [info] <0.297.0> Running boot step rabbit_alarm defined by app rabbit
2021-10-06 11:30:27.651 [info] <0.305.0> Memory high watermark set to 1568 MiB (1644219596 bytes) of 3920 MiB (4110548992 bytes) total
2021-10-06 11:30:27.651 [info] <0.305.0> vm_memory_high_watermark set. Memory used:1688391680 allowed:1644219596
2021-10-06 11:30:27.652 [warning] <0.303.0> memory resource limit alarm set on node rabbit@2058e5b3864d.

**********************************************************
*** Publishers will be blocked until this alarm clears ***
**********************************************************
2021-10-06 11:30:27.657 [info] <0.316.0> Enabling free disk space monitoring
2021-10-06 11:30:27.657 [info] <0.316.0> Disk free limit set to 50MB
2021-10-06 11:30:27.662 [info] <0.297.0> Running boot step code_server_cache defined by app rabbit
2021-10-06 11:30:27.662 [info] <0.297.0> Running boot step file_handle_cache defined by app rabbit
2021-10-06 11:30:27.662 [info] <0.340.0> Limiting to approx 1073741719 file handles (966367545 sockets)
2021-10-06 11:30:27.663 [info] <0.341.0> FHC read buffering:  OFF
2021-10-06 11:30:27.663 [info] <0.341.0> FHC write buffering: ON
2021-10-06 11:30:27.665 [info] <0.297.0> Running boot step worker_pool defined by app rabbit
2021-10-06 11:30:27.665 [info] <0.300.0> Will use 1 processes for default worker pool
2021-10-06 11:30:27.665 [info] <0.300.0> Starting worker pool 'worker_pool' with 1 processes in it
2021-10-06 11:30:27.665 [info] <0.297.0> Running boot step database defined by app rabbit
2021-10-06 11:30:27.667 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-06 11:30:27.676 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-06 11:30:27.714 [info] <0.297.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-10-06 11:30:27.715 [info] <0.297.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step database_sync defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step feature_flags defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step codec_correctness_check defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step external_infrastructure defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_registry defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_event defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-10-06 11:30:27.715 [info] <0.297.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step kernel_ready defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-10-06 11:30:27.716 [info] <0.297.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-10-06 11:30:27.719 [info] <0.374.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-10-06 11:30:27.720 [info] <0.297.0> Running boot step guid_generator defined by app rabbit
2021-10-06 11:30:27.730 [info] <0.297.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-10-06 11:30:27.730 [info] <0.380.0> Starting rabbit_node_monitor
2021-10-06 11:30:27.730 [info] <0.297.0> Running boot step delegate_sup defined by app rabbit
2021-10-06 11:30:27.731 [info] <0.297.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-10-06 11:30:27.731 [info] <0.297.0> Running boot step core_initialized defined by app rabbit
2021-10-06 11:30:27.731 [info] <0.297.0> Running boot step upgrade_queues defined by app rabbit
2021-10-06 11:30:27.744 [info] <0.297.0> Running boot step rabbit_connection_tracking defined by app rabbit
2021-10-06 11:30:27.744 [info] <0.297.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-10-06 11:30:27.744 [info] <0.297.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-10-06 11:30:27.745 [info] <0.297.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-10-06 11:30:27.745 [info] <0.297.0> Running boot step rabbit_policies defined by app rabbit
2021-10-06 11:30:27.746 [info] <0.297.0> Running boot step rabbit_policy defined by app rabbit
2021-10-06 11:30:27.746 [info] <0.297.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-10-06 11:30:27.746 [info] <0.297.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-10-06 11:30:27.746 [info] <0.297.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-10-06 11:30:27.746 [info] <0.297.0> Running boot step recovery defined by app rabbit
2021-10-06 11:30:27.747 [info] <0.410.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@2058e5b3864d/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-10-06 11:30:27.756 [info] <0.410.0> Starting message stores for vhost '/'
2021-10-06 11:30:27.757 [info] <0.414.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-10-06 11:30:27.758 [info] <0.410.0> Started message store of type transient for vhost '/'
2021-10-06 11:30:27.759 [info] <0.417.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-10-06 11:30:27.760 [warning] <0.417.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-10-06 11:30:27.762 [info] <0.410.0> Started message store of type persistent for vhost '/'
2021-10-06 11:30:27.781 [info] <0.297.0> Running boot step load_core_definitions defined by app rabbit
2021-10-06 11:30:27.781 [info] <0.297.0> Running boot step empty_db_check defined by app rabbit
2021-10-06 11:30:27.782 [info] <0.297.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-10-06 11:30:27.782 [info] <0.297.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-10-06 11:30:27.782 [info] <0.297.0> Running boot step background_gc defined by app rabbit
2021-10-06 11:30:27.782 [info] <0.297.0> Running boot step connection_tracking defined by app rabbit
2021-10-06 11:30:27.782 [info] <0.297.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@2058e5b3864d
2021-10-06 11:30:27.782 [info] <0.297.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@2058e5b3864d
2021-10-06 11:30:27.782 [info] <0.297.0> Running boot step routing_ready defined by app rabbit
2021-10-06 11:30:27.783 [info] <0.297.0> Running boot step pre_flight defined by app rabbit
2021-10-06 11:30:27.783 [info] <0.297.0> Running boot step notify_cluster defined by app rabbit
2021-10-06 11:30:27.783 [info] <0.297.0> Running boot step networking defined by app rabbit
2021-10-06 11:30:27.785 [info] <0.472.0> started TCP listener on [::]:5672
2021-10-06 11:30:27.785 [info] <0.297.0> Running boot step cluster_name defined by app rabbit
2021-10-06 11:30:27.785 [info] <0.297.0> Running boot step direct_client defined by app rabbit
2021-10-06 11:30:27.785 [notice] <0.104.0> Changed loghwm of /var/log/rabbitmq/rabbit@2058e5b3864d.log to 50
2021-10-06 11:30:28.142 [info] <0.8.0> Server startup complete; 0 plugins started.
2021-10-06 11:30:31.752 [info] <0.485.0> accepting AMQP connection <0.485.0> (127.0.0.1:57060 -> 127.0.0.1:5672)
2021-10-06 11:30:31.759 [info] <0.485.0> connection <0.485.0> (127.0.0.1:57060 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:30:31.817 [info] <0.504.0> accepting AMQP connection <0.504.0> (127.0.0.1:57062 -> 127.0.0.1:5672)
2021-10-06 11:30:31.822 [info] <0.504.0> connection <0.504.0> (127.0.0.1:57062 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:30:31.835 [info] <0.524.0> accepting AMQP connection <0.524.0> (127.0.0.1:57064 -> 127.0.0.1:5672)
2021-10-06 11:30:31.838 [info] <0.524.0> connection <0.524.0> (127.0.0.1:57064 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:30:33.656 [info] <0.305.0> vm_memory_high_watermark clear. Memory used:1623572480 allowed:1644219596
2021-10-06 11:30:33.656 [warning] <0.303.0> memory resource limit alarm cleared on node rabbit@2058e5b3864d
2021-10-06 11:30:33.656 [warning] <0.303.0> memory resource limit alarm cleared across the cluster
2021-10-06 11:31:27.132 [warning] <0.524.0> closing AMQP connection <0.524.0> (127.0.0.1:57064 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-06 11:31:27.132 [warning] <0.504.0> closing AMQP connection <0.504.0> (127.0.0.1:57062 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-06 11:31:28.435 [warning] <0.485.0> closing AMQP connection <0.485.0> (127.0.0.1:57060 -> 127.0.0.1:5672, vhost: '/', user: 'guest'):
client unexpectedly closed TCP connection
2021-10-06 11:31:28.916 [info] <0.630.0> accepting AMQP connection <0.630.0> (127.0.0.1:57066 -> 127.0.0.1:5672)
2021-10-06 11:31:28.924 [info] <0.630.0> connection <0.630.0> (127.0.0.1:57066 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:31:28.936 [info] <0.650.0> accepting AMQP connection <0.650.0> (127.0.0.1:57068 -> 127.0.0.1:5672)
2021-10-06 11:31:28.941 [info] <0.650.0> connection <0.650.0> (127.0.0.1:57068 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
2021-10-06 11:31:39.154 [info] <0.688.0> accepting AMQP connection <0.688.0> (127.0.0.1:57070 -> 127.0.0.1:5672)
2021-10-06 11:31:39.158 [info] <0.688.0> connection <0.688.0> (127.0.0.1:57070 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

Do you know why rabbitmq is using so much memory?

No, I do not know, I think this may be a side-effect of an original problem which we currently cannot localize Incorrect configuration all DocumentServer operations, including rabbitmq enough 2gb of memory

So the last time i tried to run documentserver I go the following error:

 * Starting PostgreSQL 12 database server
   ...done.
 * Starting RabbitMQ Messaging Server rabbitmq-server
 * FAILED - check /var/log/rabbitmq/startup_\{log, _err\}
   ...fail!
Waiting for connection to the localhost host on port 5672

now I'm having a hard time trying to read those logs because a) those logs dont seem to be in my mounted logs volume - if i go to the directory that is mounted for logs I have the following directories: logs | -- documentserver | |-- converter <-- empty | | -- docservice <-- empty | | -- metrics <-- empty | --documentserver-example <-- empty

b) when i go into the container none of the normal commands for viewing a file work , after cd to /var/log/rabbitmq I've tried cat startup_err which returns mkdir: cannot create directory ‘/var/run/rabbitmq’: Permission denied and tail startup_err returns mkdir: cannot create directory ‘/var/run/rabbitmq’: Permission denied , and for more startup_err i get same thing mkdir: cannot create directory ‘/var/run/rabbitmq’: Permission denied and then for less startup_err i just get bash: less: command not found

Is there a way to view this log file?

EDIT: I'm not sure why i cant access those files however I do have and can access files such as rabbit@#######.log but get just about the same things as FHFS

Actually I have a slightly different log output : Looks like something is asking rabbitmq to stop?

2021-09-19 19:56:13.087 [info] <0.8.0> Log file opened with Lager
2021-09-19 19:56:15.590 [info] <0.8.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.591 [info] <0.8.0> Feature flags:   [ ] implicit_default_bindings
2021-09-19 19:56:15.591 [info] <0.8.0> Feature flags:   [ ] quorum_queue
2021-09-19 19:56:15.591 [info] <0.8.0> Feature flags:   [ ] virtual_host_metadata
2021-09-19 19:56:15.591 [info] <0.8.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.622 [info] <0.258.0> ra: meta data store initialised. 0 record(s) recovered
2021-09-19 19:56:15.623 [info] <0.263.0> WAL: recovering []
2021-09-19 19:56:15.624 [info] <0.267.0> 
 Starting RabbitMQ 3.8.2 on Erlang 22.2.7
 Copyright (c) 2007-2019 Pivotal Software, Inc.
 Licensed under the MPL 1.1. Website: https://rabbitmq.com
2021-09-19 19:56:15.624 [info] <0.267.0> 
 node           : rabbit@e25344c49247
 home dir       : /var/lib/rabbitmq
 config file(s) : (none)
 cookie hash    : lPuRN8ZUrJEcO7Y6Ae/DhA==
 log(s)         : /var/log/rabbitmq/rabbit@e25344c49247.log
                : /var/log/rabbitmq/rabbit@e25344c49247_upgrade.log
 database dir   : /var/lib/rabbitmq/mnesia/rabbit@e25344c49247
2021-09-19 19:56:15.635 [info] <0.267.0> Running boot step pre_boot defined by app rabbit
2021-09-19 19:56:15.635 [info] <0.267.0> Running boot step rabbit_core_metrics defined by app rabbit
2021-09-19 19:56:15.636 [info] <0.267.0> Running boot step rabbit_alarm defined by app rabbit
2021-09-19 19:56:15.640 [info] <0.273.0> Memory high watermark set to 3193 MiB (3348478361 bytes) of 7983 MiB (8371195904 bytes) total
2021-09-19 19:56:15.644 [info] <0.275.0> Enabling free disk space monitoring
2021-09-19 19:56:15.644 [info] <0.275.0> Disk free limit set to 50MB
2021-09-19 19:56:15.648 [info] <0.267.0> Running boot step code_server_cache defined by app rabbit
2021-09-19 19:56:15.648 [info] <0.267.0> Running boot step file_handle_cache defined by app rabbit
2021-09-19 19:56:15.649 [info] <0.278.0> Limiting to approx 524191 file handles (471769 sockets)
2021-09-19 19:56:15.649 [info] <0.279.0> FHC read buffering:  OFF
2021-09-19 19:56:15.649 [info] <0.279.0> FHC write buffering: ON
2021-09-19 19:56:15.649 [info] <0.267.0> Running boot step worker_pool defined by app rabbit
2021-09-19 19:56:15.649 [info] <0.268.0> Will use 4 processes for default worker pool
2021-09-19 19:56:15.649 [info] <0.268.0> Starting worker pool 'worker_pool' with 4 processes in it
2021-09-19 19:56:15.650 [info] <0.267.0> Running boot step database defined by app rabbit
2021-09-19 19:56:15.650 [info] <0.267.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@e25344c49247 is empty. Assuming we need to join an existing cluster or initialise from scratch...
2021-09-19 19:56:15.650 [info] <0.267.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
2021-09-19 19:56:15.650 [info] <0.267.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
2021-09-19 19:56:15.650 [info] <0.267.0> Peer discovery backend does not support locking, falling back to randomized delay
2021-09-19 19:56:15.650 [info] <0.267.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping randomized startup delay.
2021-09-19 19:56:15.650 [info] <0.267.0> All discovered existing cluster peers: 
2021-09-19 19:56:15.650 [info] <0.267.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
2021-09-19 19:56:15.652 [info] <0.43.0> Application mnesia exited with reason: stopped
2021-09-19 19:56:15.733 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.752 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.752 [info] <0.267.0> Feature flag `implicit_default_bindings`: supported, attempt to enable...
2021-09-19 19:56:15.752 [info] <0.267.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing
2021-09-19 19:56:15.761 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.761 [info] <0.267.0> Feature flags:   [~] implicit_default_bindings
2021-09-19 19:56:15.761 [info] <0.267.0> Feature flags:   [ ] quorum_queue
2021-09-19 19:56:15.761 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-09-19 19:56:15.761 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.772 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
2021-09-19 19:56:15.772 [info] <0.267.0> Feature flag `implicit_default_bindings`: mark as enabled=true
2021-09-19 19:56:15.781 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.781 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-09-19 19:56:15.781 [info] <0.267.0> Feature flags:   [ ] quorum_queue
2021-09-19 19:56:15.781 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-09-19 19:56:15.781 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.792 [info] <0.267.0> Feature flag `quorum_queue`: supported, attempt to enable...
2021-09-19 19:56:15.792 [info] <0.267.0> Feature flag `quorum_queue`: mark as enabled=state_changing
2021-09-19 19:56:15.800 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.800 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-09-19 19:56:15.800 [info] <0.267.0> Feature flags:   [~] quorum_queue
2021-09-19 19:56:15.800 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-09-19 19:56:15.800 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.812 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.813 [info] <0.267.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_queue...
2021-09-19 19:56:15.825 [info] <0.267.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_durable_queue...
2021-09-19 19:56:15.837 [info] <0.267.0> Feature flag `quorum_queue`:   Mnesia tables migration done
2021-09-19 19:56:15.837 [info] <0.267.0> Feature flag `quorum_queue`: mark as enabled=true
2021-09-19 19:56:15.846 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.847 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-09-19 19:56:15.847 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-09-19 19:56:15.847 [info] <0.267.0> Feature flags:   [ ] virtual_host_metadata
2021-09-19 19:56:15.847 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.858 [info] <0.267.0> Feature flag `virtual_host_metadata`: supported, attempt to enable...
2021-09-19 19:56:15.858 [info] <0.267.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing
2021-09-19 19:56:15.866 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.866 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-09-19 19:56:15.867 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-09-19 19:56:15.867 [info] <0.267.0> Feature flags:   [~] virtual_host_metadata
2021-09-19 19:56:15.867 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.877 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.892 [info] <0.267.0> Feature flag `virtual_host_metadata`: mark as enabled=true
2021-09-19 19:56:15.902 [info] <0.267.0> Feature flags: list of feature flags found:
2021-09-19 19:56:15.902 [info] <0.267.0> Feature flags:   [x] implicit_default_bindings
2021-09-19 19:56:15.902 [info] <0.267.0> Feature flags:   [x] quorum_queue
2021-09-19 19:56:15.902 [info] <0.267.0> Feature flags:   [x] virtual_host_metadata
2021-09-19 19:56:15.902 [info] <0.267.0> Feature flags: feature flag states written to disk: yes
2021-09-19 19:56:15.915 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.938 [info] <0.267.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
2021-09-19 19:56:15.938 [info] <0.267.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
2021-09-19 19:56:15.938 [info] <0.267.0> Running boot step database_sync defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step feature_flags defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step codec_correctness_check defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step external_infrastructure defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step rabbit_registry defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step rabbit_queue_location_random defined by app rabbit
2021-09-19 19:56:15.939 [info] <0.267.0> Running boot step rabbit_event defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
2021-09-19 19:56:15.940 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_priority_queue defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Priority queues enabled, real BQ is rabbit_variable_queue
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step kernel_ready defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_sysmon_minder defined by app rabbit
2021-09-19 19:56:15.941 [info] <0.267.0> Running boot step rabbit_epmd_monitor defined by app rabbit
2021-09-19 19:56:15.945 [info] <0.503.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
2021-09-19 19:56:15.945 [info] <0.267.0> Running boot step guid_generator defined by app rabbit
2021-09-19 19:56:15.948 [info] <0.267.0> Running boot step rabbit_node_monitor defined by app rabbit
2021-09-19 19:56:15.948 [info] <0.509.0> Starting rabbit_node_monitor
2021-09-19 19:56:15.948 [info] <0.267.0> Running boot step delegate_sup defined by app rabbit
2021-09-19 19:56:15.949 [info] <0.267.0> Running boot step rabbit_memory_monitor defined by app rabbit
2021-09-19 19:56:15.949 [info] <0.267.0> Running boot step core_initialized defined by app rabbit
2021-09-19 19:56:15.949 [info] <0.267.0> Running boot step upgrade_queues defined by app rabbit
2021-09-19 19:56:15.967 [info] <0.267.0> message_store upgrades: 1 to apply
2021-09-19 19:56:15.967 [info] <0.267.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
2021-09-19 19:56:15.967 [info] <0.267.0> message_store upgrades: No durable queues found. Skipping message store migration
2021-09-19 19:56:15.967 [info] <0.267.0> message_store upgrades: Removing the old message store data
2021-09-19 19:56:15.970 [info] <0.267.0> message_store upgrades: All upgrades applied successfully
2021-09-19 19:56:15.992 [info] <0.267.0> Running boot step rabbit_connection_tracking defined by app rabbit
2021-09-19 19:56:15.993 [info] <0.267.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
2021-09-19 19:56:15.993 [info] <0.267.0> Running boot step rabbit_exchange_parameters defined by app rabbit
2021-09-19 19:56:15.994 [info] <0.267.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
2021-09-19 19:56:15.994 [info] <0.267.0> Running boot step rabbit_policies defined by app rabbit
2021-09-19 19:56:15.995 [info] <0.267.0> Running boot step rabbit_policy defined by app rabbit
2021-09-19 19:56:15.995 [info] <0.267.0> Running boot step rabbit_queue_location_validator defined by app rabbit
2021-09-19 19:56:15.995 [info] <0.267.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
2021-09-19 19:56:15.995 [info] <0.267.0> Running boot step rabbit_vhost_limit defined by app rabbit
2021-09-19 19:56:15.996 [info] <0.267.0> Running boot step recovery defined by app rabbit
2021-09-19 19:56:15.997 [info] <0.267.0> Running boot step load_core_definitions defined by app rabbit
2021-09-19 19:56:15.997 [info] <0.267.0> Running boot step empty_db_check defined by app rabbit
2021-09-19 19:56:15.997 [info] <0.267.0> Adding vhost '/' (description: 'Default virtual host')
2021-09-19 19:56:16.016 [info] <0.546.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@e25344c49247/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
2021-09-19 19:56:16.020 [info] <0.546.0> Starting message stores for vhost '/'
2021-09-19 19:56:16.020 [info] <0.550.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
2021-09-19 19:56:16.021 [info] <0.546.0> Started message store of type transient for vhost '/'
2021-09-19 19:56:16.021 [info] <0.553.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
2021-09-19 19:56:16.022 [warning] <0.553.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
2021-09-19 19:56:16.023 [info] <0.546.0> Started message store of type persistent for vhost '/'
2021-09-19 19:56:16.024 [info] <0.267.0> Creating user 'guest'
2021-09-19 19:56:16.027 [info] <0.267.0> Setting user tags for user 'guest' to [administrator]
2021-09-19 19:56:16.031 [info] <0.267.0> Setting permissions for 'guest' in '/' to '.*', '.*', '.*'
2021-09-19 19:56:16.033 [info] <0.267.0> Running boot step rabbit_looking_glass defined by app rabbit
2021-09-19 19:56:16.033 [info] <0.267.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
2021-09-19 19:56:16.033 [info] <0.267.0> Running boot step background_gc defined by app rabbit
2021-09-19 19:56:16.033 [info] <0.267.0> Running boot step connection_tracking defined by app rabbit
2021-09-19 19:56:16.037 [info] <0.267.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@e25344c49247
2021-09-19 19:56:16.041 [info] <0.267.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@e25344c49247
2021-09-19 19:56:16.041 [info] <0.267.0> Running boot step routing_ready defined by app rabbit
2021-09-19 19:56:16.041 [info] <0.267.0> Running boot step pre_flight defined by app rabbit
2021-09-19 19:56:16.041 [info] <0.267.0> Running boot step notify_cluster defined by app rabbit
2021-09-19 19:56:16.041 [info] <0.267.0> Running boot step networking defined by app rabbit
2021-09-19 19:56:16.043 [info] <0.599.0> started TCP listener on [::]:5672
2021-09-19 19:56:16.043 [info] <0.267.0> Running boot step cluster_name defined by app rabbit
2021-09-19 19:56:16.044 [info] <0.267.0> Running boot step direct_client defined by app rabbit
2021-09-19 19:56:16.044 [notice] <0.104.0> Changed loghwm of /var/log/rabbitmq/rabbit@e25344c49247.log to 50
2021-09-19 19:56:16.309 [info] <0.8.0> Server startup complete; 0 plugins started.
2021-09-19 19:58:12.736 [info] <0.732.0> RabbitMQ is asked to stop...
2021-09-19 19:58:12.824 [info] <0.732.0> Stopping RabbitMQ applications and their dependencies in the following order:
    rabbit
    mnesia
    rabbit_common
    sysmon_handler
    os_mon
    ra
2021-09-19 19:58:12.824 [info] <0.732.0> Stopping application 'rabbit'
2021-09-19 19:58:12.824 [info] <0.267.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping unregistration.
2021-09-19 19:58:12.824 [info] <0.599.0> stopped TCP listener on [::]:5672
2021-09-19 19:58:12.825 [info] <0.530.0> Closing all connections in vhost '/' on node 'rabbit@e25344c49247' because the vhost is stopping
2021-09-19 19:58:12.825 [info] <0.553.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@e25344c49247/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent'
2021-09-19 19:58:12.829 [info] <0.553.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@e25344c49247/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent' is stopped
2021-09-19 19:58:12.830 [info] <0.550.0> Stopping message store for directory '/var/lib/rabbitmq/mnesia/rabbit@e25344c49247/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient'
2021-09-19 19:58:12.833 [info] <0.550.0> Message store for directory '/var/lib/rabbitmq/mnesia/rabbit@e25344c49247/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L/msg_store_transient' is stopped
2021-09-19 19:58:12.837 [info] <0.43.0> Application rabbit exited with reason: stopped
2021-09-19 19:58:12.837 [info] <0.732.0> Stopping application 'mnesia'
2021-09-19 19:58:12.840 [info] <0.732.0> Stopping application 'rabbit_common'
2021-09-19 19:58:12.840 [info] <0.43.0> Application mnesia exited with reason: stopped
2021-09-19 19:58:12.840 [info] <0.43.0> Application rabbit_common exited with reason: stopped
2021-09-19 19:58:12.840 [info] <0.732.0> Stopping application 'sysmon_handler'
2021-09-19 19:58:12.842 [info] <0.43.0> Application sysmon_handler exited with reason: stopped
2021-09-19 19:58:12.842 [info] <0.732.0> Stopping application 'os_mon'
2021-09-19 19:58:12.843 [info] <0.43.0> Application os_mon exited with reason: stopped
2021-09-19 19:58:12.843 [info] <0.732.0> Stopping application 'ra'
2021-09-19 19:58:12.846 [info] <0.43.0> Application ra exited with reason: stopped
2021-09-19 19:58:12.846 [info] <0.732.0> Successfully stopped RabbitMQ and its dependencies
2021-09-19 19:58:12.846 [info] <0.732.0> Halting Erlang VM with the following applications:
    aten
    jsx
    inets
    xmerl
    ranch
    ssl
    public_key
    asn1
    gen_batch_server
    tools
    observer_cli
    stdout_formatter
    recon
    credentials_obfuscation
    crypto
    lager
    goldrush
    compiler
    syntax_tools
    sasl
    stdlib
    kernel

hose logs dont seem to be in my mounted logs volume - if i go to the directory that is mounted for logs I have the following directories:

I think I've already said that but you should remove ALL mounts from docker run command, and check only that situation for now. Beacuse incorrect permissions on mounted dirs can cause any kind of strange behavior

2021-09-19 19:56:16.309 [info] <0.8.0> Server startup complete; 0 plugins started.
2021-09-19 19:58:12.736 [info] <0.732.0> RabbitMQ is asked to stop...

I don't like those lines So for some reason rabbit is started successfully and in 4 seconds it started to shutdown

Still I have no idea why this happened and still I need an exact docker run command which cause same behavior on my pc

To sum up, @ShockwaveNN you think this is a problem with the docker configuration? Where do you think I should look? Or do you know where I could get some help?

you think this is a problem with the docker configuration?

I think, but not sure

Where do you think I should look?

You need to be able to reproduce this error on any server. You can try to create new server, for example Ubuntu in VirtualBox and docker run -itd -p 80:80 onlyoffice/documentserver - make sure this will work fine (I think it will) and after that compare all env variables and docker service configuration between them

Or do you know where I could get some help?

I do not know, because I think we still cannot understand that's the reason - our product or your docker configuration or something else

@RT-Tap maybe it is relevant to know what is similar between our systems since you are also experiencing this issue.

@FHFS OS: Linux 5.14.2-arch1-2 x86_64 (Arch Linux- up to date with rolling release)

Docker:
Client: Version: 20.10.8 API version: 1.41 Go version: go1.16.6 Git commit: 3967b7d28e Built: Wed Aug 4 10:59:01 2021 OS/Arch: linux/amd64 Context: default Experimental: true Server: Engine: Version: 20.10.8 API version: 1.41 (minimum version 1.12) Go version: go1.16.6 Git commit: 75249d88bc Built: Wed Aug 4 10:58:48 2021 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.5.5 GitCommit: 72cec4be58a9eb6b2910f5d10f1c01ca47d231c0.m runc: Version: 1.0.2 GitCommit: v1.0.2-0-g52b36a2d docker-init: Version: 0.19.0 GitCommit: de40ad0

Hardware: description: CPU product: Intel(R) Core(TM)2 Duo CPU T6400 @ 2.00GHz vendor: Intel Corp. physical id: 12 bus info: cpu@0 version: Intel(R) Core(TM)2 Duo CPU T6400 @ 2.00GHz slot: CPU size: 1197MHz capacity: 2GHz width: 64 bits clock: 800MHz description: System Memory physical id: 16 slot: System board or motherboard size: 4GiB

@ShockwaveNN so can we bypass this issue by maybe running an instance of rabbitmq in a separate container and use that instance with document server? Kind of how some services (like your/onlyoffice's mail service - have not tried yet but plan to eventually) allow you to supply mysql credentials and point it at the mysql database? I don't have any experience with rabbitmq and only very minimal knowledge of what it does/is used for so not sure if this would even be possible or worth the effort.

Yes, you can

Here we got description of AMQP_URI and AMQP_TYPE parameters and in this compose file there is example of it's usage

@ShockwaveNN I ran the compose file you mentioned, alot more output. The only thing I changed is the ports. No change however, I am still unable to wget localhost:8080/welcome

Memory and CPU usage seem normal now! Less than 50% of my memory is used (compared to 100% running the image) and no erl_child_setup running at 100% CPU.

       │ File: docker-compose.yml
   1   │ version: '2'
   2   │ services:
   3   │   onlyoffice-documentserver:
   4   │     build:
   5   │       context: .
   6   │     container_name: onlyoffice-documentserver
   7   │     depends_on:
   8   │       - onlyoffice-postgresql
   9   │       - onlyoffice-rabbitmq
  10   │     environment:
  11   │       - DB_TYPE=postgres
  12   │       - DB_HOST=onlyoffice-postgresql
  13   │       - DB_PORT=5432
  14   │       - DB_NAME=onlyoffice
  15   │       - DB_USER=onlyoffice
  16   │       - AMQP_URI=amqp://guest:guest@onlyoffice-rabbitmq
  17   │       # Uncomment strings below to enable the JSON Web Token validation.
  18   │       #- JWT_ENABLED=true
  19   │       #- JWT_SECRET=secret
  20   │       #- JWT_HEADER=Authorization
  21   │       #- JWT_IN_BODY=true
  22   │     ports:
  23 ~ │       - '8080:80'
  24 ~ │       - '8443:443'
  25   │     stdin_open: true
  26   │     restart: always
  27   │     stop_grace_period: 60s
  28   │     volumes:
  29   │        - /var/www/onlyoffice/Data
  30   │        - /var/log/onlyoffice
  31   │        - /var/lib/onlyoffice/documentserver/App_Data/cache/files
  32   │        - /var/www/onlyoffice/documentserver-example/public/files
  33   │        - /usr/share/fonts
  34   │
  35   │   onlyoffice-rabbitmq:
  36   │     container_name: onlyoffice-rabbitmq
  37   │     image: rabbitmq
  38   │     restart: always
  39   │     expose:
  40   │       - '5672'
  41   │
  42   │   onlyoffice-postgresql:
  43   │     container_name: onlyoffice-postgresql
  44   │     image: postgres:9.5
  45   │     environment:
  46   │       - POSTGRES_DB=onlyoffice
  47   │       - POSTGRES_USER=onlyoffice
  48   │       - POSTGRES_HOST_AUTH_METHOD=trust
  49   │     restart: always
  50   │     expose:
  51   │       - '5432'
  52   │     volumes:
  53   │       - postgresql_data:/var/lib/postgresql
  54   │
  55   │ volumes:
  56   │   postgresql_data:

Output:

 fhfs@FHFServ  ~/Docker-DocumentServer   master ±  docker-compose up
[+] Building 724.3s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 32B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                    0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 210B                                                                                  0.0s
 => CACHED [1/5] FROM docker.io/library/ubuntu:20.04                                                               0.0s
 => [2/5] RUN echo "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d &&     apt-get -y update &&     apt-get -yq inst  532.1s
 => [3/5] COPY config /app/ds/setup/config/                                                                        0.1s
 => [4/5] COPY run-document-server.sh /app/ds/run-document-server.sh                                               0.1s
 => [5/5] RUN wget -q -P /tmp "http://download.onlyoffice.com/install/documentserver/linux/onlyoffice-documents  139.9s
 => exporting to image                                                                                            52.0s
 => => exporting layers                                                                                           51.9s
 => => writing image sha256:616c3b595913191a1294e1bb01ae49572fcd274964167b42ad7b9342402b70e6                       0.0s
 => => naming to docker.io/library/docker-documentserver_onlyoffice-documentserver                                 0.0s
[+] Running 4/4
 ⠿ Volume "docker-documentserver_postgresql_data"  Created                                                         0.0s
 ⠿ Container onlyoffice-postgresql                 Created                                                         0.2s
 ⠿ Container onlyoffice-rabbitmq                   Created                                                         0.2s
 ⠿ Container onlyoffice-documentserver             Created                                                         0.6s
Attaching to onlyoffice-documentserver, onlyoffice-postgresql, onlyoffice-rabbitmq
onlyoffice-postgresql      | ********************************************************************************
onlyoffice-postgresql      | WARNING: POSTGRES_HOST_AUTH_METHOD has been set to "trust". This will allow
onlyoffice-postgresql      |          anyone with access to the Postgres port to access your database without
onlyoffice-postgresql      |          a password, even if POSTGRES_PASSWORD is set. See PostgreSQL
onlyoffice-postgresql      |          documentation about "trust":
onlyoffice-postgresql      |          https://www.postgresql.org/docs/current/auth-trust.html
onlyoffice-postgresql      |          In Docker's default configuration, this is effectively any other
onlyoffice-postgresql      |          container on the same system.
onlyoffice-postgresql      |
onlyoffice-postgresql      |          It is not recommended to use POSTGRES_HOST_AUTH_METHOD=trust. Replace
onlyoffice-postgresql      |          it with "-e POSTGRES_PASSWORD=password" instead to set a password in
onlyoffice-postgresql      |          "docker run".
onlyoffice-postgresql      | ********************************************************************************
onlyoffice-postgresql      | The files belonging to this database system will be owned by user "postgres".
onlyoffice-postgresql      | This user must also own the server process.
onlyoffice-postgresql      |
onlyoffice-postgresql      | The database cluster will be initialized with locale "en_US.utf8".
onlyoffice-postgresql      | The default database encoding has accordingly been set to "UTF8".
onlyoffice-postgresql      | The default text search configuration will be set to "english".
onlyoffice-postgresql      |
onlyoffice-postgresql      | Data page checksums are disabled.
onlyoffice-postgresql      |
onlyoffice-postgresql      | fixing permissions on existing directory /var/lib/postgresql/data ... ok
onlyoffice-postgresql      | creating subdirectories ... ok
onlyoffice-postgresql      | selecting default max_connections ... 100
onlyoffice-postgresql      | selecting default shared_buffers ... 128MB
onlyoffice-postgresql      | selecting default timezone ... Etc/UTC
onlyoffice-postgresql      | selecting dynamic shared memory implementation ... posix
onlyoffice-postgresql      | creating configuration files ... ok
onlyoffice-postgresql      | creating template1 database in /var/lib/postgresql/data/base/1 ... ok
onlyoffice-postgresql      | initializing pg_authid ... ok
onlyoffice-postgresql      | setting password ... ok
onlyoffice-postgresql      | initializing dependencies ... ok
onlyoffice-postgresql      | creating system views ... ok
onlyoffice-postgresql      | loading system objects' descriptions ... ok
onlyoffice-postgresql      | creating collations ... ok
onlyoffice-postgresql      | creating conversions ... ok
onlyoffice-postgresql      | creating dictionaries ... ok
onlyoffice-postgresql      | setting privileges on built-in objects ... ok
onlyoffice-postgresql      | creating information schema ... ok
onlyoffice-postgresql      | loading PL/pgSQL server-side language ... ok
onlyoffice-postgresql      | vacuuming database template1 ... ok
onlyoffice-postgresql      | copying template1 to template0 ... ok
onlyoffice-postgresql      | copying template1 to postgres ... ok
onlyoffice-documentserver  | sed: -e expression #1, char 25: unknown option to `s'
onlyoffice-postgresql      | syncing data to disk ... ok
onlyoffice-postgresql      |
onlyoffice-postgresql      | Success. You can now start the database server using:
onlyoffice-postgresql      |
onlyoffice-postgresql      |     pg_ctl -D /var/lib/postgresql/data -l logfile start
onlyoffice-postgresql      |
onlyoffice-postgresql      |
onlyoffice-postgresql      | WARNING: enabling "trust" authentication for local connections
onlyoffice-postgresql      | You can change this by editing pg_hba.conf or using the option -A, or
onlyoffice-postgresql      | --auth-local and --auth-host, the next time you run initdb.
onlyoffice-postgresql      | waiting for server to start....LOG:  database system was shut down at 2021-10-08 12:26:39 UTC
onlyoffice-postgresql      | LOG:  MultiXact member wraparound protections are now enabled
onlyoffice-postgresql      | LOG:  autovacuum launcher started
onlyoffice-postgresql      | LOG:  database system is ready to accept connections
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-postgresql host on port 5432
onlyoffice-postgresql      |  done
onlyoffice-postgresql      | server started
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-postgresql host on port 5432
onlyoffice-postgresql      | CREATE DATABASE
onlyoffice-postgresql      |
onlyoffice-postgresql      |
onlyoffice-postgresql      | /usr/local/bin/docker-entrypoint.sh: ignoring /docker-entrypoint-initdb.d/*
onlyoffice-postgresql      |
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-postgresql host on port 5432
onlyoffice-postgresql      | LOG:  received fast shutdown request
onlyoffice-postgresql      | LOG:  aborting any active transactions
onlyoffice-postgresql      | LOG:  autovacuum launcher shutting down
onlyoffice-postgresql      | LOG:  shutting down
onlyoffice-postgresql      | waiting for server to shut down....LOG:  database system is shut down
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-postgresql host on port 5432
onlyoffice-postgresql      |  done
onlyoffice-postgresql      | server stopped
onlyoffice-postgresql      |
onlyoffice-postgresql      | PostgreSQL init process complete; ready for start up.
onlyoffice-postgresql      |
onlyoffice-postgresql      | LOG:  database system was shut down at 2021-10-08 12:26:43 UTC
onlyoffice-postgresql      | LOG:  MultiXact member wraparound protections are now enabled
onlyoffice-postgresql      | LOG:  autovacuum launcher started
onlyoffice-postgresql      | LOG:  database system is ready to accept connections
onlyoffice-postgresql      | LOG:  incomplete startup packet
onlyoffice-postgresql      | LOG:  incomplete startup packet
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.561786+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578311+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578362+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578393+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578412+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578507+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578526+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:51.578547+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.025698+00:00 [noti] <0.44.0> Application syslog exited with reason: stopped
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.025812+00:00 [noti] <0.222.0> Logging: switching to configured handler(s); following messages may not be visible in this log output
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.055092+00:00 [noti] <0.222.0> Logging: configured log handlers are now ACTIVE
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.313423+00:00 [info] <0.222.0> ra: starting system quorum_queues
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.313517+00:00 [info] <0.222.0> starting Ra system: quorum_queues in directory: /var/lib/rabbitmq/mnesia/rabbit@6990c2668b94/quorum/rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.432006+00:00 [info] <0.259.0> ra: meta data store initialised for system quorum_queues. 0 record(s) recovered
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.457251+00:00 [noti] <0.264.0> WAL: ra_log_wal init, open tbls: ra_log_open_mem_tables, closed tbls: ra_log_closed_mem_tables
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.478118+00:00 [info] <0.222.0> ra: starting system coordination
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.478175+00:00 [info] <0.222.0> starting Ra system: coordination in directory: /var/lib/rabbitmq/mnesia/rabbit@6990c2668b94/coordination/rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.482789+00:00 [info] <0.272.0> ra: meta data store initialised for system coordination. 0 record(s) recovered
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.483207+00:00 [noti] <0.277.0> WAL: ra_coordination_log_wal init, open tbls: ra_coordination_log_open_mem_tables, closed tbls: ra_coordination_log_closed_mem_tables
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.495462+00:00 [info] <0.222.0>
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.495462+00:00 [info] <0.222.0>  Starting RabbitMQ 3.9.7 on Erlang 24.1.2 [jit]
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.495462+00:00 [info] <0.222.0>  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.495462+00:00 [info] <0.222.0>  Licensed under the MPL 2.0. Website: https://rabbitmq.com
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   ##  ##      RabbitMQ 3.9.7
onlyoffice-rabbitmq        |   ##  ##
onlyoffice-rabbitmq        |   ##########  Copyright (c) 2007-2021 VMware, Inc. or its affiliates.
onlyoffice-rabbitmq        |   ######  ##
onlyoffice-rabbitmq        |   ##########  Licensed under the MPL 2.0. Website: https://rabbitmq.com
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   Erlang:      24.1.2 [jit]
onlyoffice-rabbitmq        |   TLS Library: OpenSSL - OpenSSL 1.1.1l  24 Aug 2021
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   Doc guides:  https://rabbitmq.com/documentation.html
onlyoffice-rabbitmq        |   Support:     https://rabbitmq.com/contact.html
onlyoffice-rabbitmq        |   Tutorials:   https://rabbitmq.com/getstarted.html
onlyoffice-rabbitmq        |   Monitoring:  https://rabbitmq.com/monitoring.html
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   Logs: /var/log/rabbitmq/rabbit@6990c2668b94_upgrade.log
onlyoffice-rabbitmq        |         <stdout>
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   Config file(s): /etc/rabbitmq/conf.d/10-default-guest-user.conf
onlyoffice-rabbitmq        |                   /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
onlyoffice-rabbitmq        |
onlyoffice-rabbitmq        |   Starting broker...2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  node           : rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  home dir       : /var/lib/rabbitmq
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  config file(s) : /etc/rabbitmq/conf.d/10-default-guest-user.conf
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>                 : /etc/rabbitmq/conf.d/management_agent.disable_metrics_collector.conf
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  cookie hash    : UuEQEfT+mxTXWFP9bDH7JQ==
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  log(s)         : /var/log/rabbitmq/rabbit@6990c2668b94_upgrade.log
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>                 : <stdout>
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.502713+00:00 [info] <0.222.0>  database dir   : /var/lib/rabbitmq/mnesia/rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.732838+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.732903+00:00 [info] <0.222.0> Feature flags:   [ ] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.732974+00:00 [info] <0.222.0> Feature flags:   [ ] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733009+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733029+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733101+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733133+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733155+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733182+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:52.733290+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.320031+00:00 [info] <0.222.0> Running boot step pre_boot defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.320135+00:00 [info] <0.222.0> Running boot step rabbit_global_counters defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.320455+00:00 [info] <0.222.0> Running boot step rabbit_osiris_metrics defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.320646+00:00 [info] <0.222.0> Running boot step rabbit_core_metrics defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.321389+00:00 [info] <0.222.0> Running boot step rabbit_alarm defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.350213+00:00 [info] <0.289.0> Memory high watermark set to 1568 MiB (1644219596 bytes) of 3920 MiB (4110548992 bytes) total
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.367122+00:00 [info] <0.291.0> Enabling free disk space monitoring
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.367199+00:00 [info] <0.291.0> Disk free limit set to 50MB
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.371493+00:00 [info] <0.222.0> Running boot step code_server_cache defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.371653+00:00 [info] <0.222.0> Running boot step file_handle_cache defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.372058+00:00 [info] <0.294.0> Limiting to approx 1073741719 file handles (966367545 sockets)
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.372232+00:00 [info] <0.295.0> FHC read buffering: OFF
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.372275+00:00 [info] <0.295.0> FHC write buffering: ON
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.374285+00:00 [info] <0.222.0> Running boot step worker_pool defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.374387+00:00 [info] <0.279.0> Will use 4 processes for default worker pool
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.374419+00:00 [info] <0.279.0> Starting worker pool 'worker_pool' with 4 processes in it
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.374956+00:00 [info] <0.222.0> Running boot step database defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.375638+00:00 [info] <0.222.0> Node database directory at /var/lib/rabbitmq/mnesia/rabbit@6990c2668b94 is empty. Assuming we need to join an existing cluster or initialise from scratch...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.375701+00:00 [info] <0.222.0> Configured peer discovery backend: rabbit_peer_discovery_classic_config
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.375732+00:00 [info] <0.222.0> Will try to lock with peer discovery backend rabbit_peer_discovery_classic_config
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.375822+00:00 [info] <0.222.0> All discovered existing cluster peers:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.375846+00:00 [info] <0.222.0> Discovered no peer nodes to cluster with. Some discovery backends can filter nodes out based on a readiness criteria. Enabling debug logging might help troubleshoot.
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.379550+00:00 [noti] <0.44.0> Application mnesia exited with reason: stopped
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.616035+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.616292+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.635853+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.636044+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.636113+00:00 [info] <0.222.0> Feature flag `drop_unroutable_metric`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.636257+00:00 [info] <0.222.0> Feature flag `drop_unroutable_metric`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637315+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637396+00:00 [info] <0.222.0> Feature flags:   [~] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637433+00:00 [info] <0.222.0> Feature flags:   [ ] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637455+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637539+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637564+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637581+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637637+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637698+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.637721+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.648129+00:00 [info] <0.222.0> Feature flag `drop_unroutable_metric`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650320+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650433+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650476+00:00 [info] <0.222.0> Feature flags:   [ ] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650500+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650537+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650568+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650626+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650646+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650663+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.650722+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.661087+00:00 [info] <0.222.0> Feature flag `empty_basic_get_metric`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.661305+00:00 [info] <0.222.0> Feature flag `empty_basic_get_metric`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663291+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663455+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663493+00:00 [info] <0.222.0> Feature flags:   [~] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663522+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663652+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663717+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663746+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663771+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663826+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.663915+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.676512+00:00 [info] <0.222.0> Feature flag `empty_basic_get_metric`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680651+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680710+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680768+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680804+00:00 [info] <0.222.0> Feature flags:   [ ] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680825+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680843+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680914+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680932+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.680948+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.681002+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.691586+00:00 [info] <0.222.0> Feature flag `implicit_default_bindings`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.691772+00:00 [info] <0.222.0> Feature flag `implicit_default_bindings`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.692908+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.692976+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693018+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693037+00:00 [info] <0.222.0> Feature flags:   [~] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693073+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693095+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693113+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693131+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693172+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.693201+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.704350+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 0 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.704579+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.704702+00:00 [info] <0.222.0> Feature flag `implicit_default_bindings`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708300+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708383+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708430+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708451+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708483+00:00 [info] <0.222.0> Feature flags:   [ ] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708555+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708598+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708677+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708751+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.708840+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.721872+00:00 [info] <0.222.0> Feature flag `maintenance_mode_status`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.722118+00:00 [info] <0.222.0> Feature flag `maintenance_mode_status`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724028+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724126+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724165+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724187+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724229+00:00 [info] <0.222.0> Feature flags:   [~] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724266+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724283+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724302+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724498+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.724597+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.736475+00:00 [info] <0.222.0> Creating table rabbit_node_maintenance_states for feature flag `maintenance_mode_status`
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.748213+00:00 [info] <0.222.0> Feature flag `maintenance_mode_status`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751694+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751788+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751819+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751839+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751870+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.751959+00:00 [info] <0.222.0> Feature flags:   [ ] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.752054+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.752087+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.752126+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.752217+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.763676+00:00 [info] <0.222.0> Feature flag `quorum_queue`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.763798+00:00 [info] <0.222.0> Feature flag `quorum_queue`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765478+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765535+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765560+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765593+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765709+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765749+00:00 [info] <0.222.0> Feature flags:   [~] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765785+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765810+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765910+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.765935+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.775995+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.776233+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.776292+00:00 [info] <0.222.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_queue...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.818105+00:00 [info] <0.222.0> Feature flag `quorum_queue`:   migrating Mnesia table rabbit_durable_queue...
onlyoffice-documentserver  | Waiting for connection to the onlyoffice-rabbitmq host on port 5672
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.854838+00:00 [info] <0.222.0> Feature flag `quorum_queue`:   Mnesia tables migration done
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.854936+00:00 [info] <0.222.0> Feature flag `quorum_queue`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.857967+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858030+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858083+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858140+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858178+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858263+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858289+00:00 [info] <0.222.0> Feature flags:   [ ] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858307+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858323+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.858390+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.869369+00:00 [info] <0.222.0> Feature flag `stream_queue`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.869544+00:00 [info] <0.222.0> Feature flag `stream_queue`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871133+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871251+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871320+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871358+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871423+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871464+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871496+00:00 [info] <0.222.0> Feature flags:   [~] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871530+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871574+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.871653+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.882656+00:00 [info] <0.222.0> Feature flag `stream_queue`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.886837+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.886923+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.886984+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887011+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887032+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887051+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887135+00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887154+00:00 [info] <0.222.0> Feature flags:   [ ] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887185+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.887232+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.897953+00:00 [info] <0.222.0> Feature flag `user_limits`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.898147+00:00 [info] <0.222.0> Feature flag `user_limits`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.900157+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.900241+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.900284+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.900388+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901236+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901285+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901312+00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901338+00:00 [info] <0.222.0> Feature flags:   [~] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901369+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.901421+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.911793+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.911986+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.954999+00:00 [info] <0.222.0> Feature flag `user_limits`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.957983+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958031+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958077+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958100+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958119+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958137+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958179+00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958197+00:00 [info] <0.222.0> Feature flags:   [x] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958213+00:00 [info] <0.222.0> Feature flags:   [ ] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.958276+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.969334+00:00 [info] <0.222.0> Feature flag `virtual_host_metadata`: supported, attempt to enable...
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.969482+00:00 [info] <0.222.0> Feature flag `virtual_host_metadata`: mark as enabled=state_changing
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.970924+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.970975+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971000+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971068+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971105+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971133+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971180+00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971202+00:00 [info] <0.222.0> Feature flags:   [x] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971240+00:00 [info] <0.222.0> Feature flags:   [~] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.971259+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.981565+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:56.981820+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.020429+00:00 [info] <0.222.0> Feature flag `virtual_host_metadata`: mark as enabled=true
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023374+00:00 [info] <0.222.0> Feature flags: list of feature flags found:
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023447+00:00 [info] <0.222.0> Feature flags:   [x] drop_unroutable_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023520+00:00 [info] <0.222.0> Feature flags:   [x] empty_basic_get_metric
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023550+00:00 [info] <0.222.0> Feature flags:   [x] implicit_default_bindings
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023577+00:00 [info] <0.222.0> Feature flags:   [x] maintenance_mode_status
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023603+00:00 [info] <0.222.0> Feature flags:   [x] quorum_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023628+00:00 [info] <0.222.0> Feature flags:   [x] stream_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023653+00:00 [info] <0.222.0> Feature flags:   [x] user_limits
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023690+00:00 [info] <0.222.0> Feature flags:   [x] virtual_host_metadata
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.023729+00:00 [info] <0.222.0> Feature flags: feature flag states written to disk: yes
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.033070+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.033256+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058087+00:00 [info] <0.222.0> Waiting for Mnesia tables for 30000 ms, 9 retries left
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058254+00:00 [info] <0.222.0> Successfully synced tables from a peer
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058286+00:00 [info] <0.222.0> Peer discovery backend rabbit_peer_discovery_classic_config does not support registration, skipping registration.
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058329+00:00 [info] <0.222.0> Will try to unlock with peer discovery backend rabbit_peer_discovery_classic_config
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058439+00:00 [info] <0.222.0> Running boot step database_sync defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.058823+00:00 [info] <0.222.0> Running boot step feature_flags defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059077+00:00 [info] <0.222.0> Running boot step codec_correctness_check defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059170+00:00 [info] <0.222.0> Running boot step external_infrastructure defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059245+00:00 [info] <0.222.0> Running boot step rabbit_registry defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059471+00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_cr_demo defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059672+00:00 [info] <0.222.0> Running boot step rabbit_queue_location_random defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059755+00:00 [info] <0.222.0> Running boot step rabbit_event defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059916+00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_amqplain defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.059971+00:00 [info] <0.222.0> Running boot step rabbit_auth_mechanism_plain defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060014+00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_direct defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060055+00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_fanout defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060095+00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_headers defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060161+00:00 [info] <0.222.0> Running boot step rabbit_exchange_type_topic defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060226+00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_all defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060278+00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_exactly defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060325+00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_mode_nodes defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060382+00:00 [info] <0.222.0> Running boot step rabbit_priority_queue defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.060410+00:00 [info] <0.222.0> Priority queues enabled, real BQ is rabbit_variable_queue
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.061052+00:00 [info] <0.222.0> Running boot step rabbit_queue_location_client_local defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.061491+00:00 [info] <0.222.0> Running boot step rabbit_queue_location_min_masters defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.061608+00:00 [info] <0.222.0> Running boot step kernel_ready defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.061709+00:00 [info] <0.222.0> Running boot step rabbit_sysmon_minder defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.062101+00:00 [info] <0.222.0> Running boot step rabbit_epmd_monitor defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.063026+00:00 [info] <0.557.0> epmd monitor knows us, inter-node communication (distribution) port: 25672
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.063256+00:00 [info] <0.222.0> Running boot step guid_generator defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.072127+00:00 [info] <0.222.0> Running boot step rabbit_node_monitor defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.072545+00:00 [info] <0.561.0> Starting rabbit_node_monitor
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.072735+00:00 [info] <0.222.0> Running boot step delegate_sup defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.075248+00:00 [info] <0.222.0> Running boot step rabbit_memory_monitor defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.075642+00:00 [info] <0.222.0> Running boot step core_initialized defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.075699+00:00 [info] <0.222.0> Running boot step upgrade_queues defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.084909+00:00 [info] <0.222.0> message_store upgrades: 1 to apply
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.085072+00:00 [info] <0.222.0> message_store upgrades: Applying rabbit_variable_queue:move_messages_to_vhost_store
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.085706+00:00 [info] <0.222.0> message_store upgrades: No durable queues found. Skipping message store migration
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.085784+00:00 [info] <0.222.0> message_store upgrades: Removing the old message store data
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.086601+00:00 [info] <0.222.0> message_store upgrades: All upgrades applied successfully
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.103073+00:00 [info] <0.222.0> Running boot step channel_tracking defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.112173+00:00 [info] <0.222.0> Setting up a table for channel tracking on this node: tracked_channel_on_node_rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.120985+00:00 [info] <0.222.0> Setting up a table for channel tracking on this node: tracked_channel_table_per_user_on_node_rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.121323+00:00 [info] <0.222.0> Running boot step rabbit_channel_tracking_handler defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.121416+00:00 [info] <0.222.0> Running boot step connection_tracking defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.130477+00:00 [info] <0.222.0> Setting up a table for connection tracking on this node: tracked_connection_on_node_rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.140156+00:00 [info] <0.222.0> Setting up a table for per-vhost connection counting on this node: tracked_connection_per_vhost_on_node_rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.148931+00:00 [info] <0.222.0> Setting up a table for per-user connection counting on this node: tracked_connection_table_per_user_on_node_rabbit@6990c2668b94
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.149341+00:00 [info] <0.222.0> Running boot step rabbit_connection_tracking_handler defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.149447+00:00 [info] <0.222.0> Running boot step rabbit_exchange_parameters defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.149738+00:00 [info] <0.222.0> Running boot step rabbit_mirror_queue_misc defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.150048+00:00 [info] <0.222.0> Running boot step rabbit_policies defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.150620+00:00 [info] <0.222.0> Running boot step rabbit_policy defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.150717+00:00 [info] <0.222.0> Running boot step rabbit_queue_location_validator defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.150815+00:00 [info] <0.222.0> Running boot step rabbit_quorum_memory_manager defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.150901+00:00 [info] <0.222.0> Running boot step rabbit_stream_coordinator defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.151135+00:00 [info] <0.222.0> Running boot step rabbit_vhost_limit defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.151286+00:00 [info] <0.222.0> Running boot step rabbit_mgmt_db_handler defined by app rabbitmq_management_agent
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.151347+00:00 [info] <0.222.0> Management plugin: using rates mode 'basic'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.152136+00:00 [info] <0.222.0> Running boot step recovery defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.153335+00:00 [info] <0.222.0> Running boot step empty_db_check defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.153406+00:00 [info] <0.222.0> Will seed default virtual host and user...
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.153533+00:00 [info] <0.222.0> Adding vhost '/' (description: 'Default virtual host', tags: [])
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.220419+00:00 [info] <0.622.0> Making sure data directory '/var/lib/rabbitmq/mnesia/rabbit@6990c2668b94/msg_stores/vhosts/628WB79CIFDYO9LJI6DKMI09L' for vhost '/' exists
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.221931+00:00 [info] <0.622.0> Setting segment_entry_count for vhost '/' with 0 queues to '2048'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.230636+00:00 [info] <0.622.0> Starting message stores for vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.231005+00:00 [info] <0.626.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_transient": using rabbit_msg_store_ets_index to provide index
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.232409+00:00 [info] <0.622.0> Started message store of type transient for vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.232710+00:00 [info] <0.630.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": using rabbit_msg_store_ets_index to provide index
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.233819+00:00 [warn] <0.630.0> Message store "628WB79CIFDYO9LJI6DKMI09L/msg_store_persistent": rebuilding indices from scratch
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.236362+00:00 [info] <0.622.0> Started message store of type persistent for vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.249338+00:00 [info] <0.222.0> Created user 'guest'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.256654+00:00 [info] <0.222.0> Successfully set user tags for user 'guest' to [administrator]
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.263635+00:00 [info] <0.222.0> Successfully set permissions for 'guest' in virtual host '/' to '.*', '.*', '.*'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.263748+00:00 [info] <0.222.0> Running boot step rabbit_looking_glass defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.263782+00:00 [info] <0.222.0> Running boot step rabbit_core_metrics_gc defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.263990+00:00 [info] <0.222.0> Running boot step background_gc defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264170+00:00 [info] <0.222.0> Running boot step routing_ready defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264219+00:00 [info] <0.222.0> Running boot step pre_flight defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264245+00:00 [info] <0.222.0> Running boot step notify_cluster defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264279+00:00 [info] <0.222.0> Running boot step networking defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264342+00:00 [info] <0.222.0> Running boot step definition_import_worker_pool defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.264403+00:00 [info] <0.279.0> Starting worker pool 'definition_import_pool' with 4 processes in it
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.265216+00:00 [info] <0.222.0> Running boot step cluster_name defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.265343+00:00 [info] <0.222.0> Initialising internal cluster ID to 'rabbitmq-cluster-id-lfuBUO8AOUI9EazooHcpww'
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.272658+00:00 [info] <0.222.0> Running boot step direct_client defined by app rabbit
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.272903+00:00 [info] <0.663.0> Resetting node maintenance status
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.290651+00:00 [info] <0.690.0> Prometheus metrics: HTTP (non-TLS) listener started on port 15692
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.290987+00:00 [info] <0.663.0> Ready to start client connection listeners
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.293239+00:00 [info] <0.734.0> started TCP listener on [::]:5672
onlyoffice-rabbitmq        |  completed with 3 plugins.
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.376975+00:00 [info] <0.663.0> Server startup complete; 3 plugins started.
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.376975+00:00 [info] <0.663.0>  * rabbitmq_prometheus
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.376975+00:00 [info] <0.663.0>  * rabbitmq_web_dispatch
onlyoffice-rabbitmq        | 2021-10-08 12:26:57.376975+00:00 [info] <0.663.0>  * rabbitmq_management_agent
onlyoffice-documentserver  | psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:4: NOTICE:  table "doc_callbacks" does not exist, skipping
onlyoffice-documentserver  | psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:9: NOTICE:  function merge_db(pg_catalog.varchar,int2,int8,pg_catalog.timestamp,pg_catalog.varchar,int8,int8) does not exist, skipping
onlyoffice-documentserver  | psql:/var/www/onlyoffice/documentserver/server/schema/postgresql/removetbl.sql:10: NOTICE:  function merge_db(pg_catalog.varchar,int2,int4,pg_catalog.timestamp,pg_catalog.varchar,int4,int4) does not exist, skipping
onlyoffice-documentserver  | Starting supervisor: supervisord.
onlyoffice-documentserver  |  * Starting periodic command scheduler cron
onlyoffice-documentserver  |    ...done.
onlyoffice-documentserver  |  * Starting nginx nginx
onlyoffice-documentserver  |    ...done.
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.007525+00:00 [info] <0.743.0> accepting AMQP connection <0.743.0> (172.19.0.4:54652 -> 172.19.0.2:5672)
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.012830+00:00 [info] <0.743.0> connection <0.743.0> (172.19.0.4:54652 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.042941+00:00 [info] <0.767.0> accepting AMQP connection <0.767.0> (172.19.0.4:54654 -> 172.19.0.2:5672)
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.046854+00:00 [info] <0.771.0> accepting AMQP connection <0.771.0> (172.19.0.4:54656 -> 172.19.0.2:5672)
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.048429+00:00 [info] <0.767.0> connection <0.767.0> (172.19.0.4:54654 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:27:02.050884+00:00 [info] <0.771.0> connection <0.771.0> (172.19.0.4:54656 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-documentserver  | Generating AllFonts.js, please wait...Done
onlyoffice-documentserver  | Generating presentation themes, please wait...Done
onlyoffice-rabbitmq        | 2021-10-08 12:28:03.248703+00:00 [warn] <0.771.0> closing AMQP connection <0.771.0> (172.19.0.4:54656 -> 172.19.0.2:5672, vhost: '/', user: 'guest'):
onlyoffice-rabbitmq        | 2021-10-08 12:28:03.248703+00:00 [warn] <0.771.0> client unexpectedly closed TCP connectiononlyoffice-rabbitmq        | 2021-10-08 12:28:03.249243+00:00 [warn] <0.743.0> closing AMQP connection <0.743.0> (172.19.0.4:54652 -> 172.19.0.2:5672, vhost: '/', user: 'guest'):
onlyoffice-rabbitmq        | 2021-10-08 12:28:03.249243+00:00 [warn] <0.743.0> client unexpectedly closed TCP connectiononlyoffice-documentserver  | ds:docservice: stopped
onlyoffice-documentserver  | ds:docservice: started
onlyoffice-rabbitmq        | 2021-10-08 12:28:04.508005+00:00 [warn] <0.767.0> closing AMQP connection <0.767.0> (172.19.0.4:54654 -> 172.19.0.2:5672, vhost: '/', user: 'guest'):
onlyoffice-rabbitmq        | 2021-10-08 12:28:04.508005+00:00 [warn] <0.767.0> client unexpectedly closed TCP connectiononlyoffice-rabbitmq        | 2021-10-08 12:28:04.684613+00:00 [info] <0.906.0> accepting AMQP connection <0.906.0> (172.19.0.4:54658 -> 172.19.0.2:5672)
onlyoffice-rabbitmq        | 2021-10-08 12:28:04.689064+00:00 [info] <0.906.0> connection <0.906.0> (172.19.0.4:54658 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:28:04.701368+00:00 [info] <0.930.0> accepting AMQP connection <0.930.0> (172.19.0.4:54660 -> 172.19.0.2:5672)
onlyoffice-rabbitmq        | 2021-10-08 12:28:04.703719+00:00 [info] <0.930.0> connection <0.930.0> (172.19.0.4:54660 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-rabbitmq        | 2021-10-08 12:28:05.915953+00:00 [info] <0.965.0> accepting AMQP connection <0.965.0> (172.19.0.4:54662 -> 172.19.0.2:5672)
onlyoffice-documentserver  | ds:converter: stopped
onlyoffice-documentserver  | ds:converter: started
onlyoffice-rabbitmq        | 2021-10-08 12:28:05.922599+00:00 [info] <0.965.0> connection <0.965.0> (172.19.0.4:54662 -> 172.19.0.2:5672): user 'guest' authenticated and granted access to vhost '/'
onlyoffice-documentserver  |  * Reloading nginx configuration nginx
onlyoffice-documentserver  |    ...done.
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/converter/err.log <==
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/converter/out.log <==
onlyoffice-documentserver  | [2021-10-08T12:27:00.548] [WARN] nodeJS - update cluster with 1 workers
onlyoffice-documentserver  | [2021-10-08T12:27:00.565] [WARN] nodeJS - worker 298 started.
onlyoffice-documentserver  | [2021-10-08T12:27:00.570] [WARN] nodeJS - update cluster with 1 workers
onlyoffice-documentserver  | [2021-10-08T12:28:04.901] [WARN] nodeJS - update cluster with 1 workers
onlyoffice-documentserver  | [2021-10-08T12:28:04.913] [WARN] nodeJS - worker 361 started.
onlyoffice-documentserver  | [2021-10-08T12:28:04.916] [WARN] nodeJS - update cluster with 1 workers
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/docservice/err.log <==
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/docservice/out.log <==
onlyoffice-documentserver  | [2021-10-08T12:27:01.986] [WARN] nodeJS - Express server starting...
onlyoffice-documentserver  | [2021-10-08T12:27:01.993] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
onlyoffice-documentserver  | [2021-10-08T12:27:02.161] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode. Version: 6.4.1. Build: 45
onlyoffice-documentserver  | [2021-10-08T12:28:04.663] [WARN] nodeJS - Express server starting...
onlyoffice-documentserver  | [2021-10-08T12:28:04.668] [WARN] nodeJS - Failed to subscribe to plugin folder updates. When changing the list of plugins, you must restart the server. https://nodejs.org/docs/latest/api/fs.html#fs_availability
onlyoffice-documentserver  | [2021-10-08T12:28:04.728] [WARN] nodeJS - Express server listening on port 8000 in production-linux mode. Version: 6.4.1. Build: 45
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/metrics/err.log <==
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/metrics/out.log <==
onlyoffice-documentserver  | 8 Oct 12:27:00 - [255] reading config file: ./config/config.js
onlyoffice-documentserver  | 8 Oct 12:27:00 - server is up INFO
onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/nginx.error.log <==

onlyoffice-documentserver  |
onlyoffice-documentserver  | ==> /var/log/onlyoffice/documentserver/metrics/out.log <==
onlyoffice-documentserver  | Flushing stats at  Fri Oct 08 2021 12:37:00 GMT+0000 (Coordinated Universal Time)
onlyoffice-documentserver  | { counters:
onlyoffice-documentserver  |    { 'statsd.bad_lines_seen': 0,
onlyoffice-documentserver  |      'statsd.packets_received': 0,
onlyoffice-documentserver  |      'statsd.metrics_received': 0 },
onlyoffice-documentserver  |   timers: {},
onlyoffice-documentserver  |   gauges: {},
onlyoffice-documentserver  |   timer_data: {},
onlyoffice-documentserver  |   counter_rates:
onlyoffice-documentserver  |    { 'statsd.bad_lines_seen': 0,
onlyoffice-documentserver  |      'statsd.packets_received': 0,
onlyoffice-documentserver  |      'statsd.metrics_received': 0 },
onlyoffice-documentserver  |   sets: {},
onlyoffice-documentserver  |   pctThreshold: [ 90 ] }

Please note that it's just an example of compose file, do not use it directly without understanding it

For example I don't think mapping https port to 8443 is a good idea

Also as a side note - I see that your docker-compose output looks like:

[+] Building 724.3s (10/10) FINISHED
 => [internal] load build definition from Dockerfile                                                               0.0s
 => => transferring dockerfile: 32B                                                                                0.0s
 => [internal] load .dockerignore                                                                                  0.0s
 => => transferring context: 2B                                                                                    0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                    0.0s
 => [internal] load build context                                                                                  0.0s
 => => transferring context: 210B                                                                                  0.0s
 => CACHED [1/5] FROM docker.io/library/ubuntu:20.04

It's quite different to what I've used to?

Are you enabled BuildKit? I've never used it but maybe it's the reason of original errors?

I've checked 8443 on my server, it is unused. But I will change it.

I've never heard of buildkit, I did not enable it.

I've never heard of buildkit, I did not enable it.

Maybe it's enabled by default on your os, could you check if it's enabled (do not how, but I think simple google is enough to answer it)

@ShockwaveNN It is not enabled. but the plugin is there:

docker info:

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Build with BuildKit (Docker Inc., v0.6.1-docker)
  compose: Docker Compose (Docker Inc., 2.0.1)

Server:
 Containers: 7
  Running: 4
  Paused: 0
  Stopped: 3
 Images: 10
 Server Version: 20.10.9
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: false
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 8686ededfc90076914c5238eb96c883ea093a8ba.m
 runc version: v1.0.2-0-g52b36a2d
 init version: de40ad0
 Security Options:
  seccomp
   Profile: default
  cgroupns
 Kernel Version: 5.14.9-arch2-1
 Operating System: Arch Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 3.828GiB
 Name: FHFServ
 ID: AMCU:BYND:UCDJ:MLRH:NOW6:4NHC:4KGT:IUK5:7WFF:44VM:MQLX:JFT3
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

So both of you using arch Linux? Maybe it's base source of our problems

I'll try to found out Arch installation or at least Manjaro and see if problem will be actual for me too

Ok, i've setup my arch instance and installed all updated - and I can confirm

On the simplest docker run -it -p 80:80 onlyoffice/documentserver command I see
```
sed: -e expression #1, char 25: unknown option to `s'
```
And at the same time on my Kubuntu main machine with same docker image - there is no such error I need some time to figure out the reason
Even if this error happened - everything works fine

I've created issue 53046 in our private issue tracker about

sed: -e expression #1, char 25: unknown option to `s'

It's definitely actual only for Arch docker install (do not know how), but still I don't think it's the main reason of your problem - I see those error in my setup but it still working fine

So, I've been playing around with this issue and I find it strange that when documentserver is being built it starts a rabbitmq server instance. Why? Is it not already another container? Why is an instance being started inside the document server? Excuse my ignorance I don't have much experience with docker but I found it rather redundant and unnecessary?

Alright well I spent my entire weekend trying to figure this out and I'm going to give up but I'll share what i've found and hopefully one of you can carry the torch.

I've had better luck using the following Dockerfile

FROM rabbitmq:latest 
###########################################################
## Allow rabbitmq to be treated as a service
## Rabbitmq image redirects logs to tty but we dont want that so we'll
##  change it with enviroment varible although this probably isn't necessary
###########################################################
COPY rabbitmq-server /etc/init.d/
RUN chmod +x /etc/init.d/rabbitmq-server && update-rc.d rabbitmq-server defaults 
ENV RABBITMQ_LOGS=/var/log/rabbitmq

###########################################################
## This get's rid of Perl complaining about using UTF-8
###########################################################
RUN apt-get -y update && apt-get -y upgrade && DEBIAN_FRONTEND=noninteractive apt-get install -y locales && \
    sed -i -e 's/# en_US.UTF-8 UTF-8/en_US.UTF-8 UTF-8/' /etc/locale.gen && \
    dpkg-reconfigure --frontend=noninteractive locales && update-locale LANG=en_US.UTF-8

ENV LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_ALL=en_US.UTF-8 DEBIAN_FRONTEND=noninteractive PG_VERSION=12

ARG ONLYOFFICE_VALUE=onlyoffice

###########################################################
## not much is changed here besides breaking it up to allow more accurate
## debug and removal of installing rabbitmq as it;s already installed as well as
## stopping postgresql gracefully so it doesn't currupt and complain in the next 
## layer
###########################################################
RUN echo "#!/bin/sh\nexit 0" > /usr/sbin/policy-rc.d && \
    apt-get -y update && \
    apt-get -yq install wget apt-transport-https gnupg locales && \
    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 0x8320ca65cb2de8e5 && \
    locale-gen en_US.UTF-8 && \
    echo ttf-mscorefonts-installer msttcorefonts/accepted-mscorefonts-eula select true | debconf-set-selections && \
    apt-get -yq install \
        adduser \
        apt-utils \
        bomstrip \
        certbot \
        curl \
        gconf-service \
        htop \
        libasound2 \
        libboost-regex-dev \
        libcairo2 \
        libcurl3-gnutls \
        libcurl4 \
        libgtk-3-0 \
        libnspr4 \
        libnss3 \
        libstdc++6 \
        libxml2 \
        libxss1 \
        libxtst6 \
        mysql-client \
        nano \
        net-tools \
        netcat-openbsd \
        nginx-extras \
        postgresql \
        postgresql-client \
        pwgen \
        redis-server \
        software-properties-common \
        sudo \
        supervisor \
        ttf-mscorefonts-installer \
        xvfb \
        zlib1g && \
        service postgresql stop
###########################################################
## this environment variable isn't neccessary but if postgresql doesn't shut down
## properly it needs to recover and this gives it a little more time although it's more likely it 
## will have to be manually restarted
###########################################################
ENV PGCTLTIMEOUT=180 
RUN if [  $(ls -l /usr/share/fonts/truetype/msttcorefonts | wc -l) -ne 61 ]; \
        then echo 'msttcorefonts failed to download'; exit 1; fi  && \
    echo "SERVER_ADDITIONAL_ERL_ARGS=\"+S 1:1\"" | tee -a /etc/rabbitmq/rabbitmq-env.conf && \
    sed -i "s/bind .*/bind 127.0.0.1/g" /etc/redis/redis.conf && \
    sed 's|\(application\/zip.*\)|\1\n    application\/wasm wasm;|' -i /etc/nginx/mime.types && \
    pg_conftool $PG_VERSION main set listen_addresses 'localhost' && \
    service postgresql start && \
    sudo -u postgres psql -c "CREATE DATABASE $ONLYOFFICE_VALUE;" && \
    sudo -u postgres psql -c "CREATE USER $ONLYOFFICE_VALUE WITH password '$ONLYOFFICE_VALUE';" && \
    sudo -u postgres psql -c "GRANT ALL privileges ON DATABASE $ONLYOFFICE_VALUE TO $ONLYOFFICE_VALUE;" && \ 
    service postgresql stop && \
    service redis-server stop && \
    service rabbitmq-server stop && \
    service supervisor stop && \
    service nginx stop && \
    rm -rf /var/lib/apt/lists/*

COPY config /app/ds/setup/config/
COPY run-document-server.sh /app/ds/run-document-server.sh
# RUN chmod 755 /app/ds/run-document-server.sh

EXPOSE 80 443

ARG COMPANY_NAME=onlyoffice
ARG PRODUCT_NAME=documentserver
ARG PACKAGE_URL="http://download.onlyoffice.com/install/documentserver/linux/${COMPANY_NAME}-${PRODUCT_NAME}_amd64.deb"

ENV COMPANY_NAME=$COMPANY_NAME \
    PRODUCT_NAME=$PRODUCT_NAME

RUN wget -q -P /tmp "$PACKAGE_URL" && \
    apt-get -y update && \
    service postgresql start && \
    apt-get -yq install /tmp/$(basename "$PACKAGE_URL") && \
    service postgresql stop && \
    service supervisor stop && \
    chmod 755 /app/ds/*.sh && \
    rm -f /tmp/$(basename "$PACKAGE_URL") && \
    rm -rf /var/log/$COMPANY_NAME && \
    rm -rf /var/lib/apt/lists/*

VOLUME /var/log/$COMPANY_NAME /var/lib/$COMPANY_NAME /var/www/$COMPANY_NAME/Data /var/lib/postgresql /var/lib/rabbitmq /var/lib/redis /usr/share/fonts/truetype/custom

ENTRYPOINT ["/app/ds/run-document-server.sh"]

You'll need rabbitmq-server initt.d script :

#!/bin/sh
### BEGIN INIT INFO
# Provides:          rabbitmq-server
# Required-Start:    $remote_fs $network
# Required-Stop:     $remote_fs $network
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Description:       RabbitMQ broker
# Short-Description: Enable AMQP service provided by RabbitMQ broker
### END INIT INFO

#PATH=/sbin:/usr/sbin:/bin:/usr/bin
PATH=/sbin:/usr/sbin:/bin:/usr/bin:/opt/rabbitmq/sbin
NAME=rabbitmq-server
DAEMON=/usr/sbin/${NAME}
#CONTROL=/usr/sbin/rabbitmqctl
CONTROL=/opt/rabbitmq/sbin/rabbitmqctl
DESC="RabbitMQ Messaging Server"
USER=rabbitmq
ROTATE_SUFFIX=
INIT_LOG_DIR=/var/log/rabbitmq
PID_FILE=/var/run/rabbitmq/pid
RABBITMQ_STARTUP_TIMEOUT=600

test -x $DAEMON || exit 0
test -x $CONTROL || exit 0

RETVAL=0
set -e

[ -f /etc/default/${NAME} ] && . /etc/default/${NAME}

. /lib/lsb/init-functions
. /lib/init/vars.sh

ensure_pid_dir () {
    PID_DIR=`dirname ${PID_FILE}`
    if [ ! -d ${PID_DIR} ] ; then
        mkdir -p ${PID_DIR}
        chown -R ${USER}:${USER} ${PID_DIR}
        chmod 755 ${PID_DIR}
    fi
}

remove_pid () {
    rm -f ${PID_FILE}
    rmdir `dirname ${PID_FILE}` || :
}

start_rabbitmq () {
    status_rabbitmq quiet
    if [ $RETVAL != 0 ] ; then
        RETVAL=0
        ensure_pid_dir
        set +e
        RABBITMQ_PID_FILE=$PID_FILE start-stop-daemon --quiet \
            --chuid rabbitmq --start --exec $DAEMON \
            --pidfile "$RABBITMQ_PID_FILE" --background
        $CONTROL wait --timeout $RABBITMQ_STARTUP_TIMEOUT $PID_FILE >/dev/null 2>&1
        RETVAL=$?
        set -e
        if [ $RETVAL != 0 ] ; then
            remove_pid
        fi
    else
        RETVAL=3
    fi
}

stop_rabbitmq () {
    status_rabbitmq quiet
    if [ $RETVAL = 0 ] ; then
        set +e
        $CONTROL stop ${PID_FILE} > ${INIT_LOG_DIR}/shutdown_log 2> ${INIT_LOG_DIR}/shutdown_err
        RETVAL=$?
        set -e
        if [ $RETVAL = 0 ] ; then
            remove_pid
        fi
    else
        RETVAL=3
    fi
}

status_rabbitmq() {
    set +e
    if [ "$1" != "quiet" ] ; then
        $CONTROL status 2>&1
    else
        $CONTROL status > /dev/null 2>&1
    fi
    if [ $? != 0 ] ; then
        RETVAL=3
    fi
    set -e
}

rotate_logs_rabbitmq() {
    set +e
    $CONTROL -q rotate_logs ${ROTATE_SUFFIX}
    if [ $? != 0 ] ; then
        RETVAL=1
    fi
    set -e
}

restart_running_rabbitmq () {
    status_rabbitmq quiet
    if [ $RETVAL = 0 ] ; then
        restart_rabbitmq
    else
        log_warning_msg "${DESC} not running"
    fi
}

restart_rabbitmq() {
    stop_rabbitmq
    start_rabbitmq
}

restart_end() {
    if [ $RETVAL = 0 ] ; then
        log_end_msg 0
    else
        log_end_msg 1
    fi
}

start_stop_end() {
    case "$RETVAL" in
        0)
            [ -x /sbin/initctl ] && /sbin/initctl emit --no-wait "${NAME}-${1}"
            log_end_msg 0
            ;;
        3)
            log_warning_msg "${DESC} already ${1}"
            log_end_msg 0
            RETVAL=0
            ;;
        *)
            log_warning_msg "FAILED - check ${INIT_LOG_DIR}/startup_\{log, _err\}"
            log_end_msg 1
            ;;
    esac
}

case "$1" in
    start)
        log_daemon_msg "Starting ${DESC}" $NAME
        start_rabbitmq
        start_stop_end "running"
        ;;
    stop)
        log_daemon_msg "Stopping ${DESC}" $NAME
        stop_rabbitmq
        start_stop_end "stopped"
        ;;
    status)
        status_rabbitmq
        ;;
    rotate-logs)
        log_action_begin_msg "Rotating log files for ${DESC}: ${NAME}"
        rotate_logs_rabbitmq
        log_action_end_msg $RETVAL
        ;;
    force-reload|reload|restart)
        log_daemon_msg "Restarting ${DESC}" $NAME
        restart_rabbitmq
        restart_end
        ;;
    try-restart)
        log_daemon_msg "Restarting ${DESC}" $NAME
        restart_running_rabbitmq
        restart_end
        ;;
    *)
        echo "Usage: $0 {start|stop|status|rotate-logs|restart|condrestart|try-restart|reload|force-reload}" >&2
        RETVAL=1
        ;;
esac

exit $RETVAL

the other files like run-document-server.sh and config are same as the one's used in this repo.

Once you build for some reason none of the services are running so just run the follwoing: docker exec -it document_server_container bash

service postgresql start
service supervisor start
service redis-server start
service rabbitmq-server start
service nginx start

At this point you'll be able to access the site using localhost:port or what have you You'll run the two commands to start a test and when you go create a new document for the test it will look like it's almost there but then get a connection error so you'll check logs to find /var/log/onlyoffice/documentserver/docservice/out.log

    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
[2021-10-11T02:25:26.056] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
root@6680c1927426:/var/log/onlyoffice/documentserver/docservice# tail out.log
[2021-10-11T02:32:41.068] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
    at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)
[2021-10-11T02:32:42.070] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672

/var/log/onlyoffice/documentserver/nginx.error.log

2021/10/11 02:26:38 [error] 908#908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.151, server: , request: "GET /6.4.1-45/doc/192.168.1.151http___192.168.1.162_8080_example_files_192.168.1.151_new.docx1633919197518/c/info?t=1633919197971 HTTP/1.1", upstream: "http://127.0.0.1:8000/doc/192.168.1.151http___192.168.1.162_8080_example_files_192.168.1.151_new.docx1633919197518/c/info", host: "192.168.1.162:8080", referrer: "http://192.168.1.162:8080/6.4.1-45/web-apps/apps/documenteditor/main/index.html?_dc=6.4.1-45&lang=en&customer=ONLYOFFICE&frameEditorId=iframeEditor&parentOrigin=http://192.168.1.162:8080"
2021/10/11 02:26:38 [error] 908#908: *1 connect() failed (111: Connection refused) while connecting to upstream, client: 192.168.1.151, server: , request: "GET /6.4.1-45/doc/192.168.1.151http___192.168.1.162_8080_example_files_192.168.1.151_new.docx1633919197518/c/info?t=1633919197971 HTTP/1.1", upstream: "http://127.0.0.1:8000/doc/192.168.1.151http___192.168.1.162_8080_example_files_192.168.1.151_new.docx1633919197518/c/info", host: "192.168.1.162:8080", referrer: "http://192.168.1.162:8080/6.4.1-45/web-apps/apps/documenteditor/main/index.html?_dc=6.4.1-45&lang=en&customer=ONLYOFFICE&frameEditorId=iframeEditor&parentOrigin=http://192.168.1.162:8080"

at which point you'll notice this problem isn't new because it seems to be the same issue as the one here : https://github.com/ONLYOFFICE/DocumentServer/issues/1045 I may give it another crack but I've spent more time than I'd like to admit trying to figure this out, so like I said hopefully one of you can carry the torch and figure this out.

Also you most likely can use the Dockerfile in this repo and just manually go into the container and start the services (that's one of the main issues- why aren't these services starting automatically? they are setup to ) but I've been using this one and haven't checked the default one just sharing what I've been using to figure this out.

I find it strange that when documentserver is being built it starts a rabbitmq server instance

We knot what it's not proper docker way, but it was done for user's convenience - to have only one docker run command for starting fully configured DocumentServer

at which point you'll notice this problem isn't new because it seems to be the same issue as the one here : ONLYOFFICE/DocumentServer#1045

I think it may be - generally speaking it's just a default error if nodejs code cannot connect to rabbitmq for any reason (network problems or simple typos) As far as I remember - we never figure out real reason in that case and I was not able to reproduce this error on my server

I'm sorry but I still do not know that to do - only way is to be able to repdocue this error on my end, but as I said earlier - I setup arch linux with latest updates and it run fine DocumentServer in docker

I am using the snap version of documentserver now. it works, not going to spend more time figuring this out.

ONLYOFFICE / Docker-DocumentServer

Unable to start rabbitmq #386