search

ONLYOFFICE / DocumentServer

ONLYOFFICE Docs is a free collaborative online office suite comprising viewers and editors for texts, spreadsheets and presentations, forms and PDF, fully compatible with Office Open XML formats: .docx, .xlsx, .pptx and enabling collaborative editing in real time.
https://www.onlyoffice.com
GNU Affero General Public License v3.0
4.71k stars 1.07k forks source link

nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672 #1045

Closed zubanst closed 3 years ago

zubanst commented 3 years ago

Here is my configuration: Server1 ubuntu server 18.04 with ownCloud 10.4.1.3 and onlyoffice connector 6.0.1, Server2 ubuntu server 18.04 with onlyoffice document-server 6.0.1 initially installed from the repository described here https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx . and migrated to the latest version. The document server works fine, a test in a browser on the local network with https:/// is redirected tohttps:///welcome/ saying Document server is running. Client side the server is correctly configured, no error saving the configuration. Try open a document in the ownloud client says error - server cannot be connected and results in the following error server side: [2020-10-21T06:25:05.565] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T06:25:05.600] [WARN] nodeJS - worker 4737 started. [2020-10-21T06:25:05.602] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T10:36:30.914] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T10:36:30.925] [WARN] nodeJS - worker 2123 started. [2020-10-21T10:36:30.945] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14) [2020-10-21T10:36:33.136] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14) [2020-10-21T10:54:50.166] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T10:54:50.178] [WARN] nodeJS - worker 3055 started. [2020-10-21T10:54:50.179] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T11:05:04.222] [WARN] nodeJS - update cluster with 1 workers [2020-10-21T11:05:04.236] [WARN] nodeJS - worker 10699 started. [2020-10-21T11:05:04.237] [WARN] nodeJS - update cluster with 1 workers [2020-10-29T12:07:32.453] [WARN] nodeJS - update cluster with 1 workers [2020-10-29T12:07:32.461] [WARN] nodeJS - worker 2091 started. [2020-10-29T12:07:32.471] [WARN] nodeJS - update cluster with 1 workers [2020-10-29T12:07:33.490] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672 at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14) [2020-10-29T12:10:58.779] [WARN] nodeJS - update cluster with 1 workers [2020-10-29T12:10:58.816] [WARN] nodeJS - worker 13778 started. [2020-10-29T12:10:58.820] [WARN] nodeJS - update cluster with 1 workers

sudo supervisorctl status all ds:converter RUNNING pid 13646, uptime 0:30:13 ds:docservice RUNNING pid 13645, uptime 0:30:13 ds:metrics RUNNING pid 13643, uptime 0:30:13 ds:spellchecker RUNNING pid 13644, uptime 0:30:13

nc -v localhost 5672 Connection to localhost 5672 port [tcp/amqp] succeeded!

I think this is a bug server side.

ShockwaveNN commented 3 years ago

[2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672

This is not right, which version of RabbitMQ are you using?

zubanst commented 3 years ago

I use the one installed from the bundle "RabbitMQ","3.6.10"

ShockwaveNN commented 3 years ago

Version seems OK, we got 3.6.10-1ubuntu0.3 in our default Docker image

Not sure that you mean by from the bundle

Could you show contents of /etc/onlyoffice/documentserver/local.json and /etc/onlyoffice/documentserver/default.json, are change anything?

We got similar error earlier https://github.com/ONLYOFFICE/DocumentServer/issues/674 but never found the reason

zubanst commented 3 years ago

From the bundle, I mean from the repository described in https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx Here are the 2 files. No changes. json.tar.gz

ShockwaveNN commented 3 years ago

Everything seems fine in configs

Do you using some kind firewall? If you using default ubuntu firewall could you check it by ufw disable and restart all services via supervisorctl restart all

Also could you check that 

rabbitmqctl list_connections

show some connections like this in our default Docker image image

zubanst commented 3 years ago

No firewall, and rabbitmqctl list_connections give me this.

Listing connections guest 127.0.0.1 43750 running guest 127.0.0.1 43752 running guest 127.0.0.1 43754 running

Additionally, at shutdown, RabbitMQ messaging server is stuck till timeout before real shutdown.

ShockwaveNN commented 3 years ago

@zubanst Hm, maybe something wrong with RabbitMQ itself? Something interesting at /var/log/rabbitmq/?

zubanst commented 3 years ago

Well, here is the startup log; all seems fine for me:

=INFO REPORT==== 2-Nov-2020::14:13:31 ===
Starting RabbitMQ 3.6.10 on Erlang 20.2.2
Copyright (C) 2007-2017 Pivotal Software, Inc.
Licensed under the MPL.  See http://www.rabbitmq.com/

=INFO REPORT==== 2-Nov-2020::14:13:31 ===
node           : rabbit@onlyoffice
home dir       : /var/lib/rabbitmq
config file(s) : /etc/rabbitmq/rabbitmq.config (not found)
cookie hash    : cuAG0CmupG9TsAAeMlWFgA==
log            : /var/log/rabbitmq/rabbit@onlyoffice.log
sasl log       : /var/log/rabbitmq/rabbit@onlyoffice-sasl.log
database dir   : /var/lib/rabbitmq/mnesia/rabbit@onlyoffice

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Memory limit set to 797MB of 1993MB total.

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Enabling free disk space monitoring

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Disk free limit set to 50MB

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Limiting to approx 65436 file handles (58890 sockets)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
FHC read buffering:  OFF
FHC write buffering: ON

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Waiting for Mnesia tables for 30000 ms, 9 retries left

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Priority queues enabled, real BQ is rabbit_variable_queue

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Starting rabbit_node_monitor

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
msg_store_transient: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
msg_store_persistent: using rabbit_msg_store_ets_index to provide index

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
started TCP Listener on [::]:5672

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
Server startup complete; 0 plugins started.

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
accepting AMQP connection <0.311.0> (127.0.0.1:47548 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
connection <0.311.0> (127.0.0.1:47548 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
accepting AMQP connection <0.332.0> (127.0.0.1:47550 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:32 ===
connection <0.332.0> (127.0.0.1:47550 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'

=INFO REPORT==== 2-Nov-2020::14:13:33 ===
accepting AMQP connection <0.360.0> (127.0.0.1:47552 -> 127.0.0.1:5672)

=INFO REPORT==== 2-Nov-2020::14:13:33 ===
connection <0.360.0> (127.0.0.1:47552 -> 127.0.0.1:5672): user 'guest' authenticated and granted access to vhost '/'
ShockwaveNN commented 3 years ago

Log seems fine

I'm almost out of ideas,

Could you show output of rabbitmqctl list_queues, so there is no strange queues not related to DocumentServer?

image

zubanst commented 3 years ago

Timeout: 60.0 seconds ... Listing queues for vhost / ... name messages amq.gen-1-UQ0iggePbqxCrIfgpTCw 0 ds.convertdead 0 ds.converttask 0 ds.convertresponse 0 ds.delayed 0

I did a fresh install on Ubuntu server 20.04 and updated owncloud client to 6.1.1, same result. It simply doesn't work. Has anybody succeeded to have such a working setup?

ShockwaveNN commented 3 years ago

I did a fresh install on Ubuntu server 20.04 and updated owncloud client to 6.1.1, same result.

This is very strange

Could you describe which exactly version of Ubuntu 20.04 are you using? Do you install it via official image on bare-bone server, or did you use some kind of cloud version from Amazon, Google Cloud, DigitialOcean?

I want to reproduce your problem via myself

zubanst commented 3 years ago

It is a VMware virtual machine. If you want, I can export the virtual machine and share it over on my cloud. I installed the latest Ubuntu distribution https://releases.ubuntu.com/20.04.1/ubuntu-20.04.1-live-server-amd64.iso and followed this guide: https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx

ShockwaveNN commented 3 years ago

Yeah, of you export vm image - this would be great

On Sat, Nov 7, 2020, 11:31 zubanst notifications@github.com wrote:

It is a VMware virtual machine. If you want, I can export the virtual machine and share it over on my cloud. I installed the latest Ubuntu distribution https://releases.ubuntu.com/20.04.1/ubuntu-20.04.1-live-server-amd64.iso and followed this guide: https://helpcenter.onlyoffice.com/installation/docs-community-install-ubuntu.aspx

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ONLYOFFICE/DocumentServer/issues/1045#issuecomment-723418263, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDG3CKR2D3BAQZAIH5LOLSOUAXNANCNFSM4TDVXH3Q .

zubanst commented 3 years ago

Send me an email address so I could share the link.

ShockwaveNN commented 3 years ago

shockwavenn@gmail.com

On Sat, Nov 7, 2020, 19:33 zubanst notifications@github.com wrote:

Send me an email address so I could share the link.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/ONLYOFFICE/DocumentServer/issues/1045#issuecomment-723466213, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAFDG3EMTJPNT4HRPWZ2I5LSOVZFRANCNFSM4TDVXH3Q .

ShockwaveNN commented 3 years ago

@zubanst I've recieved you VM, never have any expirience with VM Sphere, so need some time to figure out how to run it and is it possible to run it using VirtualBox

Post any results here as soon as get any

ShockwaveNN commented 3 years ago

Ok, I was able setup your vm instance using VirtualBox and login, but never able to setup any network, even local, so I got troubles even reproduce your issue

So only idea or advice I have right now - could you setup some cloud provider like AWS or DigitalOcean with default instruction and see if this work (and I'm pretty sure this should work) and compare configs with yours

Or use docker

zubanst commented 3 years ago

I don't use AWS nor DigitalOcean so it is pointless I setup such. I will try a docker, though I doubt though it would work. Again, have you any knowledge of someone having set up a Ubuntu/Debian environment working with Owncloud client?

zubanst commented 3 years ago

I'll convert the machine to virtualbox myself and let you know when it's ready for download.

ShockwaveNN commented 3 years ago

Again, have you any knowledge of someone having set up a Ubuntu/Debian environment working with Owncloud client

I'm not a owncloud guy, but there is a plenty a people have setup (you can take a look here, https://github.com/ONLYOFFICE/onlyoffice-owncloud a lot of closed issues, and I think more have no issues)

But again, I don't think your error have something to do with owncloud, since rabbitmq is vital part of DocumentServer itself

zubanst commented 3 years ago

I figured out what is wrong with your Virtualbox. You need to configure bridged interface for the VM network and your vbox host need to connect to your network infrastructure through wired connection. Wireless wont do.

ShockwaveNN commented 3 years ago

Yeah, I set up network settings like this: image

But still no luck

Maybe something wrong with DNS, I tryed to changed to google via settings up DNS param to 8.8.8.8 in /etc/systemd/resolved.conf, and restarting service systemd-resolved restart but seems this is not enought

ShockwaveNN commented 3 years ago

@zubanst Also my colleague give me some additional info about situation in which this can be happen

By any chance - didn't you try to install some other edition of DocumentServer on this server? I see that onlyoffice-documentserver is installed, but didn't you try to install onlyoffice-documentserver-de before, remove it and install onlyoffice-documentserver after that by any chance?

zubanst commented 3 years ago

No, this was a fresh install strictly following the guide I mentioned, published on the official website. I'm still working to get you a vbox machine with a similar install (I had other emergencies since).

zubanst commented 3 years ago

I just shared a vbox machine with you. Check your email. Same username, same password. No need to install the certificate, I already installed a self signed one.

ShockwaveNN commented 3 years ago

@zubanst Yeah, thanks I've downloaded it, and will take a look as soon as I have time for it, I'll reply here if got any new info

ShockwaveNN commented 3 years ago

Ok, thanks for new version I was able to setup it in VirtualBox

Soo I done those steps

  1. Stopped all services via sudo supervisorctl stop all
  2. Cleared docservice log file via editing it with sudo vim /var/log/onlyoffice/documentserver/docservice/err.log
  3. Started all services via sudo supervisorctl start all
  4. Open test example and create new file got those: image Look at logs in /var/log/onlyoffice/documentserver/docservice/out.log and this seems related to self-signed cert 
    [2020-11-17T08:59:26.690] [ERROR] nodeJS - postData error: docId = 192.168.1.238https___192.168.1.237_example_files_192.168.1.238_new_20_1_.docx1605603566028;url = https://192.168.1.237/example/track?filename=new%20(1).docx&useraddress=192.168.1.238;data = {"key":"192.168.1.238https___192.168.1.237_example_files_192.168.1.238_new_20_1_.docx1605603566028","status":1,"users":["uid-1"],"actions":[{"type":1,"userid":"uid-1"}]}
Error: self signed certificate
at TLSSocket.onConnectSecure (_tls_wrap.js:1058:34)
at TLSSocket.emit (events.js:198:13)
at TLSSocket._finishInit (_tls_wrap.js:636:8)

    Seems this is related to self-signed cert, I'm not very good at that, but I know that rabbitmq error is my interset now, so I temoraly disable self-signed certs via restoring default non-ssl config:

    sudo cp /etc/onlyoffice/documentserver/nginx/ds.conf /etc/onlyoffice/documentserver/nginx/ds.conf.bak.ssl
sudo cp /etc/onlyoffice/documentserver/nginx/ds.conf.bak /etc/onlyoffice/documentserver/nginx/ds.conf
sudo service nginx start

And after that I can open test example via http://192.168.1.237/ and everyhing fine for me image

No errors for connection to rabbitmq or any errors at all at cat /var/log/onlyoffice/documentserver/docservice/out.log

zubanst commented 3 years ago

Unfortunately it cannot be used in the cloud with plain http. How can I do the same test? How to build and where to put this example html so I could use the document server 'offline' without the Onwcloud app in my own LAN?

ShockwaveNN commented 3 years ago

@zubanst To be clear - this issue was created not about problems with https and self-signed certificate, you shared VM for me with problem about RabbitMQ connection

[2020-10-21T10:36:32.131] [ERROR] nodeJS - [AMQP] Error: connect ECONNREFUSED 127.0.0.1:5672
at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1107:14)

To be able to help with this error I should be able to reproduce it, but even with your VM image I cannot see this error.

But problem with self-signed cert it's all other problem, which is not critical for me right now (for example you can use use not self-signed cert, but let's encrypt free cert)

zubanst commented 3 years ago

You sent some screenshots saying the document server I sent you in vbox works. How do I build and where I put this example html so I could use the document server 'offline' and do the same test as you with my VMware document server?

ShockwaveNN commented 3 years ago

@zubanst this test example is included in latest release of DocumentServer, you can start it by executing supervisorctl start all and access by http://local_ds_ip/example or see details of it on http://local_ds_ip/welcome

zubanst commented 3 years ago

Issue solved. Used port redirection and my true website ssl certificate instead of the self-signed one. Should someone be interested by more details, contact me.

sankayop commented 3 years ago

@zubanst : Hi zubanst, I have a slightly different setup (FreeBsd with nextcloud in one jail and oO in another jail). Both have a letsencrypt certificate and I do have the same infinite "Error: connect ECONNREFUSED 127.0.0.1:5672" in "/var/log/onlyoffice/docservice_out.log"

Aside from this issue, I'm still looking for a nginx conf specific for oO because mine looks not completely ok. Last point, while looking at the documentserver/server/welcome/index.html in the browser, I got the nice " Thank you for choosing ONLYOFFICE! Something went wrong during installation" But for these two last issues, it has nothing to do with this topic ;)

So if you would be so kind to give some hints concerning how you solved the error for port 5672, I would be very thankful :)

ShockwaveNN commented 3 years ago

" Thank you for choosing ONLYOFFICE! Something went wrong during installation"

This is correct message for older version of ONLYOFFICE

It's better to create new issue, since original was not on FreeBSD, but please note that we do not support FreeBSD officially, so I'm not sure I can help you

sankayop commented 3 years ago

" Thank you for choosing ONLYOFFICE! Something went wrong during installation"

This is correct message for older version of ONLYOFFICE

It's better to create new issue, since original was not on FreeBSD, but please note that we do not support FreeBSD officially, so I'm not sure I can help you

Hey @ShockwaveNN , thanks for your reply, I'm suprised that you wrote it's the older version of ONLYOFFICE because I did a git clone of the master!? But anyway, I've redone the build and am still struggling with some error (still one to solve) in core/DestopEditor/doctrenderer/memorystream.h/cpp. If necessary, I will open an issue, knowing that FreeBSD is not yet supported by oO. (but if you can push its support, I guess all the BSD, FreeNAS community will be super greatful :) )

Cheers,

zubanst commented 3 years ago

Hello @sankayop First of all, I started with something easier than you: I installed a community version of the server, the one provided by Ubuntu and didn't compile it. Concerning the certificate, I have the following configuration: Server1 - nginx proxy installed with the my SSL certificate; Server2 -my main www server including Owncloud on apache2; Server3 - OnlyOffice server with nginx. The workaround I found is to install the SSL certificate on the OnlyOffice nginx server too, than configure a specific port-mapping on my router to map external port 12346 to internal Server3:443 and configure the owncloud OnlyOffice API with my www:123456. This way the true SSL certificate could be used. Although the server is supposed to work with self-signed certificates in local network configurations, I never succeeded to get it work reliably. A first connection to the server to open a document went fine than another one opened by another user failed. Never figured out if it was due to the server itself or the Owncloud API and support team never figured it neither. With this work-around I am happy.

sankayop commented 3 years ago

Hey @zubanst thanks for your reply, thus, we have pretty similar setup (except I'm on FreeBSD and playing with compilations...) But I still don't get how you got rid of the 127.0.0.1:5672 error. Nevertheless, thanks :)

Btw, I was wondering, for the nginx you installed for your server3, could you show me what did you put in the server block? (of course without your private data's :) ) I'm struggling with the info I should put in it...

ShockwaveNN commented 3 years ago

I'm suprised that you wrote it's the older version of ONLYOFFICE because I did a git clone of the master!?

You not mentioned that you compiled it from source in original message, in that case there should be no messages, just empty page

Personally I have zero expirience in FreeBSD, but this issue https://github.com/ONLYOFFICE/DocumentServer/issues/79 has plenty of comments and some workarounds, so I think it's better to continue your discussion in this issue, since original issue was not on FreeBSD

sankayop commented 3 years ago

I'm suprised that you wrote it's the older version of ONLYOFFICE because I did a git clone of the master!?

You not mentioned that you compiled it from source in original message, in that case there should be no messages, just empty page

Personally I have zero expirience in FreeBSD, but this issue #79 has plenty of comments and some workarounds, so I think it's better to continue your discussion in this issue, since original issue was not on FreeBSD

you're right :) I did a little trick to get this welcome page (i.e. I asked specifically to look at the only index.html I could find in the built files). Ok for not polluting this issue ;)

SushantRathore-Admin commented 2 years ago

@ShockwaveNN ShockwaveNN Hi Team,

I have installed my Onlyoffice on oracle7 Linux. I m able to access the welcome page and started an example .

Captureoofice1

But I m facing issues while viewing the doc and if I m trying to create a new doc that is not opening properly.

Captureonlyoffice2

`/var/log/onlyoffice/documentserver/docservice/out.log

nodeJS - [AMQP] Error: connect EMFILE ::1:5672 - Local (undefined:undefined ) at internalConnect (net.js:923:16) at defaultTriggerAsyncIdScope (internal/async_hooks.js:435:18) at emitLookup (net.js:1066:9) at /snapshot/server/build/server/Common/node_modules/dnscache/lib/index.js:125:28 at /snapshot/server/build/server/Common/node_modules/dnscache/lib/cache.js:116:13 at RawTask.call (/snapshot/server/build/server/Common/node_modules/asap/asap.js:40:19) at flush (/snapshot/server/build/server/Common/node_modules/asap/raw.js:50:29) at processTicksAndRejections (internal/process/task_queues.js:77:11) at runNextTicks (internal/process/task_queues.js:64:3) at listOnTimeout (internal/timers.js:526:9) at processTimers (internal/timers.js:500:7)`

ShockwaveNN commented 2 years ago

Hi, first of all, I think your issue is separate from OP

So I've never had any experience with oracle7 Linux. So maybe this problem is specific to that os

Seems that EMFILE error is then too many files opened https://stackoverflow.com/questions/8965606/node-and-error-emfile-too-many-open-files Maybe some default limits on oracle Linux forbid that

How did you install our product? Could you provide instruction Did you check Docker version - is the problem the same

If nothing helped you - please create a new issue

SushantRathore-Admin commented 2 years ago

Hi, first of all, I think your issue is separate from OP

So I've never had any experience with oracle7 Linux. So maybe this problem is specific to that os

Seems that EMFILE error is then too many files opened https://stackoverflow.com/questions/8965606/node-and-error-emfile-too-many-open-files Maybe some default limits on oracle Linux forbid that

How did you install our product? Could you provide instruction Did you check Docker version - is the problem the same

If nothing helped you - please create a new issue

Hi i m trying to fix it but now facing redis connection issues Installing PostgreSQL database... OK Configuring redis access... Host: localhost

Trying to establish redis connection... /bin/documentserver-configure.sh: connect: Connection refused /bin/documentserver-configure.sh: line 286: FD: Connection refused FAILURE

ShockwaveNN commented 2 years ago

As I said - create a new issue and answer all my questions for me able to help you

Your errors looks like maybe some strict rules prohibit to connect some services to another

flintstoney commented 1 month ago

Issue solved. Used port redirection and my true website ssl certificate instead of the self-signed one. Should someone be interested by more details, contact me.

Hello. Tell me please how you solved this issue? I have a similar problem! Thank you!