Closed joshuaboniface closed 6 years ago
Number of Running Instances: 2 (file uploads to path /ufs/FileSystem:Uploads directed to first node only)
This could be problematic.
Let's say Instance A is the one that gets this, and Instance B is the one you are connected to via a websocket. If it were to post a file... If that path goes to Instance A it will be proxied server side over to Instance B to handle since that's where you are connected.
On the new beta version of the Rocket.Chat+ app this is no longer the case and it'll upload all files via the rest api endpoint.
Might be worth giving the beta a try, though you will loose your connection info when you upgrade to the beta version. You'll have to re-input it.
But at any rate this is an issue that is/will be fixed on the Android app side. Known issue to have it hang like this on the mobile app. As you can see above... The complexity of how it has to upload makes perfect sense why 😁
Going to go ahead and close this
Thanks @geekgonecrazy that does make sense. Just to clarify though, the problem seems to occur behind load balancing regardless of the force-to-A rule (I had it off for the last few weeks, and added it because of this issue - it works at least 25% of the time with the rule, and 0% of the time without), but I added it back as a test. I'll give the beta app a try though!
I've had it happen with a single instance also. Uploading via the real time api is just painful. :). Please make sure to let the mobile team know what you think. They have been working hard on it 😁
Description:
When uploading a mid-sized (~2-3MB) image file from RocketChat+ on Android, the server throws an error. The upload fails with some sort of "timeout"/"Your connection seems off..."/"Authenticating" loop on the client, and the server goes into a socket error loop.
Server Setup Information:
tar
via Ansible role/ufs/FileSystem:Uploads
directed to first node only)Steps to Reproduce:
Expected behavior:
The upload works.
Actual behavior:
Server throws errors; instance dies for some time (<60s usually) before returning to service, upload invariably fails.
Relevant logs:
systemd-journald
during an error:The last ~16 lines repeat 12+ times in a couple seconds, then the instance eventually stabilizes with no further errors in the logs.
The "Too many open files" prompted me to try
lsof
, but I'm unable to see RocketChat using any excessive number of files./tmp
is a 1GB ramdisk./var/lib/rocket.chat/uploads
is a symbolic link to/srv/rocketchat/uploads
, which is a CephFS filesystem (shared storage).