RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.61k stars 10.62k forks source link

Memory increases abnormally when uploading file. #29164

Open kevvvinyao opened 1 year ago

kevvvinyao commented 1 year ago

Description:

When I upload files in the general channel, memory increases to 90%+

Steps to reproduce:

  1. deploy rocket chat via docker.
  2. remove the limitation of the size of uploading files. set the storage type to FileSystem, whose location is /app/uploads
  3. upload a 2GB file
  4. ram increases uniformly. Actually it is a linear function of time at the early stage of upload 图片

Expected behavior:

I think ram should be more and less than a certian value. Then get stable after completing uploading.

Actual behavior:

Memory increase to more than 90% when I upload an about 2GB size file. Here is a interesting phenomenon I found: If I drag another file to upload it, the memory will drop suddenly to a normal value, then increase uniformly as previous. When I upload a file larger than 2 or 3GB, the ram won't stop increasing until it reaches 100%. I use the command top to check resource occupancy, finding a process named node take almost all memory. When I upload a file about 200mb, the ram will increase a little then get stable. There is no problem when I try uploading files of small size.(less than 100mb

Server Setup Information:

Client Setup Information

Additional context

I used a reverse proxy via nginx. Nginx proxy manager was deployed on another server, which could reverse origin server's ip and port.

Relevant logs:

{"level":50,"time":"2023-05-06T13:54:00.107Z","pid":1,"hostname":"823dc3d9f94f","name":"System","msg":"Exception while invoking method saveSettings","err":{"type":"errorClass","message":"TOTP Required [totp-required]","stack":"Error: TOTP Required [totp-required]\n    at checkCodeForUser (app/2fa/server/code/index.ts:194:9)\n    at MethodInvocation.<anonymous> (app/2fa/server/twoFactorRequired.ts:34:4)\n    at MethodInvocation.methodsMap.<computed> (app/lib/server/lib/debug.js:74:34)\n    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1902:12)\n    at packages/ddp-server/livedata_server.js:1820:15\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at packages/ddp-server/livedata_server.js:1818:36\n    at new Promise (<anonymous>)\n    at Server.applyAsync (packages/ddp-server/livedata_server.js:1817:12)\n    at Server.apply (packages/ddp-server/livedata_server.js:1756:26)\n    at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n    at Object.post (app/api/server/v1/misc.ts:554:27)\n    at app/api/server/api.js:463:96\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n    at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n    at packages/rocketchat_restivus/lib/route.coffee:59:33\n    at packages/simple_json-routes.js:100:9\n => awaited here:\n    at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)\n    at Server.apply (packages/ddp-server/livedata_server.js:1769:22)\n    at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n    at Object.post (app/api/server/v1/misc.ts:554:27)\n    at app/api/server/api.js:463:96\n    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n    at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n    at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n    at packages/rocketchat_restivus/lib/route.coffee:59:33\n    at packages/simple_json-routes.js:100:9","isClientSafe":true,"error":"totp-required","reason":"TOTP Required","details":{"method":"password","codeGenerated":false,"availableMethods":[]},"errorType":"Meteor.Error"},"msg":"TOTP Required [totp-required]"}

{"level":50,"time":"2023-05-06T13:54:00.107Z","pid":1,"hostname":"823dc3d9f94f","name":"System","msg":"Exception while invoking method saveSettings","err":{"type":"errorClass","message":"TOTP Required [totp-required]","stack":"Error: TOTP Required [totp-required]\n at checkCodeForUser (app/2fa/server/code/index.ts:194:9)\n at MethodInvocation. (app/2fa/server/twoFactorRequired.ts:34:4)\n at MethodInvocation.methodsMap. (app/lib/server/lib/debug.js:74:34)\n at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1902:12)\n at packages/ddp-server/livedata_server.js:1820:15\n at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n at packages/ddp-server/livedata_server.js:1818:36\n at new Promise ()\n at Server.applyAsync (packages/ddp-server/livedata_server.js:1817:12)\n at Server.apply (packages/ddp-server/livedata_server.js:1756:26)\n at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n at Object.post (app/api/server/v1/misc.ts:554:27)\n at app/api/server/api.js:463:96\n at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n at packages/rocketchat_restivus/lib/route.coffee:59:33\n at packages/simple_json-routes.js:100:9\n => awaited here:\n at Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:60:12)\n at Server.apply (packages/ddp-server/livedata_server.js:1769:22)\n at Server.call (packages/ddp-server/livedata_server.js:1738:17)\n at Object.post (app/api/server/v1/misc.ts:554:27)\n at app/api/server/api.js:463:96\n at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1329:12)\n at Object._internalRouteActionHandler [as action] (app/api/server/api.js:463:39)\n at Route._callEndpoint (packages/rocketchat_restivus/lib/route.coffee:150:32)\n at packages/rocketchat_restivus/lib/route.coffee:59:33\n at packages/simple_json-routes.js:100:9","isClientSafe":true,"error":"totp-required","reason":"TOTP Required","details":{"method":"password","codeGenerated":false,"availableMethods":[]},"errorType":"Meteor.Error"},"msg":"TOTP Required [totp-required]"}

jburel-ca commented 11 months ago

Same problem on my side, after X file uploads the memory usage approaches 100% and never goes back down. This seems related to #22513 : the file is downloaded to the memory buffer before being written anywhere (local fs, s3, ...) :(

dzjbet commented 1 month ago

All files are read into the memory at one time without fragment processing.