RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
39.62k stars 10.15k forks source link

Rocket 6.4.2 wont start due to meteor errors #30838

Open Kot-o-pes opened 8 months ago

Kot-o-pes commented 8 months ago

Description:

Hello, i just had very strange bug with rocketchat deployed in k8s in a single pod so ive noticed increase in active handlers image and pod started to fail probes

95s         Warning   Unhealthy      pod/rocketchat-rocketchat-6596d8bc6d-glvrb   Readiness probe failed: Get "http://10.233.78.209:3000/api/info": read tcp 10.160.192.114:50060->10.233.78.209:3000: read: connection reset by peer

ive checked logs and noticed

Errors like this can cause oplog processing errors.
Setting EXIT_UNHANDLEDPROMISEREJECTION will cause the process to exit allowing your service to automatically restart the process
Future node.js versions will automatically exit the process
=================================
=== UnHandledPromiseRejection ===
errorClass [Error]: Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again. [error-too-many-requests]
    at app/api/server/api.ts:332:10
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  isClientSafe: true,
  error: 'error-too-many-requests',
  reason: 'Error, too many requests. Please slow down. You must wait 27 seconds before trying this endpoint again.',
  details: { timeToReset: 26091, seconds: 27 },
  errorType: 'Meteor.Error'
}

ive noticed that mentions method has drastically increased but there was no @all calls in any channel atm image

so ive tried to restart pod but no help with same error then i restarted mongodb instance one by one, after 3-5 minutes requests were back to normal and pod started i dont see any strange behavior of mongodb either

image

except primary was switched but that was after restart of mongodb and its seems to be expected after its stable again i still see error like

^[[BTypeError: Cannot read property 'authRequired' of undefined
    at packages/rocketchat:restivus/lib/route.js:199:16
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at packages/rocketchat:restivus/lib/route.js:155:13
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at packages/rocketchat:restivus/lib/route.js:57:18
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
{"level":40,"time":"2023-11-02T15:27:59.563Z","pid":1,"hostname":"rocketchat-rocketchat-c4999558c-tjppk","name":"DeprecationWarning","section":"API","msg":"The parameter \"query\" in the endpoint \"/api/v1/users.list\" is deprecated and will be removed on version 7.0.0"}
Error: ENOENT: no such file or directory, stat '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/uploadZipFile.ts:11:16
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/processDataDownloads.ts:232:25
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/zipFiles/c0092699-6e89-4bf4-bcc8-71c77a9c4539.zip'

and occasionally get exceptions from server

Error: Can't wait without a fiber
    at Function.wait (/app/bundle/programs/server/node_modules/fibers/future.js:159:9)
    at Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:448:10)
    at MongoConnection.findOne (packages/mongo/mongo_driver.js:829:84)
    at Collection.findOne (packages/mongo/collection.js:459:29)
    at getUserById (packages/accounts-password/password_server.js:5:51)
    at AccountsServer.Accounts.generateResetToken (packages/accounts-password/password_server.js:402:16)
    at AccountsServer.Accounts.sendEnrollmentEmail (packages/accounts-password/password_server.js:567:14)
    at Immediate.<anonymous> (app/lib/server/functions/setUsername.ts:102:14)
    at processImmediate (internal/timers.js:464:21)
    at process.topLevelDomainCallback (domain.js:152:15)
    at process.callbackTrampoline (internal/async_hooks.js:128:24)

ive no idea why this happens, tho i get a strange feeling thats its just another CE awesome new feature, hope someone will help to figure it out, thanks!

ktkd commented 8 months ago

++

McGregor700 commented 8 months ago

++

TLINDEN commented 7 months ago

I am having the same error:

Error: ENOENT: no such file or directory, stat '/tmp/zipFiles/a2d375a7-c9ea-497f-974b-1b679106c526.zip'
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/uploadZipFile.ts:11:16
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at server/lib/dataExport/processDataDownloads.ts:232:25
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40 {
  errno: -2,
  code: 'ENOENT',
  syscall: 'stat',
  path: '/tmp/zipFiles/a2d375a7-c9ea-497f-974b-1b679106c526.zip'
}

but Rocketchat is running and working ok anyway.

However, I'd really like to know what kind of data it wants to upload and to where!