App Framework suddenly throws TypeError that makes sending messages impossible

[pehlert]

Description:

Suddenly our Rocket.Chat instance stopped working and returned TypeError responses to users. We had updated to 3.0.2 about 4 days earlier and it was working stable since then. We were not able to identify any event that led to this behaviour. A restart did not fix it.

As it seems to be related to the App Framework, we tried to disable our only extension "Giphy" at first. This did not solved the issue, except the user saw the Type Error responses directly now and not as a reply from rocket.cat.

Eventually disabling the App Framework altogether in the "General" tab under "Apps" fixed the issue for us.

We love RocketChat and are happy to investigate further. Thanks!

Expected behavior:

Messages are sent

Actual behavior:

The user gets a Type Error back from rocket.cat:

Server Setup Information:

• Version of Rocket.Chat Server: 3.0.2
• Operating System: Docker / Kubernetes
• Deployment Method: Docker / Kubernetes
• Number of Running Instances: 1
• DB Replicaset Oplog: ?
• NodeJS Version: -
• MongoDB Version: 3.6

Client Setup Information

Happens on web and mobile clients

Relevant logs:

server.js:204 System ➔ error Error sending message: TypeError: Cannot freeze array buffer views with elements
    at Function.freeze (<anonymous>)
    at Function.deepFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:18:16)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:22:27
    at Array.forEach (<anonymous>)
    at Function.deepFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:19:42)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:22:27
    at Array.forEach (<anonymous>)
    at Function.deepFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:19:42)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:22:27
    at Array.forEach (<anonymous>)
    at Function.deepFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:19:42)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:22:27
    at Array.forEach (<anonymous>)
    at Function.deepFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:19:42)
    at Function.deepCloneAndFreeze (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/misc/Utilities.js:28:26)
    at AppListenerManager.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:109:71)
    at Generator.next (<anonymous>)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:3:12)
    at AppListenerManager.executePreMessageSentPrevent (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:107:16)
    at AppListenerManager.<anonymous> (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:56:33)
    at Generator.next (<anonymous>)
    at /app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:7:71
    at new Promise (<anonymous>)
    at __awaiter (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:3:12)
    at AppListenerManager.executeListener (/app/bundle/programs/server/npm/node_modules/@rocket.chat/apps-engine/server/managers/AppListenerManager.js:52:16)
    at app/apps/server/bridges/listeners.js:8:68
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at app/apps/server/bridges/listeners.js:8:15
    at /app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/fiber_pool.js:43:40
 => awaited here:
    at Function.Promise.await (/app/bundle/programs/server/npm/node_modules/meteor/promise/node_modules/meteor-promise/promise_server.js:56:12)
    at sendMessage (app/lib/server/functions/sendMessage.js:179:27)
    at executeSendMessage (app/lib/server/methods/sendMessage.js:77:10)
    at MethodInvocation.sendMessage (app/lib/server/methods/sendMessage.js:105:10)
    at maybeAuditArgumentChecks (packages/ddp-server/livedata_server.js:1771:12)
    at packages/ddp-server/livedata_server.js:1689:15
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at packages/ddp-server/livedata_server.js:1687:36
    at new Promise (<anonymous>)
    at Server.applyAsync (packages/ddp-server/livedata_server.js:1686:12)
    at Server.apply (packages/ddp-server/livedata_server.js:1625:26)
    at Server.call (packages/ddp-server/livedata_server.js:1607:17)
    at DDPCommon.MethodInvocation.<anonymous> (app/api/server/v1/chat.js:175:59)
    at packages/dispatch_run-as-user.js:211:14
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at Object.Meteor.runAsUser (packages/dispatch_run-as-user.js:210:33)
    at Object.post (app/api/server/v1/chat.js:175:23)
    at app/api/server/api.js:343:82
    at Meteor.EnvironmentVariable.EVp.withValue (packages/meteor.js:1234:12)
    at Object._internalRouteActionHandler [as action] (app/api/server/api.js:343:39)
    at Route.share.Route.Route._callEndpoint (packages/nimble_restivus/lib/route.coffee:150:32)
    at packages/nimble_restivus/lib/route.coffee:59:33
    at packages/simple_json-routes.js:98:9

[pehlert]

Just a quick update, this still happens on 3.4.0 so we are forced to leave the App Framework disabled. I suppose it's something related to our database as nobody has reported it before?

[MiStr1]

We've run into a similar error log. The problem was that we were saving python uuid type into custom fields. After converting that type to string the error was gone.

[pehlert]

Thanks for the insight @MiStr1 - we're not doing that, at least not intentionally. Chances are however that the Giphy app did it at some point. Could you point out what collection caused the issues for you?

[MiStr1]

The only problem was our python script. In which we were saving data to rocket chat room custom fields. And as I stated before the problem was with the unsuported type being saved in rocket chat custom fields.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[pehlert]

This is still happening for us on 3.7.0 when we enable the App Framework. We use LDAP auth, but that's it - everything else is a plain RocketChat installed with all Apps uninstalled now.