search

RocketChat / Rocket.Chat

The communications platform that puts data protection first.
https://rocket.chat/
Other
40.22k stars 10.42k forks source link

Minio support as AWS s3 storage broken #7411

Closed TheReal1604 closed 7 years ago

TheReal1604 commented 7 years ago

Rocket.Chat Version: 0.57.0 Running Instances: 1 DB Replicaset OpLog: yes

Since the update the support for the minio file upload backend (as used as aws s3 storage) seems to be broken:

2017-07-05T07:26:31.956702688Z { [InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.]
2017-07-05T07:26:31.956757270Z   message: 'The AWS Access Key Id you provided does not exist in our records.',
2017-07-05T07:26:31.956766140Z   code: 'InvalidAccessKeyId',
2017-07-05T07:26:31.956772463Z   region: null,
2017-07-05T07:26:31.956778177Z   time: Wed Jul 05 2017 07:26:31 GMT+0000 (UTC),
2017-07-05T07:26:31.956784338Z   requestId: 'E8816626A1DE5C11',
2017-07-05T07:26:31.956790145Z   extendedRequestId: '8ksjlcLe2x9QdkJ2yYjG5W8K2ciCyj7EbktR68E4lciOta6ps75Xxrw9XjAfeocGlnRJXxVgvnM=',
2017-07-05T07:26:31.956795671Z   cfId: undefined,
2017-07-05T07:26:31.956829773Z   statusCode: 403,
2017-07-05T07:26:31.956835701Z   retryable: false,
2017-07-05T07:26:31.956841834Z   retryDelay: 33.5321101360023 }
2017-07-05T07:26:35.184986998Z { [InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.]
2017-07-05T07:26:35.185025951Z   message: 'The AWS Access Key Id you provided does not exist in our records.',
2017-07-05T07:26:35.185035631Z   code: 'InvalidAccessKeyId',
2017-07-05T07:26:35.185042920Z   region: null,
2017-07-05T07:26:35.185048846Z   time: Wed Jul 05 2017 07:26:35 GMT+0000 (UTC),
2017-07-05T07:26:35.185054979Z   requestId: 'F2CC85170378EFD3',
2017-07-05T07:26:35.185061346Z   extendedRequestId: 'kX5MaCXh0GZojkAvWuvrkpX7v7njAr4xdQe+a0XjrfxyOtM0lC83xxmx+a3D2a0HavAfHqTNk7E=',
2017-07-05T07:26:35.185067226Z   cfId: undefined,
2017-07-05T07:26:35.185073132Z   statusCode: 403,
2017-07-05T07:26:35.185079287Z   retryable: false,
2017-07-05T07:26:35.185085603Z   retryDelay: 77.44381697848439 }
2017-07-05T07:26:42.258140215Z { [InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.]
2017-07-05T07:26:42.258206723Z   message: 'The AWS Access Key Id you provided does not exist in our records.',
2017-07-05T07:26:42.258280792Z   code: 'InvalidAccessKeyId',
2017-07-05T07:26:42.258295919Z   region: null,
2017-07-05T07:26:42.258424889Z   time: Wed Jul 05 2017 07:26:42 GMT+0000 (UTC),
2017-07-05T07:26:42.258452814Z   requestId: '9541501938ACB174',
2017-07-05T07:26:42.258537342Z   extendedRequestId: 'SEygGSZNCSLFaJHYtz+08F1wwGQQ9wvpv6/0Eojtlehz8weoodBq7HqgAGBKGyh3Tw1SwiC127k=',
2017-07-05T07:26:42.258554324Z   cfId: undefined,
2017-07-05T07:26:42.258587738Z   statusCode: 403,
2017-07-05T07:26:42.258601034Z   retryable: false,
2017-07-05T07:26:42.258630994Z   retryDelay: 5.03008661326021 }
2017-07-05T07:26:42.733549783Z Exception while invoking method 'ufsComplete' InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.
2017-07-05T07:26:42.733612588Z   at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:449:15)
2017-07-05T07:26:42.733686071Z   at [object Object].Meteor.methods.ufsComplete (/app/bundle/programs/server/packages/jalik_ufs.js:818:24)
2017-07-05T07:26:42.733700849Z   at [object Object].methodsMap.(anonymous function) (/app/bundle/programs/server/packages/rocketchat_lib.js:1028:26)
2017-07-05T07:26:42.733814665Z   at [object Object].methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30)
2017-07-05T07:26:42.733830716Z   at maybeAuditArgumentChecks (/app/bundle/programs/server/packages/ddp-server.js:1823:12)
2017-07-05T07:26:42.733876647Z   at /app/bundle/programs/server/packages/ddp-server.js:905:20
2017-07-05T07:26:42.733903938Z   at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
2017-07-05T07:26:42.733951917Z   at /app/bundle/programs/server/packages/ddp-server.js:904:41
2017-07-05T07:26:42.733981050Z   at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
2017-07-05T07:26:42.734037575Z   at /app/bundle/programs/server/packages/ddp-server.js:903:46
2017-07-05T07:26:42.734059630Z   at [object Object]._.extend.protocol_handlers.method (/app/bundle/programs/server/packages/ddp-server.js:876:21)
2017-07-05T07:26:42.734122955Z   at /app/bundle/programs/server/packages/ddp-server.js:755:85
2017-07-05T07:26:42.734138800Z 
2017-07-05T07:26:42.734171030Z     - - - - -
2017-07-05T07:26:42.734183925Z   at Request.extractError (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/services/s3.js:577:35)
2017-07-05T07:26:42.734215006Z   at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:105:20)
2017-07-05T07:26:42.734271729Z   at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
2017-07-05T07:26:42.734314778Z   at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:682:14)
2017-07-05T07:26:42.734328457Z   at Request.transition (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:22:10)
2017-07-05T07:26:42.734360475Z   at AcceptorStateMachine.runTo (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:14:12)
2017-07-05T07:26:42.734427443Z   at /app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:26:10
2017-07-05T07:26:42.734442480Z   at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:38:9)
2017-07-05T07:26:42.734499495Z   at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:684:12)
2017-07-05T07:26:42.734514104Z   at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
2017-07-05T07:26:42.734547681Z   at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:77:10)
2017-07-05T07:26:42.734560633Z   at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:682:14)
2017-07-05T07:26:42.734594453Z   at Request.transition (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:22:10)
2017-07-05T07:26:42.734619505Z   at AcceptorStateMachine.runTo (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:14:12)
2017-07-05T07:26:42.734688250Z   at /app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:26:10
2017-07-05T07:26:42.734711710Z   at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:38:9)
2017-07-05T07:26:42.734833527Z   at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:684:12)
2017-07-05T07:26:42.734873843Z   at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:115:18)
2017-07-05T07:26:42.734937284Z   at callNextListener (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:95:12)
2017-07-05T07:26:42.734953277Z   at IncomingMessage.onEnd (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/event_listeners.js:269:13)
2017-07-05T07:26:42.735060304Z   at emitNone (events.js:72:20)
2017-07-05T07:26:42.735085835Z   at IncomingMessage.emit (events.js:166:7)
2017-07-05T07:26:42.735152277Z   at endReadableNT (_stream_readable.js:923:12)
2017-07-05T07:26:42.735174423Z   at nextTickCallbackWith2Args (node.js:511:9)
2017-07-05T07:26:42.735261009Z   at process._tickDomainCallback (node.js:466:17)
2017-07-05T07:26:42.735289930Z 
2017-07-05T07:26:42.763674167Z [Error: FileNotFound: no file with id XFFkETsF5krLE7TFp found]
2017-07-05T07:26:43.225239636Z { [InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.]
2017-07-05T07:26:43.225318173Z   message: 'The AWS Access Key Id you provided does not exist in our records.',
2017-07-05T07:26:43.225512524Z   code: 'InvalidAccessKeyId',
2017-07-05T07:26:43.225553158Z   region: null,
2017-07-05T07:26:43.225626548Z   time: Wed Jul 05 2017 07:26:43 GMT+0000 (UTC),
2017-07-05T07:26:43.225644431Z   requestId: 'AE92FCD960B5E774',
2017-07-05T07:26:43.225765102Z   extendedRequestId: 'p4IHFUUbxwU5fPeYejRUZ6k8hpRFACZuXWTj3lb2nR/U0sRsxf7OOWRykWtAVnGoDUgEZHTK8Qc=',
2017-07-05T07:26:43.225792074Z   cfId: undefined,
2017-07-05T07:26:43.225848041Z   statusCode: 403,
2017-07-05T07:26:43.225875130Z   retryable: false,
2017-07-05T07:26:43.225946450Z   retryDelay: 30.06323562003672 }
2017-07-05T07:26:43.231224541Z { [InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records.]
2017-07-05T07:26:43.231278391Z   message: 'The AWS Access Key Id you provided does not exist in our records.',
2017-07-05T07:26:43.231375384Z   code: 'InvalidAccessKeyId',
2017-07-05T07:26:43.231391766Z   region: null,
2017-07-05T07:26:43.231431910Z   time: Wed Jul 05 2017 07:26:43 GMT+0000 (UTC),
2017-07-05T07:26:43.231442745Z   requestId: 'F9A9E2E24F5F35A3',
2017-07-05T07:26:43.231448901Z   extendedRequestId: 'XTvTjVbdm+gUpo1OMWuTqWHFxmKpRlcNz7t9Mn+hUaodfD72mNZ0TN+tBUJBqEmhooz7aAsWC6I=',
2017-07-05T07:26:43.231455575Z   cfId: undefined,
2017-07-05T07:26:43.231461679Z   statusCode: **403**,
2017-07-05T07:26:43.231468371Z   retryable: false,
2017-07-05T07:26:43.231474426Z   retryDelay: 59.2857604380697 }

It seems that the rocket.chat instance does the request on another way since the update and gets an http code 403. Maybe a possible direction: I doesnt see the requests on my minio instance anymore. Maybe the URL is some kind of hardcoded?

It is configured like described in the docs: https://rocket.chat/docs/administrator-guides/file-upload/minio

Any idea?

k0nsl commented 7 years ago

I can confirm that this integration is broken on 0.57.0 -- this is what I see when trying to upload anything:

0x0f.su image hosting

I am unable to retrieve any other logs at this moment. I can just confirm that custom S3 storage (using Minio, to be specific) is broken in 0.57.0. This worked fine in 0.56.0.

k0nsl commented 7 years ago

I do see the following in the logs after upgrading to 0.57.0 from 0.56.0:

[34mI20170705-10:51:19.778(0) Exception while invoking method 'ufsComplete' InvalidAccessKeyId: The AWS Access Key Id you provided does not exist in our records. at Object.Future.wait (/app/bundle/programs/server/node_modules/fibers/future.js:449:15) at [object Object].Meteor.methods.ufsComplete (/app/bundle/programs/server/packages/jalik_ufs.js:818:24) at [object Object].methodsMap.(anonymous function) (/app/bundle/programs/server/packages/rocketchat_lib.js:1028:26) at [object Object].methodMap.(anonymous function) (packages/rocketchat_monitoring.js:2731:30) at maybeAuditArgumentChecks (/app/bundle/programs/server/packages/ddp-server.js:1823:12) at /app/bundle/programs/server/packages/ddp-server.js:905:20 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at /app/bundle/programs/server/packages/ddp-server.js:904:41 at [object Object]._.extend.withValue (packages/meteor.js:1122:17) at /app/bundle/programs/server/packages/ddp-server.js:903:46 at [object Object]._.extend.protocol_handlers.method (/app/bundle/programs/server/packages/ddp-server.js:876:21) at /app/bundle/programs/server/packages/ddp-server.js:755:85 - - - - - at Request.extractError (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/services/s3.js:577:35) at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:105:20) at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:77:10) at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:682:14) at Request.transition (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:14:12) at /app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:684:12) at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:115:18) at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:77:10) at Request.emit (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:682:14) at Request.transition (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:22:10) at AcceptorStateMachine.runTo (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:14:12) at /app/bundle/programs/server/npm/node_modules/aws-sdk/lib/state_machine.js:26:10 at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:38:9) at Request.<anonymous> (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/request.js:684:12) at Request.callListeners (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:115:18) at callNextListener (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/sequential_executor.js:95:12) at IncomingMessage.onEnd (/app/bundle/programs/server/npm/node_modules/aws-sdk/lib/event_listeners.js:269:13) at emitNone (events.js:72:20) at IncomingMessage.emit (events.js:166:7) at endReadableNT (_stream_readable.js:923:12) at nextTickCallbackWith2Args (node.js:511:9) at process._tickDomainCallback (node.js:466:17)

Excuse me for the garbled format, but that is how it presented itself to me in the log.

k0nsl commented 7 years ago

The issue persists in 0.57.1.

k0nsl commented 7 years ago

I decided to try downgrading back to 0.56.0 (which sported a fully functional Minio integration) but any uploads results in the following:

0x0f.su image hosting

When I re-enabled file uploads I restarted the server, but it still only yielded the same result as shown in the screenshot. The log claims InvalidAccessKeyId (exactly the same as shown in this comment).

I do not know what is wrong. There has been no changes on my Minio instance -- it runs fine as usual.

k0nsl commented 7 years ago

S3 file upload (with Minio, to be clear) is now working with latest version (i.e 0.57.2) but all avatars were missing and the users had to re-upload them. Other than that...well, it is working fine ;)

Thanks, team!