DynamoDB Support

[deejross]

With possible support for Amazon Elasticsearch Service coming in the next version of Graylog which could solve #1461, would it also be possible to support one of the database backends available as a service within AWS? DynamoDB seems to be the closest offering to MongoDB. Setting up a MongoDB replica set using EC2 is expensive compared to AWS native services.

[jalogisch]

@deejross we can see your use case but it is not likely that we will implement this in the near future. But we would accept a PR that makes this possible.

[deejross]

Ideally, the configuration storage could be abstracted to allow pluggable backends. Once Graylog supports DynamoDB, someone else will also want to support the NoSQL solutions from other cloud providers. I'll see what the appetite is like internally for developing this functionality.

[jalogisch]

A quick FYI on the use of AWS DocumentDB instead of dedicated MongoDB instances.

Using a newly created DocumentDB cluster gives the following error in the master node server.log:

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-xxxxxx.cluster-xxxx.eu-west-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 }

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

Using AWS DMS to migrate from the MongoDB Instance to the new DocumentDB cluster and pointing the graylog server to the new cluster resolves the issue with the initial creation.

[nitingoura]

@jalogisch What version of graylog you are using. I tried with AWS DMS migrate. But keep getting an error.

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct?
Details: Command failed with error 67: 'namespace name generated from index name is too long' on server internal-xxxxxx.cluster-xxxxxxxxxxxx.us-east-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "namespace name generated from index name is too long", "code" : 67 }

Need help?

* Official documentation: http://docs.graylog.org/
* Community support: https://www.graylog.org/community-support/
* Commercial support: https://www.graylog.com/support/

Terminating. :(

I am using graylog 3

[jalogisch]

@nitingoura I used 2.5.1 for that test. But you run into another issue:

Details: Command failed with error 67: 'namespace name generated from index name is too long' on server internal-xxxxxx.cluster-xxxxxxxxxxxx.us-east-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "namespace name generated from index name is too long", "code" : 67 }

[nitingoura]

Yes, I am using version 3. Looks like Aws Doc DB does not support version 3 graylog.

[jalogisch]

@nitingoura other way around ...

AWS DocDB is not compatible with MongoDB 3.6 what is the minimal requirment for Graylog 3.0.

[nitingoura]

Minimal Requirement is 3.6 or later. Which make sense why Aws DocDB is not working with Graylog 3.0

[razvanphp]

Is official support planed for DocumentDB?

Version incompatibility should not be a problem, it is stated here that:

Amazon DocumentDB is compatible with version 3.6 of MongoDB.

[henrysachs]

I would like to "revive" this ticket or atleast get a no we won't do this? Anyone here that just tried it and it works? :D

[adrienmornet]

Hi there,

Still having the issue with Graylog 4.2 and AWS DocumentDB with Mongo 4.0.0 :

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct? Details: Command failed with error 67: 'namespace name generated from index name is too long' on server xxxxxx.docdb.amazonaws.com:27017. The full response is {"ok": 0.0, "code": 67, "errmsg": "namespace name generated from index name is too long", "operationTime": {"$timestamp": {"t": 1640683465, "i": 1}}}