search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.06k forks source link

DocumentDB support #5927

Open runningman84 opened 5 years ago

runningman84 commented 5 years ago

Expected Behavior

Graylog should work with AWS DocumentDB which is very similar to mongodb.

Current Behavior

Graylog does not work with DocumentDB because it uses a function "capped" which is not supported by DocumentDB.

Possible Solution

Maybe this function is not really needed and the same feature could be implemented in a way which does not need this function.

Steps to Reproduce (for bugs)

2019-04-16T12:08:18.846Z INFO  [connection] Opened connection [connectionId{localValue:4}] to docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017 
2019-04-16T12:08:18.865Z WARN  [GeoIpResolverEngine] GeoIP database file does not exist: /etc/graylog/server/GeoLite2-City.mmdb 
2019-04-16T12:08:18.909Z INFO  [connection] Opened connection [connectionId{localValue:5}] to docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017 
2019-04-16T12:08:19.003Z INFO  [connection] Opened connection [connectionId{localValue:6}] to docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017 
2019-04-16T12:08:19.012Z INFO  [connection] Opened connection [connectionId{localValue:8}] to docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017 
2019-04-16T12:08:19.031Z INFO  [connection] Opened connection [connectionId{localValue:7}] to docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017 
2019-04-16T12:08:19.700Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.701Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.701Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.701Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.701Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.701Z ERROR [CmdLineTool] Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 } 
2019-04-16T12:08:19.702Z ERROR [Server] 

################################################################################ 

ERROR: Unable to connect to MongoDB. Is it running and the configuration correct? 
Details: Command failed with error 303: 'Feature not supported: capped:true' on server docdb-2019-04-16-09-37-19.cbtciege4xon.eu-central-1.docdb.amazonaws.com:27017. The full response is { "ok" : 0.0, "errmsg" : "Feature not supported: capped:true", "code" : 303 }

Context

This feature would allow users to run Graylog in AWS Fargate and use AWS DocumentDB and AWS Elasticsearch for the heavy lifting.

Your Environment

lefthand commented 5 years ago

I have my Graylog cluster setup using DocumentDB and everything else works other than that one collection. I'm still testing some ideas for keeping it at a reasonable size. Also 🤞that DocumentDB will support capped collections sometime soon.

benishak commented 5 years ago

I migrated an existing graylog setup from mongo to documentdb by using AWS DMS ... it is working fine so far, all collections were created on documentdb but they lost the capped capability is there any disadvantage from not recycling these collections?

lefthand commented 5 years ago

@benishak Since the collection isn't capped it can get to a point where when Graylog tries to scan it, all of the read credits will get used up. You'll have pretty bad time once that happens.

runningman84 commented 5 years ago

Isn't there an option to disable the whole dead letter feature?

benishak commented 5 years ago

from source code index failures logs is the one using a capped collection and limiting logs to 50MB these logs or this collection is not being actively scanned or read as I understood. I believe if this collection grows significantly very fast then there is an issue in the cluster.

lefthand commented 5 years ago

That's exactly what happened to me, I had an issue where 1,000s of records were being written to the index failure table. I never figure which part of the system was trying to read that table but it caused major issues at that point.

I'm running 2.5 still so I don't know if anything is changed in 3.x.

joaquin386 commented 4 years ago

Came across this issue today deploying onto AWS DocDB. 

2020-05-04 00:52:37,036 ERROR: org.graylog2.bootstrap.CmdLineTool - Guice error (more detail on log level debug): Error injecting constructor, com.mongodb.MongoCommandException: Command failed with error 303: 'Feature not supported: capped:true' on server XXX:27017. The full response is 
{
    "ok": 0,
    "errmsg": "Feature not supported: capped:true",
    "code": 303
}

From AWS what is supported and what not.

Any work around this issue?

epierce78 commented 4 years ago

Have you opened a feature request with AWS? They don't support capped capability. I think that would be the only thing stopping this from working, unfortunately. :(

cameronattard commented 4 years ago

There's an open feature request with AWS. I recommend adding your name to it via your TAMs.