E11000 duplicate key error in mongodb

orlando-jamie commented 5 years ago

Expected Behavior

Would expect GL to run with no errors.

Current Behavior

After upgrading to GL 3.1, I frequently see the following alert in server.log. Seems to be caused by mongodb.

2019-08-21T14:47:21.865Z ERROR [IndexFieldTypePollerPeriodical] Uncaught exception in periodical
com.mongodb.DuplicateKeyException: Write failed with error code 11000 and error message 'E11000 duplicate key error index: graylog.index_field_types.$index_name_1 dup key: { : "gl-events_0" }'
    at com.mongodb.operation.BaseWriteOperation.convertBulkWriteException(BaseWriteOperation.java:190) ~[graylog.jar:?]
    at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:154) ~[graylog.jar:?]
    at com.mongodb.operation.BaseWriteOperation.execute(BaseWriteOperation.java:52) ~[graylog.jar:?]
    at com.mongodb.Mongo$3.execute(Mongo.java:837) ~[graylog.jar:?]
    at com.mongodb.Mongo$3.execute(Mongo.java:818) ~[graylog.jar:?]
    at com.mongodb.DBCollection.executeWriteOperation(DBCollection.java:341) ~[graylog.jar:?]
    at com.mongodb.DBCollection.update(DBCollection.java:563) ~[graylog.jar:?]
    at com.mongodb.DBCollection.update(DBCollection.java:486) ~[graylog.jar:?]
    at com.mongodb.DBCollection.update(DBCollection.java:463) ~[graylog.jar:?]
    at com.mongodb.DBCollection.update(DBCollection.java:441) ~[graylog.jar:?]
    at org.mongojack.JacksonDBCollection.update(JacksonDBCollection.java:516) ~[graylog.jar:?]
    at org.mongojack.JacksonDBCollection.update(JacksonDBCollection.java:592) ~[graylog.jar:?]
    at org.graylog2.indexer.fieldtypes.IndexFieldTypesService.upsert(IndexFieldTypesService.java:73) ~[graylog.jar:?]
    at java.lang.Iterable.forEach(Iterable.java:75) ~[?:1.8.0_222]
    at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.lambda$doRun$2(IndexFieldTypePollerPeriodical.java:120) ~[graylog.jar:?]
    at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:407) ~[graylog.jar:?]
    at org.graylog2.indexer.fieldtypes.IndexFieldTypePollerPeriodical.doRun(IndexFieldTypePollerPeriodical.java:111) ~[graylog.jar:?]
    at org.graylog2.plugin.periodical.Periodical.run(Periodical.java:77) [graylog.jar:?]
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_222]
    at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [?:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_222]
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_222]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_222]
    at java.lang.Thread.run(Thread.java:748) [?:1.8.0_222]

Steps to Reproduce (for bugs)

Upgrade to 3.1
See error in log

Context

Your Environment

Graylog Version: graylog-server-3.1.0-6.noarch
Elasticsearch Version: elasticsearch-6.8.2-1.noarch
MongoDB Version: mongodb-org-server-3.6.13-1.el7.x86_64
Operating System: CentOS Linux release 7.6.1810 (Core)
Browser version: N/A

alarex commented 5 years ago

Same here, any updates on issue?

esapozhnikov-wish commented 4 years ago

I am seeing same problem after update to 3.1

Aenima4six2 commented 4 years ago

Also getting this error. Logs are flooded with this exception until the offending document is manually removed. Recommend using upsert operation instead of insertOne.

andloh commented 4 years ago

also getting this error. mongo 4.2 and GL 3.1

bernd commented 4 years ago

@orlando-jamie @alarex @esapozhnikov @andloh I am unable to reproduce this, unfortunately.

Can you please show me the output of the following commands executed on the MongoDB server:

mongo graylog --eval "db.index_sets.find().pretty()"
mongo graylog --eval "db.index_field_types.find().pretty()"

(Please replace graylog with the actual database name if it isn't named graylog in your setup)

That should help us to debug the issue. Thank you

@Aenima4six2 We are already using an upsert operation, so that shouldn't be the issue.

andloh commented 4 years ago

Hi, solved this by persisting the graylog server id. We was running it in k8s, and every time the container restarts, graylog gets a new id

no-response[bot] commented 4 years ago

This issue has been automatically closed because there has been no response to our request for more information from the original author. With only the information that is currently in the issue, we don't have enough information to take action. Please reach out if you have or find the answers we need so that we can investigate further.

bernd commented 4 years ago

@orlando-jamie @alarex @esapozhnikov @Aenima4six2 We believe we found the root cause of this. It has been documented in https://github.com/Graylog2/graylog2-server/issues/7258.

A fix for that has been committed and will be included in the upcoming 3.2.0 release.

Thank you for the report!

sinanpostaci commented 4 years ago

Hi, solved this by persisting the graylog server id. We was running it in k8s, and every time the container restarts, graylog gets a new id

Hello andloh, we're using version 3.3.5-1 and we couldn't fix this yet. Would you mind telling us how to persist server id? Thanks.

andloh commented 4 years ago

add the node-id to this file /usr/share/graylog/data/config/node-id

Graylog2 / graylog2-server