richardwilly98 / elasticsearch-river-mongodb

MongoDB River Plugin for ElasticSearch
1.12k stars 215 forks source link

elasticsearch doesnt index mongodb data #494

Closed fatihpolatli closed 9 years ago

fatihpolatli commented 9 years ago

hi,

i am using mongodb 3.0.0, elasticsearch 1.4.2 and river driver 2.0.6. when i start elasticsearch, i observe mongodb logs and i see that a new connection is opened for it and elasticsearch start with driver without error. however no data is indexed even though there is data on mongodb. here is elasticsearch. can you help please...?

[2015-03-23 10:45:47,105][INFO ][node ] [Wundarr the Aquarian] version[1.4.2], pid[2840], build[927caff/2014-12-16T14:11:12Z] [2015-03-23 10:45:47,106][INFO ][node ] [Wundarr the Aquarian] initializing ... [2015-03-23 10:45:47,234][INFO ][plugins ] [Wundarr the Aquarian] loaded [mongodb-river, mapper-attachments], sites [river-mongodb] [2015-03-23 10:45:52,549][INFO ][node ] [Wundarr the Aquarian] initialized [2015-03-23 10:45:52,550][INFO ][node ] [Wundarr the Aquarian] starting ... [2015-03-23 10:45:55,964][INFO ][transport ] [Wundarr the Aquarian] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.1.249:9300]} [2015-03-23 10:46:02,122][INFO ][discovery ] [Wundarr the Aquarian] elasticsearch/L7_mQH1OQ-GaO4nmUMeUEg [2015-03-23 10:46:05,914][INFO ][cluster.service ] [Wundarr the Aquarian] new_master [Wundarr the Aquarian][L7_mQH1OQ-GaO4nmUMeUEg][MTRCW077][inet[/192.168.1.249:9300]], reason: zen-disco-join (elected_as_master) [2015-03-23 10:46:07,016][INFO ][http ] [Wundarr the Aquarian] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.1.249:9200]} [2015-03-23 10:46:07,017][INFO ][node ] [Wundarr the Aquarian] started [2015-03-23 10:46:07,441][INFO ][gateway ] [Wundarr the Aquarian] recovered [2] indices into cluster_state [2015-03-23 10:46:08,238][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.6] - hash[3c38a8a] - time[2015-03-10T03:06:51Z] [2015-03-23 10:46:08,243][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - RUNNING [2015-03-23 10:46:08,265][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] River mongodb startup pending [2015-03-23 10:46:08,276][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb [2015-03-23 10:46:08,277][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [true], filter [null], db [test], collection [users], script [null], indexing to [mongotest]/[users] [2015-03-23 10:46:08,282][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] GridFS Mapping: {"users":{"properties":{"content":{"type":"attachment"},"filename":{"type":"string"},"contentType":{"type":"string"},"md5":{"type":"string"},"length":{"type":"long"},"chunkSize":{"type":"long"}}}} [2015-03-23 10:46:08,309][INFO ][river.mongodb ] [Wundarr the Aquarian] Creating MongoClient for [[127.0.0.1:27017]] [2015-03-23 10:46:10,004][INFO ][org.elasticsearch.river.mongodb.MongoConfigProvider] MongoDB version - 3.0.0 [2015-03-23 10:46:10,073][INFO ][org.elasticsearch.river.mongodb.CollectionSlurper] MongoDBRiver is beginning initial import of test.users [2015-03-23 10:46:10,591][INFO ][river.mongodb ] [Wundarr the Aquarian] Creating MongoClient for [[MTRCW077:27017]] [2015-03-23 10:46:10,601][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Started river mongodb

ewgRa commented 9 years ago

Can you provide your river initial config and when you stop river at host:9200/_plugin/river-mongodb/ what you have in logs?

Also please format display logs for better read and use markdown for configs and so on.

fatihpolatli commented 9 years ago

hi, here is river configuration and logs, i hope it looks more readable, thanks...

{
"took": 9,
"timed_out": false,
"_shards": {
    "total": 1,
    "successful": 1,
    "failed": 0
},
"hits": {
    "total": 3,
    "max_score": 1,
    "hits": [
        {
            "_index": "_river",
            "_type": "mongodb",
            "_id": "_meta",
            "_score": 1,
            "_source": {
                "type": "mongodb",
                "mongodb": {
                    "servers": [
                        {
                            "host": "127.0.0.1",
                            "port": 27017
                        }
                    ],
                    "options": {
                        "secondary_read_preference": true
                    },
                    "db": "test",
                    "collection": "users",
                    "gridfs": true
                },
                "index": {
                    "name": "mongotest",
                    "type": "users"
                }
            }
        },
        {
            "_index": "_river",
            "_type": "mongodb",
            "_id": "_riverstatus",
            "_score": 1,
            "_source": {
                "mongodb": {
                    "status": "RUNNING"
                }
            }
        },
        {
            "_index": "_river",
            "_type": "mongodb",
            "_id": "_status",
            "_score": 1,
            "_source": {
                "node": {
                    "id": "L7_mQH1OQ-GaO4nmUMeUEg",
                    "name": "Wundarr the Aquarian",
                    "transport_address": "inet[/192.168.1.249:9300]"
                }
            }
        }
    ]
}

}

logs ;

[2015-03-23 11:18:07,031][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - STOPPED [2015-03-23 11:18:07,157][INFO ][org.elasticsearch.river.mongodb.StatusChecker] About to stop river: mongodb [2015-03-23 11:18:07,158][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Stopping river mongodb [2015-03-23 11:18:07,159][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Stopped river mongodb [2015-03-23 11:18:07,159][INFO ][org.elasticsearch.river.mongodb.Indexer] river-mongodb indexer interrupted [2015-03-23 11:18:09,692][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] river-mongodb slurper interrupted [2015-03-23 11:18:09,693][INFO ][org.elasticsearch.river.mongodb.OplogSlurper] Slurper is stopping. River has status STOPPED

this is also elasticsearch start log;

[2015-03-23 11:16:56,582][INFO ][node ] [Sean Cassidy] version[1.4.2], pid[5576], build[927caff/2014-12-16T14:11:12Z] [2015-03-23 11:16:56,583][INFO ][node ] [Sean Cassidy] initializing ... [2015-03-23 11:16:56,716][INFO ][plugins ] [Sean Cassidy] loaded [mongodb-river, mapper-attachments], sites [river-mongodb] [2015-03-23 11:17:01,854][INFO ][node ] [Sean Cassidy] initialized [2015-03-23 11:17:01,855][INFO ][node ] [Sean Cassidy] starting ... [2015-03-23 11:17:04,825][INFO ][transport ] [Sean Cassidy] bound_address {inet[/0:0:0:0:0:0:0:0:9300]}, publish_address {inet[/192.168.1.249:9300]} [2015-03-23 11:17:10,131][INFO ][discovery ] [Sean Cassidy] elasticsearch/0p1Uz_ZNTB-ZBS5EXeRsCQ [2015-03-23 11:17:13,923][INFO ][cluster.service ] [Sean Cassidy] new_master [Sean Cassidy][0p1Uz_ZNTB-ZBS5EXeRsCQ][MTRCW077][inet[/192.168.1.249:9300]], reason: zen-disco-join (elected_as_master) [2015-03-23 11:17:15,046][INFO ][http ] [Sean Cassidy] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/192.168.1.249:9200]} [2015-03-23 11:17:15,047][INFO ][node ] [Sean Cassidy] started [2015-03-23 11:17:15,430][INFO ][gateway ] [Sean Cassidy] recovered [2] indices into cluster_state [2015-03-23 11:17:16,085][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.6] - hash[3c38a8a] - time[2015-03-10T03:06:51Z] [2015-03-23 11:17:16,090][INFO ][river.mongodb.util ] setRiverStatus called with mongodb - RUNNING [2015-03-23 11:17:16,111][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] River mongodb startup pending [2015-03-23 11:17:16,123][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river mongodb [2015-03-23 11:17:16,124][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttlesize [5000], gridfs [true], filter [null], db [test], collection [users], script [null], indexing to [mongotest]/[users] [2015-03-23 11:17:16,129][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] GridFS Mapping: {"users":{"properties":{"content":{"type":"attachment"},"filename":{"type":"string"},"contentType":{"type":"string"},"md5":{"type":"string"},"length":{"type":"long"},"chunkSize":{"type":"long"}}}} [2015-03-23 11:17:16,155][INFO ][river.mongodb ] [Sean Cassidy] Creating MongoClient for [[127.0.0.1:27017]] [2015-03-23 11:17:18,907][INFO ][org.elasticsearch.river.mongodb.MongoConfigProvider] MongoDB version - 3.0.0 [2015-03-23 11:17:18,984][INFO ][org.elasticsearch.river.mongodb.CollectionSlurper] MongoDBRiver is beginning initial import of test.users [2015-03-23 11:17:19,502][INFO ][river.mongodb ] [Sean Cassidy] Creating MongoClient for [[MTRCW077:27017]] [2015-03-23 11:17:19,512][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Started river mongodb

benmccann commented 9 years ago

A new version 2.0.7 was released. 2.0.6 is not reccomended

fatihpolatli commented 9 years ago

hi,

i have tried 2.0.7 version at my home in my windows 7 PC, it worked perfect but i cannot make data index in my windows 8 work PC. When elasticsearch starts, it opens connection on mongodb and log that start import index data, but it doesnt.

Another thing, in my home at windows 7 PC, ES logs as somthing like [Alyysium] but at work in windows 8 PC logs as [Victorious]. does this mean anything?

thanks...

fatihpolatli commented 9 years ago

hi,

when i make gridfs = false, it worked on windows 8 work PC, thanks...