Recently I had to migrate my production environment to Amazon AWS Windows instance. I didn't change anything and the data isn't syncing any more.
Let me describe the problem.
I'm using MongoDB 2.6.3 running on Ubuntu server.
ES 1.1.1 with River 2.0.0 on Windows Server 2008 R2.
All working fine except sporadic River crashes.
On AWS EC2 Windows Server 2012 replicated the setup and River isn't pulling any data from MongoDB on remote server. As soon as I've installed the MongoDB locally it works fine. Problem is I can't have MongoDB running on the same server as ES.
There is some problem around the publish address being obtained by River from what I've been able to work out. The server is behind firewall within private network and the transport_address": "inet[/x.x.x.x:9300]" is basically set to that private network IP. On the currently working server this transport address is set to the server address. Any work around that possible?
Here are the two most common exceptions:
[2015-04-16 15:50:12,001][INFO ][cluster.metadata ] [Stardust] [_river] update_mapping [index_name] (dynamic)
[2015-04-16 15:50:12,587][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Initializing river : [index_name]
[2015-04-16 15:50:12,859][INFO ][river.mongodb ] Parse river settings for index_name
[2015-04-16 15:50:13,132][TRACE][river.mongodb ] mongoServersSettings: [{port=27017, host=linux_box_ip}]
[2015-04-16 15:50:13,216][INFO ][river.mongodb ] Server: linux_box_ip - 27017
[2015-04-16 15:50:13,481][TRACE][river.mongodb ] mongoOptionsSettings: {secondary_read_preference=true}
[2015-04-16 15:50:13,770][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] Starting river index_name
[2015-04-16 15:50:13,804][DEBUG][river.mongodb.util ] setRiverStatus called with index_name - RUNNING
[2015-04-16 15:50:13,865][INFO ][cluster.metadata ] [Stardust] [_river] update_mapping [index_name] (dynamic)
[2015-04-16 15:50:13,880][DEBUG][org.elasticsearch.river.mongodb.MongoDBRiver] Using mongodb server(s): host [linux_box_ip], port [27017]
[2015-04-16 15:50:13,965][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] MongoDB River Plugin - version[2.0.0] - hash[a0c23f1] - time[2014-02-23T20:40:05Z]
[2015-04-16 15:50:13,994][INFO ][org.elasticsearch.river.mongodb.MongoDBRiver] starting mongodb stream. options: secondaryreadpreference [true], drop_collection [false], include_collection [], throttl
esize [5000], gridfs [false], filter [null], db [db_name], collection [entities], script [null], indexing to [index_name]/[entity]
[2015-04-16 15:50:15,170][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] MongoAdminUser: - authenticated: false
[2015-04-16 15:50:15,359][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] Found admin database
[2015-04-16 15:50:15,543][TRACE][org.elasticsearch.river.mongodb.MongoDBRiver] About to execute: { "serverStatus" : 1 , "asserts" : 0 , "backgroundFlushing" : 0 , "connections" : 0 , "cursors" : 0 , "
dur" : 0 , "extra_info" : 0 , "globalLock" : 0 , "indexCounters" : 0 , "locks" : 0 , "metrics" : 0 , "network" : 0 , "opcounters" : 0 , "opcountersRepl" : 0 , "recordStats" : 0 , "repl" : 0}
Apr 16, 2015 3:50:36 PM com.mongodb.ConnectionStatus$UpdatableNode update
WARNING: Server seen down: user/77.x.x.x:27017 <-- THIS IP IS DIFFERENT THAN MONGODB SERVER IP !!!
java.net.SocketTimeoutException: connect timed out
at java.net.DualStackPlainSocketImpl.waitForConnect(Native Method)
at java.net.DualStackPlainSocketImpl.socketConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.doConnect(Unknown Source)
at java.net.AbstractPlainSocketImpl.connectToAddress(Unknown Source)
at java.net.AbstractPlainSocketImpl.connect(Unknown Source)
at java.net.PlainSocketImpl.connect(Unknown Source)
at java.net.SocksSocketImpl.connect(Unknown Source)
at java.net.Socket.connect(Unknown Source)
at com.mongodb.DBPort._open(DBPort.java:223)
at com.mongodb.DBPort.go(DBPort.java:125)
at com.mongodb.DBPort.go(DBPort.java:106)
at com.mongodb.DBPort.findOne(DBPort.java:162)
at com.mongodb.DBPort.runCommand(DBPort.java:170)
at com.mongodb.ConnectionStatus$UpdatableNode.update(ConnectionStatus.java:196)
at com.mongodb.ReplicaSetStatus$UpdatableReplicaSetNode.update(ReplicaSetStatus.java:603)
at com.mongodb.ReplicaSetStatus$Updater.updateAll(ReplicaSetStatus.java:755)
at com.mongodb.ReplicaSetStatus$Updater.run(ReplicaSetStatus.java:725)
[2015-04-16 15:50:56,785][WARN ][org.elasticsearch.river.mongodb.MongoDBRiver] Fail to start river index_name
com.mongodb.MongoException: can't find a master
at com.mongodb.DBTCPConnector.checkMaster(DBTCPConnector.java:517)
at com.mongodb.DBTCPConnector.innerCall(DBTCPConnector.java:236)
at com.mongodb.DBTCPConnector.call(DBTCPConnector.java:216)
at com.mongodb.DBApiLayer$MyCollection.__find(DBApiLayer.java:288)
at com.mongodb.DB.command(DB.java:262)
at com.mongodb.DB.command(DB.java:244)
at com.mongodb.DB.command(DB.java:301)
at com.mongodb.DB.command(DB.java:199)
at org.elasticsearch.river.mongodb.MongoDBRiver.isMongos(MongoDBRiver.java:264)
at org.elasticsearch.river.mongodb.MongoDBRiver.start(MongoDBRiver.java:208)
at org.elasticsearch.river.RiversService.createRiver(RiversService.java:148)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:275)
at org.elasticsearch.river.RiversService$ApplyRivers$2.onResponse(RiversService.java:269)
at org.elasticsearch.action.support.TransportAction$ThreadedActionListener$1.run(TransportAction.java:93)
at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
[2015-04-16 15:50:56,872][DEBUG][river.mongodb.util ] setRiverStatus called with index_name - START_FAILED
[2015-04-16 15:50:56,961][INFO ][cluster.metadata ] [Stardust] [_river] update_mapping [index_name] (dynamic)
Recently I had to migrate my production environment to Amazon AWS Windows instance. I didn't change anything and the data isn't syncing any more.
Let me describe the problem. I'm using MongoDB 2.6.3 running on Ubuntu server. ES 1.1.1 with River 2.0.0 on Windows Server 2008 R2. All working fine except sporadic River crashes.
On AWS EC2 Windows Server 2012 replicated the setup and River isn't pulling any data from MongoDB on remote server. As soon as I've installed the MongoDB locally it works fine. Problem is I can't have MongoDB running on the same server as ES.
There is some problem around the publish address being obtained by River from what I've been able to work out. The server is behind firewall within private network and the transport_address": "inet[/x.x.x.x:9300]" is basically set to that private network IP. On the currently working server this transport address is set to the server address. Any work around that possible?
Here are the two most common exceptions: