search

ozlerhakan / mongolastic

:traffic_light: A dataset migration tool from MongoDB to Elasticsearch and vice versa.
MIT License
136 stars 34 forks source link

After run mongolastic, no data are transfer from mongodb to elasticsearch #23

Closed weijingj closed 7 years ago

weijingj commented 7 years ago

My config file:

misc:
    dindex:
        name: productstore02
    ctype:
        name: Execution
mongo:
    host: win-8727iunk90s 
    port: 27017
elastic:
    host: bpmznsvt04
    port: 9300

Config Output: {elastic=Elastic{host='bpmznsvt04', port=9300, clusterName=null, dateFormat=null, longToString=false, auth=null}, misc=Misc{batch=200, direction='me', dindex=Namespace{as='my-index', name='productstore02'}, ctype=Namespace{as='null', name='Execution'}, dropDataset=true}, mongo=Mongo{host='win-8727iunk90s', port=27017, query='{}', project='null', auth=null}}

[INFO ] [2017-04-14 18:25:02] [main] [INFO]: - no modules loaded [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty3Plugin] [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty4Plugin] [INFO ] [2017-04-14 18:25:02] [main] [INFO]: - loaded plugin [org.elasticsearch.xpack.XPackPlugin] [WARN ] [2017-04-14 18:25:04] [elasticsearch[client][transport_client_boss][T#1]] [WARN]: - Transport response handler not found of id [1] [WARN ] [2017-04-14 18:25:04] [elasticsearch[client][transport_client_boss][T#2]] [WARN]: - Transport response handler not found of id [3] [INFO ] [2017-04-14 18:25:04] [main] [INFO]: - Cluster created with settings {hosts=[win-8727iunk90s:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='5000 ms', maxWaitQueueSize=500} [INFO ] [2017-04-14 18:25:04] [main] [INFO]: - Adding discovered server win-8727iunk90s:27017 to client view of cluster [INFO ] [2017-04-14 18:25:04] [main] [INFO]: - No server chosen by ReadPreferenceServerSelector{readPreference=ReadPreference{name=primaryPreferred}} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=win-8727iunk90s:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 5000 ms before timing out [INFO ] [2017-04-14 18:25:04] [cluster-ClusterId{value='58f0a3803a75c75a33cf00a5', description='null'}-win-8727iunk90s:27017] [INFO]: - Opened connection [connectionId{localValue:1, serverValue:17}] to win-8727iunk90s:27017 [INFO ] [2017-04-14 18:25:04] [cluster-ClusterId{value='58f0a3803a75c75a33cf00a5', description='null'}-win-8727iunk90s:27017] [INFO]: - Monitor thread successfully connected to server with description ServerDescription{address=win-8727iunk90s:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 3]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=7228015} [INFO ] [2017-04-14 18:25:04] [cluster-ClusterId{value='58f0a3803a75c75a33cf00a5', description='null'}-win-8727iunk90s:27017] [INFO]: - Discovered cluster type of STANDALONE [INFO ] [2017-04-14 18:25:04] [main] [INFO]: - Opened connection [connectionId{localValue:2, serverValue:18}] to win-8727iunk90s:27017 [INFO ] [2017-04-14 18:25:04] [main] [INFO]: - Mongo collection count: 121468 [INFO ] [2017-04-14 18:25:05] [main] [INFO]: - Transferring data began to elasticsearch. [INFO ] [2017-04-14 18:25:08] [main] [INFO]: - Transferring data began to elasticsearch. [INFO ] [2017-04-14 18:25:08] [main] [INFO]: - Closed connection [connectionId{localValue:2, serverValue:18}] to win-8727iunk90s:27017 because the pool has been closed. [INFO ] [2017-04-14 18:25:10] [main] [INFO]: - Load duration: 8533ms

Anything I have configured incorrect? Thanks.

ozlerhakan commented 7 years ago

Hi @weijingj ,

I see 2 warnings on your log:

[WARN ] [2017-04-14 18:25:04] [elasticsearch[client][transport_client_boss][T#1]] [WARN]: - Transport response handler not found of id [1]
[WARN ] [2017-04-14 18:25:04] [elasticsearch[client][transport_client_boss][T#2]] [WARN]: - Transport response handler not found of id [3]

I think you have 2 es nodes, right?

  1. Which version of elasticsearch do you have?
  2. Do your es nodes have the same version?

  3. Do you use the latest mongolastic jar or docker image?

    I have tested it with a elasticsearch 5.3.0 node.

weijingj commented 7 years ago
  1. I am using elasticsearch 5.1.1
  2. I have only one es node on my machine.
  3. I am using version 1.4.2 mongolastic jar.
weijingj commented 7 years ago

I just try elasticsearch 5.3. The warning is gone. But I still don't get any index or type in es.

Here is the output:

[INFO ] [2017-04-16 19:54:39] [main] [INFO]: -
Config Output:
{elastic=Elastic{host='localhost', port=9300, clusterName=null, dateFormat=null, longToString=false, auth=null}, misc=Misc{batch=200, direction='me', dindex=Namespace{as='null', name='productstore02'}, ctype=Namespace{as='null', name='Execution'}, dropDataset=true}, mongo=Mongo{host='win-8727iunk90s', port=27017, query='{}', project='null', auth=null}}

[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - no modules loaded
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin]
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin]
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin]
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty3Plugin]
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty4Plugin]
[INFO ] [2017-04-16 19:54:40] [main] [INFO]: - loaded plugin [org.elasticsearch.xpack.XPackPlugin]
[INFO ] [2017-04-16 19:54:42] [main] [INFO]: - Cluster created with settings {hosts=[win-8727iunk90s:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='5000 ms', maxWaitQueueSize=500}
[INFO ] [2017-04-16 19:54:42] [main] [INFO]: - Adding discovered server win-8727iunk90s:27017 to client view of cluster
[INFO ] [2017-04-16 19:54:42] [cluster-ClusterId{value='58f35b82c24aef72fb6d5260', description='null'}-win-8727iunk90s:27017] [INFO]: - Opened connection [connectionId{localValue:1, serverValue:1}] to win-8727iunk90s:27017
[INFO ] [2017-04-16 19:54:42] [main] [INFO]: - No server chosen by ReadPreferenceServerSelector{readPreference=ReadPreference{name=primaryPreferred}} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=win-8727iunk90s:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 5000 ms before timing out
[INFO ] [2017-04-16 19:54:42] [cluster-ClusterId{value='58f35b82c24aef72fb6d5260', description='null'}-win-8727iunk90s:27017] [INFO]: - Monitor thread successfully connected to server with description ServerDescription{address=win-8727iunk90s:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[3, 4, 3]}, minWireVersion=0, maxWireVersion=5, maxDocumentSize=16777216, roundTripTimeNanos=1061320}
[INFO ] [2017-04-16 19:54:42] [cluster-ClusterId{value='58f35b82c24aef72fb6d5260', description='null'}-win-8727iunk90s:27017] [INFO]: - Discovered cluster type of STANDALONE
[INFO ] [2017-04-16 19:54:42] [main] [INFO]: - Opened connection [connectionId{localValue:2, serverValue:2}] to win-8727iunk90s:27017
[INFO ] [2017-04-16 19:54:42] [main] [INFO]: - Mongo collection count: 121468
[INFO ] [2017-04-16 19:54:43] [main] [INFO]: - Transferring data began to elasticsearch.
[INFO ] [2017-04-16 19:54:43] [main] [INFO]: - Transferring data began to elasticsearch.
[INFO ] [2017-04-16 19:54:43] [main] [INFO]: - Transferring data began to elasticsearch.
[INFO ] [2017-04-16 19:54:45] [main] [INFO]: - Transferring data began to elasticsearch.
[INFO ] [2017-04-16 19:54:45] [main] [INFO]: - Closed connection [connectionId{localValue:2, serverValue:2}] to win-8727iunk90s:27017 because the pool has been closed.
[INFO ] [2017-04-16 19:54:46] [main] [INFO]: - Load duration: 6963ms
ozlerhakan commented 7 years ago

Hi @weijingj ,

What will happen if you increase the batch size more than 1000? ex:

misc:
    dindex:
        name: productstore02
    ctype:
        name: Execution
    batch: 5000
mongo:
    host: win-8727iunk90s 
    port: 27017
elastic:
    host: bpmznsvt04
    port: 9300
weijingj commented 7 years ago

Even set batch as 5000. It does not work. Here is the mongodb log:

2017-04-17T15:50:11.081+0800 I NETWORK  [conn2] received client metadata from 9.
110.177.24:49897 conn2: { driver: { name: "mongo-java-driver", version: "unknown
" }, os: { type: "Linux", name: "Linux", architecture: "amd64", version: "3.10.0
-217.el7.x86_64" }, platform: "Java/Oracle Corporation/1.8.0_92-b14" }
2017-04-17T15:50:11.512+0800 I COMMAND  [conn2] command productstore02.Execution
 command: getMore { getMore: 52421158820, collection: "Execution" } originatingC
ommand: { aggregate: "Execution", pipeline: [ { $match: {} }, { $skip: 0 } ], cu
rsor: {}, allowDiskUse: true } planSummary: COLLSCAN cursorid:52421158820 keysEx
amined:0 docsExamined:37344 numYields:316 nreturned:39207 reslen:16776964 locks:
{ Global: { acquireCount: { r: 664 } }, MMAPV1Journal: { acquireCount: { r: 332
} }, Database: { acquireCount: { r: 332 } }, Collection: { acquireCount: { R: 33
2 } } } protocol:op_query 313ms
2017-04-17T15:50:12.762+0800 I COMMAND  [conn2] command productstore02.Execution
 command: getMore { getMore: 52421158820, collection: "Execution" } originatingC
ommand: { aggregate: "Execution", pipeline: [ { $match: {} }, { $skip: 0 } ], cu
rsor: {}, allowDiskUse: true } planSummary: COLLSCAN cursorid:52421158820 keysEx
amined:0 docsExamined:34985 numYields:294 nreturned:33872 reslen:16775802 locks:
{ Global: { acquireCount: { r: 620 } }, MMAPV1Journal: { acquireCount: { r: 310
} }, Database: { acquireCount: { r: 310 } }, Collection: { acquireCount: { R: 31
0 } } } protocol:op_query 337ms
2017-04-17T15:50:13.722+0800 I COMMAND  [conn2] command productstore02.Execution
 command: getMore { getMore: 52421158820, collection: "Execution" } originatingC
ommand: { aggregate: "Execution", pipeline: [ { $match: {} }, { $skip: 0 } ], cu
rsor: {}, allowDiskUse: true } planSummary: COLLSCAN cursorid:52421158820 keysEx
amined:0 docsExamined:22496 numYields:202 nreturned:22875 reslen:16777011 locks:
{ Global: { acquireCount: { r: 430 } }, MMAPV1Journal: { acquireCount: { r: 215
} }, Database: { acquireCount: { r: 215 } }, Collection: { acquireCount: { R: 21
5 } } } protocol:op_query 363ms
2017-04-17T15:50:14.339+0800 I COMMAND  [conn2] command productstore02.Execution
 command: getMore { getMore: 52421158820, collection: "Execution" } originatingC
ommand: { aggregate: "Execution", pipeline: [ { $match: {} }, { $skip: 0 } ], cu
rsor: {}, allowDiskUse: true } planSummary: COLLSCAN cursorid:52421158820 keysEx
amined:0 docsExamined:22281 numYields:190 nreturned:21698 reslen:16777085 locks:
{ Global: { acquireCount: { r: 406 } }, MMAPV1Journal: { acquireCount: { r: 203
} }, Database: { acquireCount: { r: 203 } }, Collection: { acquireCount: { R: 20
3 } } } protocol:op_query 246ms

No log at elastic search side. Is it possible that no record are got from mongodb, so mongolastic did not try to insert into es?

Do I need to create index and type in ES before mongolastic run? Thanks.

ozlerhakan commented 7 years ago

Hi @weijingj ,

Is it possible that no record are got from mongodb, so mongolastic did not try to insert into es?

what do you get when typing db.Execution.count() on mongo shell? If you have some data, the problem is in the es part.

Do I need to create index and type in ES before mongolastic run?

No need to create them beforehand. Once the data move to the es node, es will create the specified index and type for you.

weijingj commented 7 years ago

I have data in mongo DB. I finally manage to export data from mongo to ES using transporter

ozlerhakan commented 7 years ago

nice that you manage the data, I have tested with a few collections with no errors, I keep this in my mind, for now I close this issue. thanks!