search

ozlerhakan / mongolastic

:traffic_light: A dataset migration tool from MongoDB to Elasticsearch and vice versa.
MIT License
137 stars 34 forks source link

mongolastic with ElasticSearch 6.5.4 / Mongo 4.0 #31

Open ChuOue opened 5 years ago

ChuOue commented 5 years ago

Hello Hakan,

I have tried to use mongolastic with elasticsearch version 6.5.4 and Mongo 4.0, both as Docker containers at my local machine.

my conf file is as follow : misc: dindex: name: gfatest as: clusterdb
ctype: name: gfatest as: elas direction: em mongo: host: 127.0.0.1 port: 27017 elastic: host: 127.0.0.1 port: 9300 clusterName: docker-cluster

When I run the mongolastic container I get : `[INFO ] [2019-03-04 10:54:15] [main] [INFO]: - Config Output: {elastic=Elastic{host='127.0.0.1', port=9300, clusterName=docker-cluster, dateFormat=null, longToString=false, auth=null}, misc=Misc{batch=200, direction='em', dindex=Namespace{as='clusterdb', name='gfatest'}, ctype=Namespace{as='elas', name='gfatest'}, dropDataset=true}, mongo=Mongo{host='127.0.0.1', port=27017, query='{}', project='null', auth=null}}

[INFO ] [2019-03-04 10:54:15] [main] [INFO]: - no modules loaded [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty3Plugin] [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty4Plugin] [INFO ] [2019-03-04 10:54:15] [main] [INFO]: - loaded plugin [org.elasticsearch.xpack.XPackPlugin] [WARN ] [2019-03-04 10:54:16] [elasticsearch[client][management][T#1]] [WARN]: - Failed to find a usable hardware address from the network interfaces; using random bytes: 3f:fc:94:ab:38:2a:71:18 [INFO ] [2019-03-04 10:54:17] [main] [INFO]: - Cluster created with settings {hosts=[127.0.0.1:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='5000 ms', maxWaitQueueSize=500} [INFO ] [2019-03-04 10:54:17] [main] [INFO]: - Adding discovered server 127.0.0.1:27017 to client view of cluster [INFO ] [2019-03-04 10:54:17] [main] [INFO]: - Load duration: 2152ms Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{zMQOF4MMQ4-Ge3umdq63iA}{127.0.0.1}{127.0.0.1:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:344) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:242) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:59) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:366) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:404) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1237) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:80) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:54) at com.kodcu.provider.ElasticToMongoProvider.getCount(ElasticToMongoProvider.java:39) at com.kodcu.provider.Provider.transfer(Provider.java:16) [INFO ] [2019-03-04 10:54:17] [cluster-ClusterId{value='5c7d03d9d02e7d0001c51d28', description='null'}-127.0.0.1:27017] [INFO]: - Exception in monitor thread while connecting to server 127.0.0.1:27017 com.mongodb.MongoSocketOpenException: Exception opening socket at com.mongodb.connection.SocketStream.open(SocketStream.java:63) ~[mongolastic.jar:?] at com.mongodb.connection.InternalStreamConnection.open(InternalStreamConnection.java:115) ~[mongolastic.jar:?] at com.mongodb.connection.DefaultServerMonitor$ServerMonitorRunnable.run(DefaultServerMonitor.java:113) [mongolastic.jar:?] at java.lang.Thread.run(Thread.java:745) [?:1.8.0_121] Caused by: java.net.ConnectException: Connection refused (Connection refused) at java.net.PlainSocketImpl.socketConnect(Native Method) ~[?:1.8.0_121] at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350) ~[?:1.8.0_121] at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206) ~[?:1.8.0_121] at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188) ~[?:1.8.0_121] at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~[?:1.8.0_121] at java.net.Socket.connect(Socket.java:589) ~[?:1.8.0_121] at com.mongodb.connection.SocketStreamHelper.initialize(SocketStreamHelper.java:57) ~[mongolastic.jar:?] at com.mongodb.connection.SocketStream.open(SocketStream.java:58) ~[mongolastic.jar:?] ... 3 more at com.kodcu.main.Mongolastic.proceedService(Mongolastic.java:69) at java.util.Optional.ifPresent(Optional.java:159) at com.kodcu.main.Mongolastic.start(Mongolastic.java:58) at com.kodcu.main.Mongolastic.main(Mongolastic.java:35)`

I have tried version 1.4.2 and latest version of mongolastic but the result remain the same. I don't know what I'm doing wrong, so I would really appreciate your help !

Thank you in advance.

ozlerhakan commented 5 years ago

Hi @ChuOue ,

It says that NoNodeAvailableException[None of the configured nodes are available:. Do you access both via terminal?

VolodymyrBiryuk commented 5 years ago

I have the same issue with MongoDB 4.0.5 and Elasticsearch 7. the config file looks like this:

misc:
    dindex:
        name: somename
    ctype:
        name: anothername
    direction: me
    batch: 1000
    dropDataset: true
mongo:
    host: 0.0.0.0
    port: 27017
    auth:                   
        user: someuser
        pwd: somepassword
        source: admin
        mechanism: scram-sha-1
elastic:
    host: 0.0.0.0
    port: 9200
    clusterName: docker-cluster

The Elasticsearch is reachable with curl 0.0.0.0:9200 so the cluster is running. Any ideas for a solution?

ozlerhakan commented 5 years ago

Hi @ChuOue ,

It should be 9300. could you try again with the number.

VolodymyrBiryuk commented 5 years ago

Unfortunately, the same error with port 9300.

ayminovitch commented 5 years ago

any solution??

KraJacob commented 5 years ago

i have the same issues with elastic 7.1.1

misc: dindex: name: referencements ctype: name: startup mongo: host: xx.xx.xx.xxx port: 27017 elastic: host: xx.xxx.xx.xxx port: 9200

$ java -jar mongolastic.jar -f config.yaml

Config Output: {elastic=Elastic{host='35.195.55.160', port=9200, clusterName=null, dateFormat=null, longToString=false, auth=null}, misc=Misc{batch=200, direction='me', dindex=Namespace{as='null', name='referencements'}, ctype=Namespace{as='null', name='startup'}, dropDataset=true}, mongo=Mongo{host='34.76.89.100', port=27017, query='{}', project='null', auth=null}}

[INFO ] [2019-06-12 17:11:52] [main] [INFO]: - no modules loaded [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.join.ParentJoinPlugin] [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty4Plugin] [INFO ] [2019-06-12 17:11:52] [main] [INFO]: - loaded plugin [org.elasticsearch.xpack.XPackPlugin] [INFO ] [2019-06-12 17:12:32] [main] [INFO]: - Cluster created with settings {hosts=[34.76.89.100:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='5000 ms', maxWaitQueueSize=500} [INFO ] [2019-06-12 17:12:32] [main] [INFO]: - Adding discovered server 34.76.89.100:27017 to client view of cluster [INFO ] [2019-06-12 17:12:32] [main] [INFO]: - No server chosen by ReadPreferenceServerSelector{readPreference=ReadPreference{name=primaryPreferred}} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=34.76.89.100:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 5000 ms before timing out [INFO ] [2019-06-12 17:12:33] [cluster-ClusterId{value='5d01328013999847ecfe0d60', description='null'}-34.76.89.100:27017] [INFO]: - Opened connection [connectionId{localValue:1, serverValue:118}] to 34.76.89.100:27017 [INFO ] [2019-06-12 17:12:33] [cluster-ClusterId{value='5d01328013999847ecfe0d60', description='null'}-34.76.89.100:27017] [INFO]: - Monitor thread successfully connected to server with description ServerDescription{address=34.76.89.100:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 10]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=117537224} [INFO ] [2019-06-12 17:12:33] [cluster-ClusterId{value='5d01328013999847ecfe0d60', description='null'}-34.76.89.100:27017] [INFO]: - Discovered cluster type of STANDALONE [INFO ] [2019-06-12 17:12:33] [main] [INFO]: - Opened connection [connectionId{localValue:2, serverValue:119}] to 34.76.89.100:27017 [INFO ] [2019-06-12 17:12:33] [main] [INFO]: - Mongo collection count: 51 [INFO ] [2019-06-12 17:12:33] [main] [INFO]: - Load duration: 47939ms Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{3oxfdTE5RVOqbAHlEOLArw}{35.195.55.160}{35.195.55.160:9200}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:360) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1247) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) at com.kodcu.service.ElasticBulkService.dropDataSet(ElasticBulkService.java:90) at com.kodcu.provider.Provider.transfer(Provider.java:21) at com.kodcu.main.Mongolastic.proceedService(Mongolastic.java:69) at java.util.Optional.ifPresent(Unknown Source) at com.kodcu.main.Mongolastic.start(Mongolastic.java:58) at com.kodcu.main.Mongolastic.main(Mongolastic.java:35)

jagadish20 commented 5 years ago

even i have the same error.

{elastic=Elastic{host='XXXXXXXXXXX', port=9300, clusterName=null, dateFormat=null, longToString=false, auth=null}, misc=Misc{batch=400, direction='me', dindex=Namespace{as='null', name='GenBot_reportingDB'}, ctype=Namespace{as='null', name='Reporting'}, dropDataset=true}, mongo=Mongo{host='10.47.32.154', port=27017, query='{}', project='null', auth=Auth [user=root, pwd=XXXXXX, source=admin, mechanism=scram-sha-1]}}

[INFO ] [2019-06-19 10:43:49] [main] [INFO]: - no modules loaded [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.index.reindex.ReindexPlugin] [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.join.ParentJoinPlugin] [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.percolator.PercolatorPlugin] [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.script.mustache.MustachePlugin] [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.transport.Netty4Plugin] [INFO ] [2019-06-19 10:43:49] [main] [INFO]: - loaded plugin [org.elasticsearch.xpack.XPackPlugin] [INFO ] [2019-06-19 10:43:51] [elasticsearch[client][generic][T#2]] [INFO]: - failed to get local cluster state for {#transport#-1}{tC6boXcfQdChO5hu7NvrBw}{rb-atm-lp3.apac.bosch.com}{10.47.35.218:9300}, disconnecting... org.elasticsearch.transport.NodeDisconnectedException: [][10.47.35.218:9300][cluster:monitor/state] disconnected [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - Cluster created with settings {hosts=[10.47.32.154:27017], mode=MULTIPLE, requiredClusterType=UNKNOWN, serverSelectionTimeout='5000 ms', maxWaitQueueSize=500} [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - Adding discovered server 10.47.32.154:27017 to client view of cluster [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - No server chosen by ReadPreferenceServerSelector{readPreference=ReadPreference{name=primaryPreferred}} from cluster description ClusterDescription{type=UNKNOWN, connectionMode=MULTIPLE, serverDescriptions=[ServerDescription{address=10.47.32.154:27017, type=UNKNOWN, state=CONNECTING}]}. Waiting for 5000 ms before timing out [INFO ] [2019-06-19 10:43:51] [cluster-ClusterId{value='5d09c48ffd384909edd4eb17', description='null'}-10.47.32.154:27017] [INFO]: - Opened connection [connectionId{localValue:1, serverValue:201477}] to 10.47.32.154:27017 [INFO ] [2019-06-19 10:43:51] [cluster-ClusterId{value='5d09c48ffd384909edd4eb17', description='null'}-10.47.32.154:27017] [INFO]: - Monitor thread successfully connected to server with description ServerDescription{address=10.47.32.154:27017, type=STANDALONE, state=CONNECTED, ok=true, version=ServerVersion{versionList=[4, 0, 9]}, minWireVersion=0, maxWireVersion=7, maxDocumentSize=16777216, roundTripTimeNanos=662969} [INFO ] [2019-06-19 10:43:51] [cluster-ClusterId{value='5d09c48ffd384909edd4eb17', description='null'}-10.47.32.154:27017] [INFO]: - Discovered cluster type of STANDALONE [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - Opened connection [connectionId{localValue:2, serverValue:201478}] to 10.47.32.154:27017 [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - Mongo collection count: 21 [INFO ] [2019-06-19 10:43:51] [main] [INFO]: - Load duration: 3946ms Exception in thread "main" NoNodeAvailableException[None of the configured nodes are available: [{#transport#-1}{tC6boXcfQdChO5hu7NvrBw}{****}{10.47.35.218:9300}]] at org.elasticsearch.client.transport.TransportClientNodesService.ensureNodesAreAvailable(TransportClientNodesService.java:347) at org.elasticsearch.client.transport.TransportClientNodesService.execute(TransportClientNodesService.java:245) at org.elasticsearch.client.transport.TransportProxyClient.execute(TransportProxyClient.java:60) at org.elasticsearch.client.transport.TransportClient.doExecute(TransportClient.java:360) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:405) at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:394) at org.elasticsearch.client.support.AbstractClient$IndicesAdmin.execute(AbstractClient.java:1247) at org.elasticsearch.action.ActionRequestBuilder.execute(ActionRequestBuilder.java:46) at com.kodcu.service.ElasticBulkService.dropDataSet(ElasticBulkService.java:90) at com.kodcu.provider.Provider.transfer(Provider.java:21) at com.kodcu.main.Mongolastic.proceedService(Mongolastic.java:69) at java.util.Optional.ifPresent(Optional.java:159) at com.kodcu.main.Mongolastic.start(Mongolastic.java:58) at com.kodcu.main.Mongolastic.main(Mongolastic.java:35)

Baom92 commented 5 years ago

Hi ! Any solution on the issue ?

ozlerhakan commented 5 years ago

Hi All,

Feel free to make a PR to resolve this problem. I have limited time so I couldn't look into it :/