search

jprante / elasticsearch-jdbc

JDBC importer for Elasticsearch
Apache License 2.0
2.84k stars 709 forks source link

using elastic search 1.6 with JDBC importer for Elasticsearch #563

Open dgrene opened 9 years ago

dgrene commented 9 years ago

getting Error: Could not find or load main class org.xbib.tools.Runner

guyajun commented 9 years ago

!/bin/sh

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" bin=${DIR}/../bin lib=${DIR}/../lib

echo '{ "type" : "jdbc", "jdbc" : { "driver": "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/student", "user" : "root", "password" : "123", "sql" : "select _, id as _id from user", "sql.write" : "true", "strategy":"standard", "interval":"5", "bulk_size": 100, "max_bulk_requests": 30, "bulk_timeout": "10s", "flushinterval":"5s", "index":"mysql", "type":"mytype" } } ' | java \ -cp "${lib}/" \ -Dlog4j.configurationFile=file://localhost/C:/elasticsearch-jdbc-1.6.0.0-dist/elasticsearch-jdbc-1.6.0.0/bin/log4j2.xml \ org.xbib.tools.Runner \ org.xbib.tools.JDBCImporter

dgrene commented 9 years ago

thanks guyajun it is working now but what about realtime update elastic search

ApurvaHP commented 9 years ago

Hi,

I want to get the JDBC importer running.

I am using a Mac and I also got the error: Could not find or load main class org.xbib.tools.Runner error.

I do not seem to follow this step Step: run script with a command that starts org.xbib.tools.JDBCImporter with the lib directory on the classpath

Can someone please elaborate on how to get the importer running. I dont find much help online.

dgrene commented 9 years ago

hi apurva, just run the above shell script with your dir path of jdbc importer like

!/bin/sh

bin=/home/deepak/elasticsearch-jdbc-1.6.0.0/bin lib=/home/deepak/elasticsearch-jdbc-1.6.0.0/lib echo '{ "type" : "jdbc", "jdbc" : { "driver": "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/bt", "user" : "root", "password" : "", "sql" : "select _,sno as _id from patient", "index" : "myindex", "type" : "mytype", "sql.write" : "true", "strategy":"standard", "interval":"5", "bulk_size": 100, "max_bulk_requests": 30, "bulk_timeout": "10s", "flushinterval":"5s" } }' | java \ -cp "${lib}/" \ -Dlog4j.configurationFile=/home/deepak/elasticsearch-jdbc-1.6.0.0/bin/log4j2.xml \ org.xbib.tools.Runner \ org.xbib.tools.JDBCImporter

vaniaravinda commented 9 years ago

Hi,

Can someone help me in executing below points 3, 5, & 6 for JDBC driver. 1.download the JDBC importer distribution a. wget http://xbib.org/repository/org/xbib/elasticsearch/importer/elasticsearch-jdbc/1.6.0.0/elasticsearch-jdbc-1.6.0.0-dist.zip  Completed 2.unpack unzip elasticsearch-jdbc-1.6.0.0-dist.zip  Completed 3.go to the unpacked directory (we call it $JDBC_IMPORTER_HOME) cd elasticsearch-jdbc-1.6.0.0  Can you please elaborate this point. Got stuck here. 4.if you do not find the JDBC driver jar in the lib directory, download it from your vendor's site and put the driver jar into the lib folder  Already jdbc-driver-csv-1.0.0 is there in lib folder. 5.modify script in the bin directory to your needs (Elasticsearch cluster address)  Can you please elaborate this point. 6.run script with a command that starts org.xbib.tools.JDBCImporter with the lib directory on the classpath  Can you please elaborate this point.

Thanks, Vani Aravinda

Jrizzi1 commented 9 years ago

I also am at a loss here

I previously was using ES 1.3.2 and JDBC river 1.3.0.4, and I would execute and store my river in _meta

and now I am really unsure on how to proceed past step 4, i modified the oracle script to suit my needs but now what?

aakshaye commented 9 years ago

I have the same issue as dgrene and ApurvaHP on Ubuntu 14.04. Do I have to change something?

edouardkombo commented 9 years ago

Also stucked for three days now, with the error: Error: Could not find or load main class org.xbib.tools.Runner

No solutions is working.

MrReda commented 9 years ago

I found out the solution ,for windows users please follow the same steps i did to make it work I'm using elastic search 1.6.0 ES jdbc importer C:\elasticsearch-jdbc-1.6.0.0\ .Tip avoid spaces in the path download cygwin for windows 7 Mysql database which has data to be imported inside Elastic Search Once cygwin has been installed change the working directory to :
/cygdrive/c/elasticsearch-jdbc-1.6.0.0/lib After that ,configure the following script to import the data from mysql to ES like the following :

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" bin=${DIR}/bin lib=${DIR}/lib echo $lib echo $bin

echo '{ "type" : "jdbc", "jdbc" : { "driver": "com.mysql.jdbc.Driver", "url" : "jdbc:mysql://localhost:3306/world", "user" : "root", "password" : "root", "sql" : "select * from city", "treat_binary_as_string" : true, "elasticsearch" : { "cluster" : "elasticsearch", "host" : "localhost", "port" : 9300 }, "max_bulk_actions" : 20000, "max_concurrent_bulk_requests" : 10, "index" : "my_index", "type":"my_type" } } ' | java \ -cp "elasticsearch-jdbc-1.6.0.0-uberjar.jar;mysql-connector-java-5.1.33.jar" \ -Dlog4j.configurationFile="file://localhost/C:/elasticsearch-jdbc-1.6.0.0/bin/log4j2.xml" \ org.xbib.tools.Runner \ org.xbib.tools.JDBCImporter Press enter to run and here you go all is done .Check now the data has been imported successfully inside elastic search within your index.

Thanks MrReda

itarafath commented 9 years ago

@MrReda it wont work. :( it reutrn Error: Could not find or load main class org.xbib.tools.Runner this error

MrReda commented 9 years ago

Please check out my recent post http://discuss.elastic.co/t/how-do-i-import-data-from-mysql-to-elastic-search-1-6/2658/26

Thanks

RajnikantBaflipara commented 8 years ago

@MrReda its working Fine but when i search data into elastic search then i get IndexMissingException[[indexname] missing]. Please Help me.

MrReda commented 8 years ago

This issue is coming because you are typing a wrong index name or missing index name which is not existing. Check that the index is created Basically its created automatically when you import the data. The correct index name will be existing inside the file that you have configured to import the data. Thanks

MrReda

On Thu, Dec 10, 2015 at 7:45 AM, Rajnikant notifications@github.com wrote:

@MrReda https://github.com/MrReda its working Fine but when i search data into elastic search then i get IndexMissingException[[indexname] missing]. Please Help me.

— Reply to this email directly or view it on GitHub https://github.com/jprante/elasticsearch-jdbc/issues/563#issuecomment-163525723 .

viks113 commented 8 years ago

I am getting below error to run above mentioned script.

17:28:27,778 ERROR [ProfileServiceBootstrap] Failed to load profile: Summary of incomplete deployments (SEE PREVIOUS ERRORS FOR DETAILS):

DEPLOYMENTS IN ERROR: Deployment "jboss:service=invoker,type=pooled" is in error due to the following reason(s): java.net.BindException: Address already in use: JVM_Bind

17:28:27,794 INFO [Http11Protocol] Starting Coyote HTTP/1.1 on http-127.0.0.1-8080 17:28:27,809 INFO [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009 17:28:27,809 INFO [ServerImpl] JBoss (Microcontainer) [5.1.0.GA (build: SVNTag=JBoss_5_1_0_GA date=200905221634)] Started in 32s:336ms

jprante commented 8 years ago

@viks113 please do not steal old issues. Your issue is not related to JDBC importer.