theangryangel / logstash-output-jdbc

JDBC output for Logstash
MIT License
256 stars 101 forks source link

Multi-table insert is not supported, please help me?And give examples,3Q #152

Closed hlyaowan closed 5 years ago

hlyaowan commented 5 years ago

Hello, I want to use type to distinguish different tables to insert data, but the error is reported, how to support this writing, the detailed code is attached

eg:

output { stdout{ codec=>rubydebug{} } if [type]=="KBS_KEYWORD"{ jdbc { driver_jar_path => "/home/logstash/logstash-7.1.1/package/mysql-connector-java-5.1.21.jar" driver_class => "com.mysql.jdbc.Driver" connection_string => "jdbc:mysql://127.0.0.1:33066/botprod?user=botprod&password=LKHptq4GPOUUHZuZ1I1f" statement => [ "INSERT INTO KBS_KEYWORD_BAK (ID,ORG_ID,KEYWORD_NAME,KEYWORD_PINYIN,ABBREV,SYNONYMS,NEAR,TYPOS,HYPERNYM,HYPONYM,RELATED,DOMAIN,TYPE,DEFINITION,SCORE,WEIGHT,SENSITIVITY,POPULARITY,ENABLED,CREATION_DATE,CREATOR,MODIFICATION_DATE,MODIFIER) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ORG_ID =values(ORG_ID),KEYWORD_NAME =values(KEYWORD_NAME),KEYWORD_PINYIN =values(KEYWORD_PINYIN),ABBREV =values(ABBREV),SYNONYMS =values(SYNONYMS),NEAR =values(NEAR),TYPOS =values(TYPOS),HYPERNYM =values(HYPERNYM),HYPONYM =values(HYPONYM),RELATED =values(RELATED),DOMAIN =values(DOMAIN),TYPE =values(TYPE),DEFINITION =values(DEFINITION),SCORE =values(SCORE),WEIGHT =values(WEIGHT),SENSITIVITY =values(SENSITIVITY),POPULARITY =values(POPULARITY),ENABLED =values(ENABLED),CREATION_DATE =values(CREATION_DATE),CREATOR =values(CREATOR),MODIFICATION_DATE =values(MODIFICATION_DATE),MODIFIER =values(MODIFIER)", "ID","ORG_ID","KEYWORD_NAME","KEYWORD_PINYIN","ABBREV","SYNONYMS","NEAR","TYPOS","HYPERNYM","HYPONYM","RELATED","DOMAIN","TYPE","DEFINITION","SCORE","WEIGHT","SENSITIVITY","POPULARITY","ENABLED","CREATION_DATE","CREATOR","MODIFICATION_DATE","MODIFIER" ] } }

if [type]=="KBS_CATEGORY"{ jdbc { driver_jar_path => "/home/logstash/logstash-7.1.1/package/mysql-connector-java-5.1.36.jar" driver_class => "com.mysql.jdbc.Driver" connection_string => "jdbc:mysql://127.0.0.1:33066/botprod?user=botprod&password=LKHptq4GPOUUHZuZ1I1f" statement => [ "INSERT INTO KBS_CATEGORY_BAK (ID,ORG_ID,NAME,PARENT_ID,ENABLED,CREATION_DATE,CREATOR,MODIFICATION_DATE,MODIFIER) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ORG_ID =values(ORG_ID),NAME =values(NAME),PARENT_ID =values(PARENT_ID),ENABLED =values(ENABLED),CREATION_DATE =values(CREATION_DATE),CREATOR =values(CREATOR),MODIFICATION_DATE =values(MODIFICATION_DATE),MODIFIER =values(MODIFIER)","ID","ORG_ID","NAME","PARENT_ID","ENABLED","CREATION_DATE","CREATOR","MODIFICATION_DATE","MODIFIER" ] } }

}

theangryangel commented 5 years ago

You need one output per table, or you need to implement a stored procedure to do it for you.

If you’re getting an error, what is it?

Sent from my iPhone

On 6 Jun 2019, at 10:47, hlyaowan notifications@github.com wrote:

Hello, I want to use type to distinguish different tables to insert data, but the error is reported, how to support this writing, the detailed code is attached

eg:

output { stdout{ codec=>rubydebug{} } if [type]=="KBS_KEYWORD"{ jdbc { driver_jar_path => "/home/logstash/logstash-7.1.1/package/mysql-connector-java-5.1.21.jar" driver_class => "com.mysql.jdbc.Driver" connection_string => "jdbc:mysql://127.0.0.1:33066/botprod?user=botprod&password=LKHptq4GPOUUHZuZ1I1f" statement => [ "INSERT INTO KBS_KEYWORD_BAK (ID,ORG_ID,KEYWORD_NAME,KEYWORD_PINYIN,ABBREV,SYNONYMS,NEAR,TYPOS,HYPERNYM,HYPONYM,RELATED,DOMAIN,TYPE,DEFINITION,SCORE,WEIGHT,SENSITIVITY,POPULARITY,ENABLED,CREATION_DATE,CREATOR,MODIFICATION_DATE,MODIFIER) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ORG_ID =values(ORG_ID),KEYWORD_NAME =values(KEYWORD_NAME),KEYWORD_PINYIN =values(KEYWORD_PINYIN),ABBREV =values(ABBREV),SYNONYMS =values(SYNONYMS),NEAR =values(NEAR),TYPOS =values(TYPOS),HYPERNYM =values(HYPERNYM),HYPONYM =values(HYPONYM),RELATED =values(RELATED),DOMAIN =values(DOMAIN),TYPE =values(TYPE),DEFINITION =values(DEFINITION),SCORE =values(SCORE),WEIGHT =values(WEIGHT),SENSITIVITY =values(SENSITIVITY),POPULARITY =values(POPULARITY),ENABLED =values(ENABLED),CREATION_DATE =values(CREATION_DATE),CREATOR =values(CREATOR),MODIFICATION_DATE =values(MODIFICATION_DATE),MODIFIER =values(MODIFIER)", "ID","ORG_ID","KEYWORD_NAME","KEYWORD_PINYIN","ABBREV","SYNONYMS","NEAR","TYPOS","HYPERNYM","HYPONYM","RELATED","DOMAIN","TYPE","DEFINITION","SCORE","WEIGHT","SENSITIVITY","POPULARITY","ENABLED","CREATION_DATE","CREATOR","MODIFICATION_DATE","MODIFIER" ] } }

if [type]=="KBS_CATEGORY"{ jdbc { driver_jar_path => "/home/logstash/logstash-7.1.1/package/mysql-connector-java-5.1.36.jar" driver_class => "com.mysql.jdbc.Driver" connection_string => "jdbc:mysql://127.0.0.1:33066/botprod?user=botprod&password=LKHptq4GPOUUHZuZ1I1f" statement => [ "INSERT INTO KBS_CATEGORY_BAK (ID,ORG_ID,NAME,PARENT_ID,ENABLED,CREATION_DATE,CREATOR,MODIFICATION_DATE,MODIFIER) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?) ON DUPLICATE KEY UPDATE ORG_ID =values(ORG_ID),NAME =values(NAME),PARENT_ID =values(PARENT_ID),ENABLED =values(ENABLED),CREATION_DATE =values(CREATION_DATE),CREATOR =values(CREATOR),MODIFICATION_DATE =values(MODIFICATION_DATE),MODIFIER =values(MODIFIER)","ID","ORG_ID","NAME","PARENT_ID","ENABLED","CREATION_DATE","CREATOR","MODIFICATION_DATE","MODIFIER" ] } }

}

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

hlyaowan commented 5 years ago

this error info like this:

{ "KEYWORD_NAME" => "流程", "SENSITIVITY" => 10, "ENABLED" => 1, "DOMAIN" => 2, "CREATOR" => "郭圣凯", "ID" => 18285, "type" => "KBS_KEYWORD", "HYPONYM" => nil, "TYPE" => 0, "SCORE" => 10, "@timestamp" => 2019-06-06T12:12:01.725Z, "ORG_ID" => 800830, "HYPERNYM" => nil, "TYPOS" => nil, "DEFINITION" => nil, "@version" => "1", "WEIGHT" => 10, "RELATED" => nil, "SYNONYMS" => nil, "NEAR" => nil, "MODIFICATION_DATE" => 2019-06-06T09:14:19.000Z, "KEYWORD_PINYIN" => nil, "ABBREV" => nil, "MODIFIER" => "郭圣凯", "CREATION_DATE" => 2019-06-06T09:14:19.000Z, "POPULARITY" => 10 } warning: thread "[main]>worker0" terminated with exception (report_on_exception is true): java.lang.IllegalAccessError: tried to access class com.mysql.jdbc.EscapeProcessor from class com.mysql.jdbc.ConnectionImpl at com.mysql.jdbc.ConnectionImpl.nativeSQL(com/mysql/jdbc/ConnectionImpl.java:3950) at com.mysql.jdbc.ConnectionImpl.prepareStatement(com/mysql/jdbc/ConnectionImpl.java:4149) at com.mysql.jdbc.ConnectionImpl.prepareStatement(com/mysql/jdbc/ConnectionImpl.java:4109) at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(com/zaxxer/hikari/pool/ProxyConnection.java:317) at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(com/zaxxer/hikari/pool/HikariProxyConnection.java) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:455) at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:316) at RUBY.submit(/home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:219) at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792) at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen) at RUBY.submit(/home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:217) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.retrying_submit(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:249) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.RUBY$method$retrying_submit$0$VARARGS(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.multi_receive(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:132) at org.jruby.RubyArray.eachSlice(org/jruby/RubyArray.java:1826) at org.jruby.RubyArray.each_slice(org/jruby/RubyArray.java:1836) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.multi_receive(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:131) at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.invokeOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:128) at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.doOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:250) at org.logstash.config.ir.compiler.OutputStrategyExt$SharedOutputStrategyExt.output(org/logstash/config/ir/compiler/OutputStrategyExt.java:283) at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:118) at org.logstash.config.ir.compiler.OutputDelegatorExt.doOutput(org/logstash/config/ir/compiler/OutputDelegatorExt.java:78) at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101) at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2) at org.logstash.generated.CompiledDataset3.compute(org/logstash/generated/CompiledDataset3) at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:64) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425) at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292) at home.logstash.logstash_minus_7_dot_1_dot_1.logstash_minus_core.lib.logstash.java_pipeline.start_workers(home/logstash/logstash_minus_7_dot_1_dot_1/logstash_minus_core/lib/logstash//home/logstash/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:235) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:295) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:274) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:270) at java.lang.Thread.run(java/lang/Thread.java:748) warning: thread "[main]>worker1" terminated with exception (report_on_exception is true): java.lang.IllegalAccessError: com/mysql/jdbc/EscapeProcessor at com.mysql.jdbc.ConnectionImpl.nativeSQL(com/mysql/jdbc/ConnectionImpl.java:3950) at com.mysql.jdbc.ConnectionImpl.prepareStatement(com/mysql/jdbc/ConnectionImpl.java:4149) at com.mysql.jdbc.ConnectionImpl.prepareStatement(com/mysql/jdbc/ConnectionImpl.java:4109) at com.zaxxer.hikari.pool.ProxyConnection.prepareStatement(com/zaxxer/hikari/pool/ProxyConnection.java:317) at com.zaxxer.hikari.pool.HikariProxyConnection.prepareStatement(com/zaxxer/hikari/pool/HikariProxyConnection.java) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:425) at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:292) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.submit(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:219) at org.jruby.RubyArray.each(org/jruby/RubyArray.java:1792) at org.jruby.RubyArray$INVOKER$i$0$0$each.call(org/jruby/RubyArray$INVOKER$i$0$0$each.gen) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.submit(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:217) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.RUBY$method$submit$0$VARARGS(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.retrying_submit(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:249) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.RUBY$method$retrying_submit$0$VARARGS(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.multi_receive(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:132) at org.jruby.RubyArray.eachSlice(org/jruby/RubyArray.java:1826) at org.jruby.RubyArray.each_slice(org/jruby/RubyArray.java:1836) at home.logstash.logstash_minus_7_dot_1_dot_1.vendor.bundle.jruby.$2_dot_5_dot_0.gems.logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0.lib.logstash.outputs.jdbc.multi_receive(home/logstash/logstash_minus_7_dot_1_dot_1/vendor/bundle/jruby/$2_dot_5_dot_0/gems/logstash_minus_output_minus_jdbc_minus_5_dot_4_dot_0/lib/logstash/outputs//home/logstash/logstash-7.1.1/vendor/bundle/jruby/2.5.0/gems/logstash-output-jdbc-5.4.0/lib/logstash/outputs/jdbc.rb:131) at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.invokeOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:128) at org.logstash.config.ir.compiler.OutputStrategyExt$SimpleAbstractOutputStrategyExt.doOutput(org/logstash/config/ir/compiler/OutputStrategyExt.java:250) at org.logstash.config.ir.compiler.OutputStrategyExt$SharedOutputStrategyExt.output(org/logstash/config/ir/compiler/OutputStrategyExt.java:283) at org.logstash.config.ir.compiler.OutputStrategyExt$AbstractOutputStrategyExt.multi_receive(org/logstash/config/ir/compiler/OutputStrategyExt.java:118) at org.logstash.config.ir.compiler.OutputDelegatorExt.doOutput(org/logstash/config/ir/compiler/OutputDelegatorExt.java:78) at org.logstash.config.ir.compiler.AbstractOutputDelegatorExt.multi_receive(org/logstash/config/ir/compiler/AbstractOutputDelegatorExt.java:101) at org.logstash.generated.CompiledDataset2.compute(org/logstash/generated/CompiledDataset2) at org.logstash.generated.CompiledDataset3.compute(org/logstash/generated/CompiledDataset3) at org.logstash.execution.WorkerLoop.run(org/logstash/execution/WorkerLoop.java:64) at java.lang.reflect.Method.invoke(java/lang/reflect/Method.java:498) at org.jruby.javasupport.JavaMethod.invokeDirectWithExceptionHandling(org/jruby/javasupport/JavaMethod.java:440) at org.jruby.javasupport.JavaMethod.invokeDirect(org/jruby/javasupport/JavaMethod.java:304) at home.logstash.logstash_minus_7_dot_1_dot_1.logstash_minus_core.lib.logstash.java_pipeline.start_workers(home/logstash/logstash_minus_7_dot_1_dot_1/logstash_minus_core/lib/logstash//home/logstash/logstash-7.1.1/logstash-core/lib/logstash/java_pipeline.rb:235) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:295) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:274) at org.jruby.RubyProc.call(org/jruby/RubyProc.java:270) at java.lang.Thread.run(java/lang/Thread.java:748) [2019-06-06T20:12:03,024][ERROR][org.logstash.Logstash ] java.lang.IllegalAccessError: tried to access class com.mysql.jdbc.EscapeProcessor from class com.mysql.jdbc.ConnectionImpl [2019-06-06T20:12:03,024][ERROR][org.logstash.Logstash ] java.lang.IllegalAccessError: tried to access class com.mysql.jdbc.EscapeProcessor from class com.mysql.jdbc.ConnectionImpl

hlyaowan commented 5 years ago

hello, If I don't specify type or tag, then there will be a situation of cross-insertion table, for example, there are two tables A and B, the data of source A is inserted into target A, and the target table B is also inserted; the source B is inserted into target B. I will also insert the target A. I don't know what to do?

theangryangel commented 5 years ago

Thats coming straight out of the MySQL JDBC driver... I don't see how the plugin would be causing that, since we literally don't touch the inside of it the drivers themselves. Have you checked you're using the correct version of the JDBC driver jar for your version of Java? That's typically what caused stuff like this

hlyaowan commented 5 years ago

Does it support type option in output?

hlyaowan commented 5 years ago

I have many tables to export ,how can I do?

theangryangel commented 5 years ago

Does it support type option in output?

I don't understand the question.

I have many tables to export ,how can I do?

This plugin uses prepared statements, which for most databases I am aware of means that you cannot put multiple insert commands into the same statement - https://dev.mysql.com/doc/refman/8.0/en/prepare.html

If you have that many tables the only thing I can do is suggest that you write a stored procedure, call your stored procedure from the plugin, and do the work in the stored procedure.

Edit: See https://github.com/theangryangel/logstash-output-jdbc/issues/42 as an example of writing a stored procedure for SQL Server, and how you'd call it.

It sounds a little like you're trying to use Logstash as a general purpose ETL system? If so you may want to look at actual ETL systems like Pentaho Kettle...

hlyaowan commented 5 years ago

Does it support type option in output? type like this;

if [type]=="KBS_KEYWORD"{ jdbc { driver_jar_path => "/home/logstash/logstash-7.1.1/package/mysql-connector-java-5.1.21.jar" driver_class => "com.mysql.jdbc.Driver" }

hlyaowan commented 5 years ago

I want like this,eg: logstash-input-plugin has three table A、B、C,now I want to export logstash-output-plugin three table A‘、B’、C’ ,if I do not use type,I don't has other idea。

stale[bot] commented 5 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.