logstash-plugins / logstash-filter-jdbc_static

Loads a DB query's result set in memory and uses it as lookup table for events.
Apache License 2.0
11 stars 12 forks source link

Incorrect error while using an wrong hostname is mysql connection string #28

Closed parallelthought closed 6 years ago

parallelthought commented 6 years ago

Plugin version: 1.0.3 Logstash: 6.2

Currently i'm trying to solve an issue mentioned at https://discuss.elastic.co/t/error-connecting-to-mysql-using-jdbc-static-filter/132163

While investigating the above issue, I observed that the plugin logs the same error when non existing hostname or IP is provided.

The mentioned error in the log is: Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::Filters::Jdbc::ConnectionJdbcException: Java::ComMysqlJdbcExceptionsJdbc4::MySQLSyntaxErrorException: Unknown database 'testdb'>

I've confirmed that the database mentioned has already been created. However even with the correct hostname or IP, I'm still getting the above error.

Is there a way to get more logs for a better clarity on the real issue ?

My config is mentioned below;

filter {
      jdbc_static {
        id => "data_enhancement"
        loaders => [
            {
                id => "test_loader"
                query => "SELECT * FROM data_dump.testdb"
                local_table => "testdbLocal"
            }
        ]
        local_db_objects => [
            {
                name => "testdbLocal"
                columns => [
                    ["dept_id", "int"],
                    ["acc_country_name", "varchar(255)"]
                ]
            }
        ]   
        local_lookups => [ 
            {
                id => "local-servers"
                query => "SELECT * FROM testdbLocal WHERE dept_id = :groupName"
                parameters => { groupName => "[group_name]"}
                target => "enhanced_info"
            }
        ]
        jdbc_user => "admin"
        jdbc_password => "XXX"
        jdbc_driver_class => "com.mysql.jdbc.Driver"
        jdbc_driver_library => "/usr/share/logstash/vendor/mysql-connector-java-5.1.46-bin.jar"
        jdbc_connection_string => "jdbc:mysql://127.0.0.1:3306/testdb"
    }
}
parallelthought commented 6 years ago

It was observed that the jdbc connection string had the table name in the end instead of the database. This was corrected and also appropriate privileges were provided to the user. Later the issue was resolved.

When a non exiting IP (Eg: 99.99.99.99) is provided, the pipeline times out and logs the below error after the timeout period

[2018-05-17T11:31:38,867][ERROR][logstash.pipeline        ] Pipeline aborted due to error {:pipeline_id=>"main", :exception=>#<LogStash::Filters::Jdbc::ConnectionJdbcException: Java::ComMysqlJdbcExceptionsJdbc4::CommunicationsException: Communications link failure

The last packet sent successfully to the server was 0 milliseconds ago. The driver has not received any packets from the server.>, :backtrace=>["java.lang.reflect.Constructor.newInstance(java/lang/reflect/Constructor.java:423)", "com.mysql.jdbc.Util.handleNewInstance(com/mysql/jdbc/Util.java:425)"