logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Oracle JDBC connection idle timeout issues: How to validate a connection (ex: sequel_opts in Oracle?) #322

Open stanuku opened 5 years ago

stanuku commented 5 years ago

I'm indexing data from Oracle DB to Elasticsearch using logstash.

The Logstash process always terminates (after about ~ 20 mins), with the below error: Sequel::DatabaseError: Java::JavaSql::SQLException: ORA-02396: exceeded maximum idle time

Is there a way I could have the jdbc connection validation from Sequel to work for Oracle? How do we use sequel_opts in the Logstash input config?

jdbc_validate_connection setting in logstash config doesn't seem to help.`

input {
    jdbc {
        clean_run => false
        jdbc_driver_library => "E:\logstash\ojdbc8.jar"
        jdbc_driver_class => "Java::oracle.jdbc.OracleDriver"
        jdbc_connection_string => "jdbc:oracle:thin:@xxxxx:1234/xyz"
        ...
        #-> Connection pool configuration. Validate connection before use.
        **jdbc_validate_connection => true**
        #->How often to validate a connection (in seconds)
        jdbc_validation_timeout => 120
        #->**??** jdbc_validation_query => "SELECT 1 FROM DUAL" **??**
        #->Maximum number of times to try connecting to database
        connection_retry_attempts => 5
        #-> Number of seconds to sleep between connection attempts
        connection_retry_attempts_wait_time => 2        

                #https://github.com/jeremyevans/sequel/blob/master/doc/opening_databases.rdoc)#
        sequel_opts =>
        {
            login_timeout => "60"
            prefetch_rows => "1000"
            jdbc_properties => #=> **Are these valid for Oracle JDBC ??** -#
            {
                "defaultRowPrefetch" => "1000"
                "loginTimeout" => "60"
                "inactiveConnectionTimeout" => "120"
                "timeoutCheckInterval" => "120"
                "tcpKeepAlive" => "true"
                "oracle.net.READ_TIMEOUT" => "5000"
                "validationQuery" => "SELECT 1 FROM DUAL"
            }
        }
    ...
    }
output {
    elasticsearch {
        hosts => ["http://localhost:9200"]
        index => "projects"
        document_id => "%{project_id}"
        document_type => "_doc"
        doc_as_upsert => true
        http_compression => true
    }
    stdout {
        codec => rubydebug
    }
}