logstash-plugins / logstash-input-jdbc

Logstash Plugin for JDBC Inputs
Apache License 2.0
449 stars 187 forks source link

Cannot serialize instance of: Sequel::SQL::Blob #177

Open guyboertje opened 7 years ago

guyboertje commented 7 years ago

Brought over from: https://github.com/elastic/logstash/issues/6112

@sempervictus tried to recreate the setup in this article: https://www.elastic.co/blog/visualising-oracle-performance-data-with-the-elastic-stack

There is an unserializable object in the event from this input.

Clearly the blog post worked for: Elasticsearch 2.2, Logstash 2.2, Kibana 4.4, pulling data from Oracle 12c (12.1.0.1.0)

input {
    jdbc {
        jdbc_validate_connection => true
        jdbc_connection_string => "jdbc:oracle:thin:@oradb:1521/orcl"
        jdbc_user => "system"
        jdbc_password => "Admin123"
        jdbc_driver_library => "/opt/ojdbc7.jar"
        jdbc_driver_class => "Java::oracle.jdbc.driver.OracleDriver"
        statement => "SELECT sysdate from dual"
       }
}
output   {
    stdout { codec => rubydebug }
}
ldh2068vip commented 6 years ago

hi, @guyboertje ,now I need hand oracle blob field process, I find out this issue still open , so, is that any function to solve ?

syuukawa commented 6 years ago

Hi , [error_message=>"Cannot serialize instance of: Sequel::SQL::Blob] ,What can i do to solve this problem?

guyboertje commented 6 years ago

@ldh2068vip @syuukawa

Please help me understand why you want to include Blob content in a Logstash event?

sempervictus commented 6 years ago

Hack around it by (compressing and) b64 encoding? :)

ldh2068vip commented 6 years ago

@guyboertje our product is a data switch service, just for our gov ,we process most data as text ,so my choice is logstash and kafka,but still a parttion are blob and binary type from rdms,so we want process them via logstash too。now,I use this plugin with rubydebug output,I found print Sequel::SQL::Blob project to str,as bellow: _20180827095802 the field image as blob type

ldh2068vip commented 6 years ago

@sempervictus I tried b64 encoding fun in Oracle,but limit 32k while decode into blob ,

I solved this as encode in source Oracle,then decode in logstash-output-jdbc and insert into my target Oracle。thanks all