search

fluent / fluent-plugin-sql

SQL input/output plugin for Fluentd
107 stars 58 forks source link

send log to clickhouse db #80

Open zamanimehdi opened 4 years ago

zamanimehdi commented 4 years ago

hello, I want to send event to clickhouse, That's why I install fluentd and fluent-plugin-sql. then i create table in clickhouse :

CREATE TABLE zamanitable1
(
    dt DateTime,
    ComputerName String, 
    RecordNumber Int32,
    EventType Int32
)
ENGINE = MergeTree
PARTITION BY toYYYYMM(dt)
ORDER BY dt

and install this gem:

gem install clickhouse-activerecord

my fluentd config :

<match **>
  @type sql
  host *.*.*.*
  port 1317
  database system
  adapter clickhouse
  username default
  password ******
  socket 1317
  <table>
    table zamanitable1
    column_mapping 'TimeGenerated:dt,ComputerName:ComputerName,RecordNumber:RecordNumber,EventType:EventType'
  </table>
</match>

i get this error on fluentd cli: 

2020-03-27 07:51:16 +0000 [warn]: #0 fluent/log.rb:342:warn: got unrecoverable error in primary and no secondary error_class=NoMethodError error="undefined method `to_sym' for false:FalseClass"
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:664:in `map'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:664:in `import_helper'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb:526:in `bulk_import'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:98:in `import'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:238:in `block in write'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/activerecord-5.2.4.2/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-sql-1.1.1/lib/fluent/plugin/out_sql.rb:231:in `write'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1123:in `try_flush'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:1423:in `flush_thread_run'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin/output.rb:452:in `block (2 levels) in start'
  2020-03-27 07:51:16 +0000 [warn]: #0 plugin/output.rb:452:block (2 levels) in start: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.3.2/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2020-03-27 07:51:16 +0000 [warn]: #0 fluent/log.rb:342:warn: bad chunk is moved to /tmp/fluent/backup/worker0/object_2ae0428ba298/5a1d15a2bc09ab3dcc3ca663644cb763.log

fluentd has successfully connected to clickhouse (Error message when table not exists) but cannot write to db, where is the problem from?

zamanimehdi commented 4 years ago

I found a line of code that caused the error file: /usr/lib/ruby/gems/2.5.0/gems/activerecord-import-0.28.2/lib/activerecord-import/import.rb

...
      # Force the primary key col into the insert if it's not on the list and we are using a sequence and stuff a nil value for it into each row so the sequencer will fire later
      symbolized_column_names = Array(column_names).map(&:to_sym)
      symbolized_primary_key = _Array(primary_key).map(&:to_sym)  # this line
...
zamanimehdi commented 4 years ago

I solved the problem by change the out_sql code and add primery key in it. my fluentd config change to :

<match **>
  @type sql
  host *.*.*.*
  port 1317
  database system
  adapter clickhouse
  username default
  password ******
  socket 1317
  <table>
    table zamanitable1
    primary_key dt 
    column_mapping 'TimeGenerated:dt,ComputerName:ComputerName,RecordNumber:RecordNumber,EventType:EventType'
  </table>
</match>