search

fluent-plugins-nursery / fluent-plugin-bigquery

Other
275 stars 98 forks source link

Date partitioned table error #173

Open meoooh opened 5 years ago

meoooh commented 5 years ago

Environments

Configuration

<match applog.**>
  @type copy

  <store>
    @type stdout
  </store>

  <store>
    @type bigquery_load
    project tibo
    auth_method application_default
    dataset "mylog#{ENV['RAILS_ENV'] != 'production' ? '_alpha' : ''}"
    request_timeout_sec 300
    request_open_timeout_sec 300
    flush_interval 1800
    ignore_unknown_values true
    auto_create_table true
    time_partitioning_type day

    schema_path /fluentd/etc/bq_request_scheme.json

    <buffer>
      flush_at_shutdown true
      flush_interval 1
      timekey 60
    </buffer>

    <inject>
      localtime true
    </inject>

    table request$%Y%m%d
  </store>
</match>

Expected Behavior

I don't get error

Actual Behavior

I get some error

Log (if you have)

2019-03-20 12:38:21 +0000 [error]: fluent/log.rb:362:error: config error file="/fluentd/etc/fluent.conf" error_class=Fluent::ConfigError error="Parameter 'bigquery_insert: project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix=' has timestamp placeholders, but chunk key 'time' is not configured"
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:639:in `validate_time!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:624:in `validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:574:in `block in placeholder_validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:573:in `each'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/output.rb:573:in `placeholder_validate!'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluent-plugin-bigquery-2.1.0/lib/fluent/plugin/out_bigquery_insert.rb:61:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin.rb:164:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:74:in `block in configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:63:in `each'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/multi_output.rb:63:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin/out_copy.rb:36:in `configure'
  2019-03-20 12:38:21 +0000 [debug]: fluent/supervisor.rb:502:run_supervisor: /usr/lib/ruby/gems/2.5.0/gems/fluentd-1.4.0/lib/fluent/plugin.rb:164:in `configure'

I think some bug...?

Because str is project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix= in that context.

And str must be passed this logic, but str never passd that logic.

So can I use %d placeholder in project=tibo/dataset=mylog_alpha/table=request$%Y%m%d/fetch_schema_table=/template_suffix= ?

joker1007 commented 5 years ago

Your configuration seems not correct. Please add time to <buffer> chunk key.

like this.

<buffer time>
# 
</buffer>