search

daichirata / fluent-plugin-gcs

Google Cloud Storage output plugin for Fluentd.
Other
42 stars 19 forks source link

Failing to upload due to Google Storage Library Updates #21

Open joshughes opened 2 years ago

joshughes commented 2 years ago 
2022-08-15 20:04:00 +0000 [warn]: #0 [out_gcs] got unrecoverable error in primary and no secondary error_class=ArgumentError error="wrong number of arguments (given 3, expected 1..2)"
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/google-cloud-storage-1.38.0/lib/google/cloud/storage/bucket.rb:1557:in `create_file'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/out_gcs.rb:134:in `block in write'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/gcs/object_creator.rb:39:in `block in create'
  2022-08-15 20:04:00 +0000 [warn]: #0 /usr/local/lib/ruby/3.1.0/tempfile.rb:358:in `create'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/gcs/object_creator.rb:35:in `create'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/out_gcs.rb:123:in `write'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1180:in `try_flush'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:1501:in `flush_thread_run'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'
  2022-08-15 20:04:00 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
2022-08-15 20:04:00 +0000 [warn]: #0 [out_gcs] bad chunk is moved to /tmp/fluent/backup/worker0/out_gcs/5e64d1e3550ff0cf7802d2dbc6c511cc.log

Seeing this issue tyring to use fluent/fluentd-kubernetes-daemonset:v1.15-debian-gcs-1

Seems like the google libraries maybe are not locked and they expect fewer arguments...

daichirata commented 2 years ago

Thank you for your report. The cause of this error was that the automatic conversion of hashes to keyword arguments was deprecated starting with Ruby 2.7 and removed starting with Ruby 3.0.

I will create a PR to fix this issue.

pnutmath commented 1 year ago

@daichirata still having similar issue with v1-debian-gcs

2023-01-05 07:52:59 +0000 [warn]: #0 [out_gcs] got unrecoverable error in primary and no secondary error_class=ArgumentError error="wrong number of arguments (given 2, expected 1)"     │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/google-cloud-storage-1.43.0/lib/google/cloud/storage/bucket.rb:1314:in `file'                              │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/out_gcs.rb:156:in `check_object_exists'                          │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/out_gcs.rb:180:in `generate_path'                                │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluent-plugin-gcs-0.4.1/lib/fluent/plugin/out_gcs.rb:121:in `write'                                        │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1180:in `try_flush'                                             │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:1501:in `flush_thread_run'                                      │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin/output.rb:501:in `block (2 levels) in start'                              │
│   2023-01-05 07:52:59 +0000 [warn]: #0 /fluentd/vendor/bundle/ruby/3.1.0/gems/fluentd-1.15.3/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'                           │
│ 2023-01-05 07:52:59 +0000 [warn]: #0 [out_gcs] bad chunk is moved to /tmp/fluent/backup/worker0/out_gcs/5f17f532dd1e11863e3da33cea4bcc88.log