search

fluent / fluent-plugin-opensearch

OpenSearch Plugin for Fluentd
Apache License 2.0
58 stars 20 forks source link

Help with ruby test setup and configuration #113

Closed renegaderyu closed 11 months ago

renegaderyu commented 1 year ago

(check apply)

Steps to replicate

While trying to add and run test cases for contributions to this repo I ran bundle exec rake test but it produces errors for most test in out_opensearch_data_stream.rb like below. I suspect there is something in environment setup that was either missed or done improperly. I'd like to figure it out and get it documented to make future contributions easier. 

====================================================================================================================
Error: test_custom_record_placeholder(OpenSearchOutputDataStreamTest): Fluent::ConfigError: Failed to create data stream: <foo_bar> unknown keyword: acknowledged
/Volumes/sourcecode/fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb:168:in `rescue in write'
/Volumes/sourcecode/fluent-plugin-opensearch/lib/fluent/plugin/out_opensearch_data_stream.rb:165:in `write'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:1225:in `try_flush'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:1538:in `flush_thread_run'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin/output.rb:510:in `block (2 levels) in start'
/Users/rmiguel/.gem/ruby/2.5.0/gems/fluentd-1.16.1/lib/fluent/plugin_helper/thread.rb:78:in `block in thread_create'
====================================================================================================================
E

Expected Behavior or What you need to ask

I'd like to be able to test locally

Using Fluentd and OpenSearch plugin versions

Gemfile.lock

PATH
  remote: .
  specs:
    fluent-plugin-opensearch (1.1.3)
      aws-sdk-core (~> 3)
      excon
      faraday (>= 2.0.0)
      faraday-excon (>= 2.0.0)
      faraday_middleware-aws-sigv4 (~> 1.0.1)
      fluentd (>= 0.14.22)
      opensearch-ruby

GEM
  remote: https://rubygems.org/
  specs:
    addressable (2.8.4)
      public_suffix (>= 2.0.2, < 6.0)
    aws-eventstream (1.2.0)
    aws-partitions (1.785.0)
    aws-sdk-core (3.177.0)
      aws-eventstream (~> 1, >= 1.0.2)
      aws-partitions (~> 1, >= 1.651.0)
      aws-sigv4 (~> 1.5)
      jmespath (~> 1, >= 1.6.1)
    aws-sigv4 (1.6.0)
      aws-eventstream (~> 1, >= 1.0.2)
    concurrent-ruby (1.2.2)
    cool.io (1.7.1)
    crack (0.4.5)
      rexml
    docile (1.4.0)
    excon (0.100.0)
    faraday (2.7.10)
      faraday-net_http (>= 2.0, < 3.1)
      ruby2_keywords (>= 0.0.4)
    faraday-excon (2.1.0)
      excon (>= 0.27.4)
      faraday (~> 2.0)
    faraday-net_http (3.0.2)
    faraday_middleware-aws-sigv4 (1.0.1)
      aws-sigv4 (~> 1.0)
      faraday (>= 2.0, < 3)
    flexmock (2.3.6)
    fluentd (1.16.1)
      bundler
      cool.io (>= 1.4.5, < 2.0.0)
      http_parser.rb (>= 0.5.1, < 0.9.0)
      msgpack (>= 1.3.1, < 2.0.0)
      serverengine (>= 2.3.2, < 3.0.0)
      sigdump (~> 0.2.2)
      strptime (>= 0.2.4, < 1.0.0)
      tzinfo (>= 1.0, < 3.0)
      tzinfo-data (~> 1.0)
      webrick (~> 1.4)
      yajl-ruby (~> 1.0)
    hashdiff (1.0.1)
    http_parser.rb (0.8.0)
    io-console (0.6.0)
    irb (1.6.3)
      reline (>= 0.3.0)
    jmespath (1.6.2)
    minitest (5.18.1)
    msgpack (1.7.1)
    multi_json (1.15.0)
    oj (3.14.2)
    opensearch-api (2.2.0)
      multi_json
    opensearch-ruby (2.1.0)
      opensearch-api (~> 2.1)
      opensearch-transport (~> 2.0)
    opensearch-transport (2.1.0)
      faraday (>= 1.0, < 3)
      multi_json
    power_assert (2.0.3)
    public_suffix (5.0.1)
    rake (13.0.6)
    reline (0.3.6)
      io-console (~> 0.5)
    rexml (3.2.5)
    ruby2_keywords (0.0.5)
    serverengine (2.3.2)
      sigdump (~> 0.2.2)
    sigdump (0.2.5)
    simplecov (0.22.0)
      docile (~> 1.1)
      simplecov-html (~> 0.11)
      simplecov_json_formatter (~> 0.1)
    simplecov-html (0.12.3)
    simplecov-lcov (0.8.0)
    simplecov_json_formatter (0.1.4)
    strptime (0.2.5)
    test-unit (3.3.9)
      power_assert
    tzinfo (2.0.6)
      concurrent-ruby (~> 1.0)
    tzinfo-data (1.2023.3)
      tzinfo (>= 1.0.0)
    webmock (3.18.1)
      addressable (>= 2.8.0)
      crack (>= 0.3.2)
      hashdiff (>= 0.4.0, < 2.0.0)
    webrick (1.7.0)
    yajl-ruby (1.4.3)

PLATFORMS
  arm64-darwin-22

DEPENDENCIES
  flexmock (~> 2.0)
  fluent-plugin-opensearch!
  irb
  minitest (~> 5.8)
  oj
  rake
  simplecov
  simplecov-lcov
  strptime
  test-unit (~> 3.3.0)
  webmock (~> 3)
  webrick (~> 1.7.0)

BUNDLED WITH
   2.3.26
renegaderyu commented 1 year ago

So I thought this was environment config (it may still be). But I realized that if I make the changes below that I can test locally and do not receive the errors mentioned above

diff --git a/test/plugin/test_out_opensearch_data_stream.rb b/test/plugin/test_out_opensearch_data_stream.rb
index 872bf75..86e30ab 100644
--- a/test/plugin/test_out_opensearch_data_stream.rb
+++ b/test/plugin/test_out_opensearch_data_stream.rb
@@ -57,24 +57,24 @@ class OpenSearchOutputDataStreamTest < Test::Unit::TestCase
     {'@timestamp' => Time.now.iso8601, 'message' => 'Sample record'}
   end

-  RESPONSE_ACKNOWLEDGED = {"acknowledged": true}
+  RESPONSE_ACKNOWLEDGED = '{"acknowledged": true}'
   DUPLICATED_DATA_STREAM_EXCEPTION = {"error": {}, "status": 400}
   NONEXISTENT_DATA_STREAM_EXCEPTION = {"error": {}, "status": 404}

   def stub_index_template(name="foo_tpl", url="http://localhost:9200")
-    stub_request(:put, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:put, "#{url}/_index_template/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end

   def stub_data_stream(name="foo", url="localhost:9200")
-    stub_request(:put, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:put, "#{url}/_data_stream/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end

   def stub_existent_data_stream?(name="foo", url="localhost:9200")
-    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:get, "#{url}/_data_stream/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end

   def stub_existent_template?(name="foo_tpl", url="localhost:9200")
-    stub_request(:get, "#{url}/_index_template/#{name}").to_return(:status => [200, RESPONSE_ACKNOWLEDGED])
+    stub_request(:get, "#{url}/_index_template/#{name}").to_return(status: 200, body: RESPONSE_ACKNOWLEDGED)
   end

   def stub_nonexistent_data_stream?(name="foo", url="localhost:9200")
24367dfa commented 1 year ago

i managed to get the test setup running in both ubuntu and wsl2 on windows following the instructions given in the readme.

cosmo0920 commented 11 months ago

Already resolved at master.