How to test nested fields

antoniocascais commented 5 years ago

Hello.

I'm trying to use logstash-devutils to test my logstash configurations. I manage to successfully do it for simple scenarios, so that's good :+1:

Now one of my logstash config files has an if like there:

if ( [@type] == "LogMessage" and [@source][type] == "ssh" ) {
        mutate {
            add_field => { "x" => "a" }
        }
} else {
        mutate {
            add_field => { "x" => "b" }
        }
}

I'm trying to write an unit test for it and I wrote

  sample("@message" => message, "@type" => "LogMessage", "[@source][type]" => "ssh") do
    # Check the ouput event/message properties
    insist { subject.get("x") } == "a"

This fails because value of x is b.

I tried changing the sample from "[@source][type]" => "ssh" to "@source.type" => "ssh" but it's the same result.

I had a look at the code and I guess that [@source][type] will be interpreted as an array here

  def sample(sample_event, &block)
    name = sample_event.is_a?(String) ? sample_event : LogStash::Json.dump(sample_event)
    name = name[0..50] + "..." if name.length > 50

    describe "\"#{name}\"" do
      let(:pipeline) { new_pipeline_from_string(config) }
      let(:event) do
        sample_event = [sample_event] unless sample_event.is_a?(Array)
        next sample_event.collect do |e|
          e = { "message" => e } if e.is_a?(String)
          next LogStash::Event.new(e)
        end
      end

Therefore it will be ignored.

My knowledge of ruby is pretty much 0, so I can't help much here. My question is: am I doing something wrong? Is there a way to make that if to be true, other than changing the config file itself?

I don't know if this project is still actively monitored, but thanks in advance anyway! :crossed_fingers:

antoniocascais commented 5 years ago

We found a workaround to this issue, will post here the solution.

First of all, we successfully confirmed that there is a bug when passing fields using the sample method: sample("@message" => message, "[@source][type]" => "ssh") do This will set 2 fields in the event: the field @message and the field type. Notice that it does NOT set the field [@source][type] as epxected, but type.

My theory is that this happens due to the logstash event gem being super outdated. But it's just a theory.

So, the workaround was changing the sample method in the logstash_helpers.rb file:

     describe "\"#{name}\"" do
      let(:pipeline) { new_pipeline_from_string(config) }
      let(:event) do
        sample_event = [sample_event] unless sample_event.is_a?(Array)
        next sample_event.collect do |e|
          e = { "message" => e } if e.is_a?(String)
      evt = LogStash::Event.new(e)

      # Workaround to be able to set the field [@source][type] in our tests
      source_type = evt.get('type')
      evt.set('[@source][type]', source_type)

          next evt
        end
      end

Notice the lines

      # Workaround to be able to set the field [@source][type] in our tests
      source_type = evt.get('type')
      evt.set('[@source][type]', source_type)

In our case we just need to set the [@source][type] field, so it's enough to read the type field and set [@source][type] explicitly in the sample method.

Then we copy this file into a docker container that we are using:

# Work around to be able to test nested fields like [@source][type]
ADD custom_logstash_helpers.rb \
/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-devutils-1.3.5-java/lib/logstash/devutils/rspec/logstash_helpers.rb

It's important to note that we got ideas from 3 different authors in order to set up our testing framework:

If anyone has any questions regarding this, feel free to ask.

kares commented 4 years ago

cool, closing as this does not look like a bug (the helper assumes some level of LS knowledge). what we could do is improve the docs for the sample helper with examples - PRs welcome!

elastic / logstash-devutils

How to test nested fields #78