Environment variable per testcase, or how to simulate it

[nicolasreich]

Description

I'd like to be able to set environment variables per testcase.

Motivation

For configs depending on environment variables, test what happens when they are and when they aren't as part of the same test suite

Exemplification

I have a config, that in essence, does the following:

filter {
  if ![important_field] {
    mutate {
      add_field => {
        "[@metadata][important_field]" => "${IMPORTANTFIELD:not_set}"
      }
    }
  }
  if [@metadata][important_field] != "not_set" {
    mutate {
      add_field => {
        "important_field" => "%{[@metadata][important_field]}"
      }
    }
  } else {
    mutate {
      add_tag => ["missing_important_field"]
    }
  }
}

There are three different cases: important_field was set before, or it wasn't but the environment variable IMPORTANTFIELD is set, or neither are set. Currently, I'm only able to test 2 of those cases in one run, as environment variables are decided for the whole run.

Benefits

Test that configuration using environment variables work as expected both in their presence and in their absence.

Possible Drawbacks

From the logstash doc:

Environment variables are immutable. If you update the environment variable, you’ll have to restart Logstash to pick up the updated value.

Hence it might not be realistic to restart Logstash between test cases.

Do you have any idea how this could achieved, be it with or without a modification of the testing engine?

[magnusbaeck]

Thank you for such a clear description! Unfortunately I don't think there's a way of solving this without starting Logstash multiple times with different environments. Doing this from within LFV is possible but it would affect the configuration interface and the code that spawns Logstash. It's also not clear how this would interact with the nearly finished feature to run a persistent Logstash instance for multiple test executions. I wouldn't count on this to be implemented in the near future.

What you could do is invoke LFV multiple times with different environments and different testcase file directories as inputs:

logstash-filter-verifier ... path/to/testcases/without-env path/to/config
IMPORTANTFIELD=foo logstash-filter-verifier --keep-env IMPORTANTFIELD ... path/to/testcases/with-env path/to/config

[breml]

@nicolasreich We are currently working on releasing version 2 of LFV. There is a new feature called filter mock, that maybe could help you with this case. In #119 you can find an updated version of the README, that gives some instructions on how to use version 2 of LFV.

The filter mock allows to replace filters from the Logstash configuration on the fly during test case execution. The main purpose of this feature is to replease filters, that doe not work during test case execution (e.g. because of a callout to an 3rd party system).

In your example, you could have a filter mock to replace your first mutate filter (please note the id attribute, that is being added):

mutate {
  add_field => {
    "id" => "importantfield_lookup"
    "[@metadata][important_field]" => "${IMPORTANTFIELD:not_set}"
  }
}

with

mutate {
  add_field => {
    "id" => "importantfield_lookup"
    "[@metadata][important_field]" => "fixed value"
  }
}

The respective config file for usage with LFV would look like this:

---

- id: mockme
  mutate {
    add_field => {
      "id" => "importantfield_lookup"
      "[@metadata][important_field]" => "fixed value"
    }
  }

You could also take this a step further and use a mock filter like this:

translate {
  "id" => "importantfield_lookup"
  "field" => "[@metadata][important_field_lookup_key]"
  "destination" => "[@metadata][important_field]"
  "dictionary" => {
    "set" => "important value"
  }
  "fallback" => "not_set"
}

With this config, you have the possibility to control the value for IMPORTANTFIELD for every test case by setting the respective value in [@metadata][important_field_lookup_key] (on test case fields is also a new feature of version 2).

So this boils down to replacing the lookup for the environment variable by a in memory hash table during test execution, where you can control, which value should be used for each test case.

[breml]

@nicolasreich FYI, v2.0.0-beta.1 is out, have a look at https://github.com/magnusbaeck/logstash-filter-verifier/releases/tag/v2.0.0-beta.1

[nicolasreich]

@breml thanks for the advice and for the 2.0 notification! I will look into it soon (I'm just back from holiday), I'm looking forward to testing with multiple pipelines, this will be solve so many problems! (And also probably require a re-design of the testing infra, but well)

[nicolasreich]

Mocks indeed offer a fine solution, and seeing as it's impossible to change environment variables without restarting Logstash, I guess this issue could be closed.

[breml]

@nicolasreich Maybe you could add this solution to the documentation as well.