How to test ingest pipeline in Beats

[zmoog]

I want to run the Beats ingest pipeline tests locally on my development machine, so I can iterate on changes better.

[zmoog]

I am using Filebeat as an example.

First, walk into the Filebeat root directory:

cd x-pack/filebeat/

In general you can use the following command:

TESTING_FILEBEAT_MODULES=<module> \
TESTING_FILEBEAT_FILESETS=<fileset> \
MODULES_PATH=module \
mage -v pythonIntegTest

For example, here's the command line to test the activitylogs fileset in the azure module:

TESTING_FILEBEAT_MODULES=azure \
TESTING_FILEBEAT_FILESETS=activitylogs \
MODULES_PATH=module \
mage -v pythonIntegTest

Additional environment variables:

# To rebuild the expected logs.
GENERATE=1

# To skip everything but the modules test.
PYTEST_ADDOPTS="-k test_modules"
PYTEST_ADDOPTS="-k test_modules -vv"

[zmoog]

Here's how to generate the expected files for the pipeline:

TESTING_FILEBEAT_MODULES=azure \
TESTING_FILEBEAT_FILESETS=activitylogs \
MODULES_PATH=module \
PYTEST_ADDOPTS="-k test_modules -vv" \
GENERATE=1 \
mage -v pythonIntegTest

[zmoog]

Test using a different stack version

If you want to use a different version of the stack in you tests, you can:

1. create a new environment file
2. edit the environment settings
3. run the tests

Here's the actual commands:

# start from an existing environment
cp testing/environments/snapshot.yml testing/environments/my-env.yml

Then edit image version:

# This should start the environment with the latest snapshots.

version: '2.3'
services:
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:8.7.0-61c35791-SNAPSHOT

Finally, run the tests using the TESTING_ENVIRONMENT env variable:

TESTING_ENVIRONMENT=my-env GENERATE=1 INTEGRATION_TESTS=1 BEAT_STRICT_PERMS=false TESTING_FILEBEAT_MODULES=iis TESTING_FILEBEAT_FILESETS=access mage pythonintegtest -v

[zmoog]

Stack versions are updated automatically with PRs like https://github.com/elastic/beats/pull/34589