Support gradle parallel in test tasks

[sbayer55]

Is your feature request related to a problem? Please describe. The following gradle tasks fail intermittently when using the --parallel flag

• test
• :e2e-test:log:basicLogEndToEndTest
• :e2e-test:trace:rawSpanEndToEndTest
• :e2e-test:trace:rawSpanCompatibilityEndToEndTest
• :e2e-test:trace:serviceMapEndToEndTest

Describe the solution you'd like

• Refactor tests that rely on a running data prepper instance to use a random open port
• Refactor integration tests to use a shared instance of OpenSearch. Each test can use a different index to prevent interference.
• Refactor gradle tasks that build a Data Prepper image for testing to run a base image then mount any jars and configuration files at runtime.

Describe alternatives you've considered (Optional) Run integrations test code in a new container to allow multiple tests to run simultaneously.

Additional context n/a

[dlvenable]

I definitely like the first two solutions.

Could you provide some additional details on the third?

[sbayer55]

Currently, e2e gradle tasks build an image containing data-prepper-core-1.3.0-SNAPSHOT.jar and then run that image. Roughly equivalent to

docker build -t e2e-test-log-pipeline-image --file $DATA_PREPPER_HOME/e2e-test/log/build/docker/Dockerfile .
docker run --rm -p 2021:2021 -p 4900:4900 --network data_prepper_network e2e-test-log-pipeline-image

In the third solution I propose that be changed to

docker run \
    --rm \
    -p 2021:2021 \
    -p 4900:4900 \
    --network data_prepper_network \
    -v $DATA_PREPPER_HOME/e2e-test/log/build/libs/data-prepper-core-1.3.0-SNAPSHOT.jar:/app/data-prepper.jar \
    -v $DATA_PREPPER_HOME/e2e-test/log/src/integrationTest/resources/basic-grok-e2e-pipeline.yml /app/basic-grok-e2e-pipeline.yml \
    -v $DATA_PREPPER_HOME/e2e-test/log/src/integrationTest/resources/data_prepper.yml /app/data_prepper.yml \
    --entrypoint "java -jar, data-prepper.jar, /app/basic-grok-e2e-pipeline.yml, /app/data_prepper.yml"
    adoptopenjdk/openjdk14:jre-14.0.1_7-alpine

This would allow e2e tests to skip building a new container each time one is run.

[dlvenable]

I like that idea if we want to stick with Docker running Data Prepper for the end-to-end tests.

But, can these tests just run the jar file and skip building a Docker container altogether?

[sbayer55]

Good idea! Skipping running DataPrepper in an image would reduce the runtime. We already have smoke tests to verify Data Prepper images.

[cmanning09]

Refactor integration tests to use a shared instance of OpenSearch. Each test can use a different index to prevent interference.

I like this idea. One thing to note is the trace pipelines have a concept of index_type with predefined values for the index in the OpenSearch Sink plugin. We are moving away from this model. Trace pipelines are meant to only write to trace-analytics-raw and trace-analytics-service-map indices. But I think using different values for the tests will be okay