search

opensearch-project / data-prepper

OpenSearch Data Prepper is a component of the OpenSearch project that accepts, filters, transforms, enriches, and routes data at scale.
https://opensearch.org/docs/latest/clients/data-prepper/index/
Apache License 2.0
263 stars 197 forks source link

[BUG] Ingesting example doesn't work, crashes with ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink #1749

Open logunovFGP opened 2 years ago

logunovFGP commented 2 years ago

Describe the bug When you are attempting to docker run by guide or docker-compose-up it is crashed with data-prepper | 2022-09-10T23:18:28,864 [main] ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink

To Reproduce Steps to reproduce the behavior:

  1. Clone repo

  2. GOTO examples

  3. GOTO log-ingestion

  4. Follow instructions

  5. By instruction enter the command docker run --name data-prepper -v /full/path/to/log_pipeline.yaml:/usr/share/data-prepper/pipelines.yaml --network "data-prepper_opensearch-net" opensearchproject/data-prepper:latest with corresponding replacement (in my case the network name was d61fd085926e log-ingestion_opensearch-net bridge local ) image

  6. Data-Prepper crashes with exception data-prepper | 2022-09-10T23:18:28,864 [main] ERROR com.amazon.dataprepper.plugin.PluginCreator - Encountered exception while instantiating the plugin OpenSearchSink

(See full log attached) CMD.report.txt

Expected behavior Data Prepper is running correctly, you should see something similar to the following line as the latest output in your terminal.

INFO  com.amazon.dataprepper.pipeline.ProcessWorker - log-pipeline Worker: No records received from buffer

Screenshots image

Environment (please complete the following information):

Additional context I also did docker compose to reproduce the issue, behavior is the same with command line used and full path specified docker-compose.txt

graytaylor0 commented 2 years ago

Hi @logunovFGP,

Did you run docker-compose --project-name data-prepper up when starting fluent-bit and opensearch? By default, the project name is the directory path (https://docs.docker.com/compose/reference/), and docker prepends the network with this project name (in this case log-ingestion). There may be a different convention for setting the project name while running Docker compose on Windows (I found this on VSCode documentation (https://code.visualstudio.com/remote/advancedcontainers/set-docker-compose-project-name), so if that does not work then you should attach data prepper to the log-ingestion_opensearch-net created when you start fluent-bit and opensearch through docker-compose from the log-ingestion directory

docker run --name data-prepper -v /full/path/to/log_pipeline.yaml:/usr/share/data-prepper/pipelines.yaml --network "log-ingestion_opensearch-net" opensearchproject/data-prepper:latest