[BUG] Docker container does not stop running after Data Prepper shuts down

[kartg]

Describe the bug I'm using the Docker version of Data Prepper (opensearch-data-prepper:2.4.0-SNAPSHOT) built from source (./gradlew :dockerSolution:buildDockerImages). When using a pipeline that includes an Opensearch source, invoking Data Prepper's shutdown API appears to terminate the Data Prepper server but does not stop the Docker container.

Snippet of Data Prepper logs:

...
Found openjdk version  of 17.0
2023-08-10T23:16:46,589 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as source component for the pipeline [test-pipeline]
....
2023-08-10T23:16:46,940 [main] INFO  org.opensearch.dataprepper.parser.PipelineParser - Building [opensearch] as sink component
....
2023-08-10T23:16:47,223 [main] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Initiating pipeline execution
....
2023-08-10T23:16:47,488 [main] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server running at :4900
....
2023-08-10T23:16:48,143 [test-pipeline-sink-worker-2-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Submitting request to initiate the pipeline processing
....
2023-08-10T23:16:51,810 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Received shutdown signal with processor shutdown timeout PT30S and sink shutdown timeout PT30S. Initiating the shutdown process
....
2023-08-10T23:17:21,813 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Shutting down processor process workers.
....
2023-08-10T23:17:23,026 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Shutting down sink process workers.
2023-08-10T23:17:23,027 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.Pipeline - Pipeline [test-pipeline] - Pipeline fully shutdown.
2023-08-10T23:17:23,057 [pool-4-thread-1] INFO  org.opensearch.dataprepper.pipeline.server.DataPrepperServer - Data Prepper server stopped

Command line:

$ docker ps
CONTAINER ID   IMAGE                                    COMMAND                  CREATED          STATUS          PORTS                    NAMES
08319e699cae   opensearch-data-prepper:2.4.0-SNAPSHOT   "/entrypoint.sh bin/…"   23 minutes ago   Up 22 minutes   0.0.0.0:4900->4900/tcp   epic_jackson
$ docker top 0831
UID                 PID                 PPID                C                   STIME               TTY                 TIME                CMD
root                48665               48639               3                   23:16               ?                   00:00:43            java -Dlog4j.configurationFile=/usr/share/data-prepper/config/log4j2-rolling.properties -Ddata-prepper.dir=/usr/share/data-prepper -cp /usr/share/data-prepper/lib/* org.opensearch.dataprepper.DataPrepperExecute
root                49046               48639               0                   23:19               ?                   00:00:00            /bin/bash

Pipeline configuration:

test-pipeline:
  sink:
  - opensearch:
      bulk_size: 2
      document_id_field: getMetadata("opensearch-document_id")
      hosts:
      - [redacted]
      index: ${getMetadata("opensearch-index")}
      password: [redacted]
      username: [redacted]
  source:
    opensearch:
      disable_authentication: true
      hosts:
      - [redacted]
      indices:
        exclude:
        - index_name_regex: \.*
        include:
        - index_name_regex: [redacted]
        - index_name_regex: [redacted]

This is possibly an issue only with the opensearch source plugin - I used the following pipeline and the Docker container stopped correctly (the sink endpoint in both cases is the same).

http-test-pipeline:
    source:
        http:
    sink:
        - opensearch:
            hosts: [redacted]
            username: [redacted]
            password: [redacted]
            index: "httptest"

To Reproduce See above

Expected behavior The Docker container stops after the Data Prepper server has shut down

Screenshots N/A

Environment (please complete the following information):

• OS: MacOS 12.6.6
• Version: Data Prepper 2.4.0-SNAPSHOT

Additional context N/A

[kartg]

Additional info:

After the Data Prepper server stopped log line above, API calls to the Docker process are not responsive i.e. there is no longer a process listening on the port

I've also attached a jstack dump that i took on the Docker container: jstack_3141.txt

[kartg]

I was able to repro the same behavior outside Docker (using the binary from ./gradlew build). Pipeline configuration remained the same.

jstack dump - jstack_dump_2.txt

[dlvenable]

I'd check to see if the opensearch source has any background threads that might not be shut down. By default, Java threads are not daemon threads, so these may interrupt the shut down.