search

opensearch-project / opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch
https://opensearch.org/docs/latest/benchmark/
Apache License 2.0
111 stars 78 forks source link

[Bug]: OSB 1.9.0 Docker Image fails when mounting file #636

Closed IanHoang closed 2 months ago

IanHoang commented 2 months ago

Describe the bug

When users attempt to mount a file to OSB 1.9.0 Docker Image, it fails with a permissions error

sudo docker run -v ./home/benchmark.ini:/opensearch-benchmark/.benchmark/benchmark.ini opensearchproject/opensearch-benchmark:1.9.0 execute-test --target-hosts https://search-hoangia-test-ee-btik57xti77rhypkon4pourlfe.us-east-1.es.amazonaws.com/ --client-options=basic_auth_user:'xxxxx',basic_auth_password:'xxxxx' --pipeline benchmark-only --kill-running-processes --workload percolator --workload-params '{"number_of_shards":"1","number_of_replicas":"0"}'

   ____                  _____                      __       ____                  __                         __
  / __ \____  ___  ____ / ___/___  ____ ___________/ /_     / __ )___  ____  _____/ /_  ____ ___  ____ ______/ /__
 / / / / __ \/ _ \/ __ \\__ \/ _ \/ __ `/ ___/ ___/ __ \   / __  / _ \/ __ \/ ___/ __ \/ __ `__ \/ __ `/ ___/ //_/
/ /_/ / /_/ /  __/ / / /__/ /  __/ /_/ / /  / /__/ / / /  / /_/ /  __/ / / / /__/ / / / / / / / / /_/ / /  / ,<
\____/ .___/\___/_/ /_/____/\___/\__,_/_/   \___/_/ /_/  /_____/\___/_/ /_/\___/_/ /_/_/ /_/ /_/\__,_/_/  /_/|_|
    /_/

[INFO] [Test Execution ID]: 6dd46fcf-76c7-44cd-bff5-4e73b859d01f
[INFO] You did not provide an explicit timeout in the client options. Assuming default of 10 seconds.
[ERROR] Cannot execute-test. Error in test execution orchestrator ([Errno 13] Permission denied: 'workloads')

Confirmed that error not present in previous versions.

Culprit

When you run the command, the docker container spawns and then closes because of the early failure. To discover the permissions issues, run the docker run command and afterwards have another window that runs export ID=$(sudo docker ps --no-trunc | awk 'NR==2 {print $1}') && sudo docker exec -it $ID ls -lrt /. Did this with OS 1.5.0, 1.8.0, and 1.9.0 Docker Images and the culprit seems to be related to ownership. 

# OS 1.5.0 
$ export ID=$(sudo docker ps --no-trunc | awk 'NR==2 {print $1}') && sudo docker exec -it $ID ls -lrt /
drwxr-xr-x.   1 benchmark opensearch-benchmark    41 Sep 11 18:29 opensearch-benchmark

# OS 1.8.0
$ export ID=$(sudo docker ps --no-trunc | awk 'NR==2 {print $1}') && sudo docker exec -it $ID ls -lrt /
total 16
...
drwxr-xr-x.   1 benchmark opensearch-benchmark    41 Sep 11 18:20 opensearch-benchmark

# OS 1.9.0
$ export ID=$(sudo docker ps --no-trunc | awk 'NR==2 {print $1}') && sudo docker exec -it $ID ls -lrt /
total 16
...
drwxr-xr-x.   1 root root    24 Aug 22 16:18 opensearch-benchmark
...

To reproduce

  1. Create a directory and run chmod -R 777.
  2. Create a benchmark.ini within the directory and populate it
  3. Run the command 
    sudo docker run -v <BENCHMARK.INI PATH>:/opensearch-benchmark/.benchmark/benchmark.ini opensearchproject/opensearch-benchmark:1.9.0 execute-test --target-hosts <ENDPOINT> --client-options=basic_auth_user:'xxxxx',basic_auth_password:'xxxxx' --pipeline benchmark-only --kill-running-processes --workload percolator --workload-params '{"number_of_shards":"1","number_of_replicas":"0"}'

Expected behavior

When users run the same command but for OSB 1.5.0 and 1.8.0 Docker Images, it works.

Screenshots

No response

Host / Environment

Additional context

No response

Relevant log output

No response

IanHoang commented 2 months ago

We'll aim to fix this in a patch release

gkamat commented 2 months ago

Fixed with #643.