opensearch-project / opensearch-benchmark

OpenSearch Benchmark - a community driven, open source project to run performance tests for OpenSearch
https://opensearch.org/docs/latest/benchmark/
Apache License 2.0
107 stars 75 forks source link

[BUG]Cannot execute-test. This test_execution ended with a fatal crash #453

Open lijie123bes opened 7 months ago

lijie123bes commented 7 months ago

Describe the bug Cannot execute-test. This test_execution ended with a fatal crash

To Reproduce opensearch-benchmark execute-test --pipeline=benchmark-only --workload=http_logs --target-host=http://192.168.0.112:9200 --client-options=basic_auth_user:admin,basic_auth_password:admin,verify_certs:false --test-mode

Logs During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 880, in dispatch_sub_command execute_test(cfg, args.kill_running_processes) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 651, in execute_test with_actor_system(test_execution_orchestrator.run, cfg) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\benchmark.py", line 678, in with_actor_system runnable(cfg) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 386, in run raise exceptions.BenchmarkError("This test_execution ended with a fatal crash.").with_traceback(tb) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 378, in run pipeline(cfg) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 69, in call self.target(cfg) File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 314, in benchmark_only return execute_test(cfg, external=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\osbenchmark\test_execution_orchestrator.py", line 263, in execute_test benchmark_actor = actor_system.createActor(BenchmarkActor, targetActorRequirements={"coordinator": True}) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\thespian\actors.py", line 702, in createActor return self._systemBase.newPrimaryActor(actorClass, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "C:\Users\User\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\thespian\system\systemBase.py", line 205, in newPrimaryActor raise NoCompatibleSystemForActor( osbenchmark.exceptions.BenchmarkError: This test_execution ended with a fatal crash. 2024-02-01 02:17:39,430 -not-actor-/PID:6308 osbenchmark.test_execution_orchestrator INFO Telling benchmark actor to exit. 2024-02-01 02:17:42,433 -not-actor-/PID:6308 osbenchmark.benchmark INFO Attempting to shutdown internal actor system. 2024-02-01 02:17:42,458 -not-actor-/PID:6308 osbenchmark.benchmark INFO Actor system is still running. Waiting... 2024-02-01 02:17:45,498 -not-actor-/PID:6308 osbenchmark.benchmark INFO Shutdown completed.

IanHoang commented 7 months ago

Hi @lijie123bes, thanks for letting us know that you're experiencing this issue. Based on NoCompatibleSystemForActor, it looks like your cluster is undiscoverable and might not be in the same network as your machine running OSB. Are you able to curl the OpenSearch cluster in the same machine that is running OSB?

To better understand your setup, could you provide more insight into the following:

It would also be helpful if you could provide the entire logs file related to this test run.