IanHoang
commented
10 months ago Error also occurs for create-workload feature when there's an error with workload template. This will need to have additional error handling.
2024-01-23 21:23:27,418 -not-actor-/PID:81790 osbenchmark.benchmark ERROR A fatal error occurred while running subcommand [create-workload].
Traceback (most recent call last):
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/benchmark.py", line 889, in dispatch_sub_command
workload_generator.create_workload(cfg)
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 193, in create_workload
render_templates(
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 123, in render_templates
write_template(template_vars, templates_path, test_procedures_path, "custom-test-procedures")
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/workload_generator/workload_generator.py", line 107, in write_template
f.write(template.render(template_vars))
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 1090, in render
self.environment.handle_exception()
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/environment.py", line 832, in handle_exception
reraise(*rewrite_traceback_stack(source=source))
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/_compat.py", line 28, in reraise
raise value.with_traceback(tb)
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/resources/custom-test-procedures.json.j2", line 35, in top-level template code
{%- block queries -%}
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/osbenchmark/resources/custom-test-procedures.json.j2", line 44, in block "queries"
"target-throughput": {{ query.target_throughput | tojson }}
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/filters.py", line 1260, in do_tojson
return htmlsafe_json_dumps(value, dumper=dumper, **options)
File "/Users/hoangia/Desktop/Development/opensearch-benchmark-ianhoang/opensearch-benchmark/.venv/lib/python3.8/site-packages/jinja2/utils.py", line 619, in htmlsafe_json_dumps
dumper(obj, **kwargs)
File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/__init__.py", line 234, in dumps
return cls(
File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/Users/hoangia/.pyenv/versions/3.8.12/lib/python3.8/json/encoder.py", line 179, in default
raise TypeError(f'Object of type {o.__class__.__name__} '
TypeError: Object of type Undefined is not JSON serializable
gkamat
Is your feature request related to a problem? Please describe.
Many OSB users have encountered Jinja and workload errors that are unclear, which lead many to believe that there is a bug in OSB tool itself. This happens in several situations but often happens when users make edits to workload files. Here are a few examples and their associated fixes.
Example 1:
This was because there was a missing comma in the jinja JSON template. It's not as simple to discover this issue through a JSON formatter because there are jinja template inputs and those are labeled as incorrect in a JSON formatter.
Example 2:
This error occurred because the Jinja template inputs were missing some important parameters. However, there's no check for them. For instance, template had
{{search_clients}}but needed{{search_clients | default(8)}}. Another common instance is when users have something like{{index_settings | default({})}}but needed{{index_settings | default({}) | tojson}}.Example 3:
This error was more helpful than others but still lacked clarity. This eventually led developers to realize that there's something wrong with the jinja formatting rather than the JSON file.
Even errors that include
The complete workload has been written to '/var/folders/yh/89c2pcg10szgzwc6qj2h04gst115h_/T/tmptoxqvjsr.json' for diagnosisin the error message are not helpful since the file for diagnosis is usually empty or unclear.Describe the solution you'd like
We should include clearer error messages for reading Jinja and workload templates.
Describe alternatives you've considered
We could have another file that checks for specific issues but that might be unwieldy. The simplest solution would be to add better error handling first.
Additional context