Error for index_not_found_exception" of elastic-package benchmark command

[gizas]

We are testing with main elastic-package and specifically we use the elastic-package benchmark rally with the templates of kubernetes integration.

The package returns an error which does not break the output extraction of corpora data

Detalis:

elastic-package benchmark rally --benchmark pod-benchmark --dry-run --rally-track-output-dir results/pod/
Run rally benchmarks for the package
2024/01/24 16:03:21  INFO License text found in "/Users/andreasgkizas/elastic/taskfile/observability-dev/docs/infraobs/cloudnative-monitoring/rally-automation/taskfliles/integrations/LICENSE.txt" will be included in package
2024/01/24 16:03:24 ERROR unable to refresh data stream at the beginning of rally run: [404 Not Found] {"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index [metrics-kubernetes.pod-ep]","resource.type":"index_or_alias","resource.id":"metrics-kubernetes.pod-ep","index_uuid":"_na_","index":"metrics-kubernetes.pod-ep"}],"type":"index_not_found_exception","reason":"no such index [metrics-kubernetes.pod-ep]","resource.type":"index_or_alias","resource.id":"metrics-kubernetes.pod-ep","index_uuid":"_na_","index":"metrics-kubernetes.pod-ep"},"status":404}

Either make the error as info or lets try to suppress it

[jsoriano]

@aspacca @ruflin how should this error be handled? is it ok to continue without collecting this data before running the benchmarks? should we retry? or fail fast?

[gizas]

@jsoriano now the error does not break the collection of data as described above. So it is not a blocker in my tests.

So my suggestion is that when command includes dry-run, then the existence of the index is not needed and we can ignore it. What do you think?

[aspacca]

@jsoriano, @gizas

I've dig a little more in the error. contrary to my initial expectation, it is not related to cleaning the index that's the target of the rally track.

when you ran the benchmark rally command you have to option to define a second ES cluster used as metrics store. in this case an index is created (if it does not already exist) on that ES cluster where to store the metrics, and a call is made to the refresh API on the index.

while if no metrics store ES cluster is defined, index creation is skipped, the same does not happen for the refresh API call. hence the error. we should change the current behaviour and skip the refresh API call in this case.

beware that in the case of the metrics store ES cluster is defined, the call is made, and the error could occur. it is ok to continue without collecting the metrics, but we should both improve the feedback to the user making clear that the metrics were not collected, and at the same time lower the level of the error to WARN, to avoid the confusion that the problem should be considered a blocking one.