search

cloud-bulldozer / benchmark-wrapper

Python Library to run benchmarks
https://benchmark-wrapper.readthedocs.io
Apache License 2.0
19 stars 56 forks source link

[FIO]: Elasticsearch index 'ripsaw-fio-analyzed-result' doesn't record the std-deviation for randomread and randomwrite #180

Open keesturam opened 4 years ago

keesturam commented 4 years ago

we depend on the std-dev value in the ripsaw-fio-analyzed-result in ocs-ci to evaluate whether the data sample is valid before proceeding the validation of regression.

jtaleric commented 4 years ago

@acalhounRH @bengland2

bengland2 commented 4 years ago

@keesturam touchstone deals with related concerns. It's not ripsaw's job to calculate %deviation. ripsaw is just an operator that runs a benchmark. snafu is just a benchmark wrapper that digests the output data and injects it into elasticsearch. What we do from there is up to us. Alex has an fio grafana dashboard that calculates it, here's an example. Avi has a script that generates %deviation for smallfile data, same method could be used by OCS QE for fio if this is appropriate. Or we could try to build this code into the fio-client pod so that when it is done injecting per-pod results into ES, it would go the extra mile and compute aggregate results, %deviation across samples, etc. @acalhounRH does this seem technically feasible?

Smallfile used to have an orchestrator process that would run all the workload processes, then aggregate per-thread results into cluster-level results before Kubernetes came along, but right now it can't because there is no shared filesystem, there is just elastic search and a bunch of smallfile pods synchronized by redis. Someone (Avi?) suggested that smallfile should have an "orchestration" pod much like fio does, so when all the smallfile worker pods finish, the orchestrator pod could go into ES and finish reducing the data, much like Avi's script is doing. Instead of using a shared filesystem like smallfile used to do, Redis could be used for synchronization, and ES is the shared repository where the data can be placed and found. I like that idea.

keesturam commented 4 years ago

@bengland2 Implementation of arriving at a standard deviation is available in the analyzed results. It works for sequential workload and doesn't work on random workload. As I understand from the conversation with @acalhounRH this is a minor bug that needs to be fixed.

acalhounRH commented 4 years ago

ack, it is a minor bug.

change that is need is an additional if statement that checks for randrw.