-
Great job with the new containerized evaluation tool! I've run it a couple of times on the golden patches on SWE-bench Lite and overall it gives a more stable result than my swe-bench-docker setup. Th…
-
**Current Behavior (bug)**
Slow sysbench when compared with docker, podman, and apptainer
**Expected Behavior (fix)**
Faster, as native OS
**Additional context**
I'm conducting some benchmark…
-
In #107 we introduced a testing infrastructure that allows us to test several migration scenarios. Unfortunately, the `streamChanges` feature uses the `spark-kinesis` module under the hood and this mo…
-
Allow users to leverage the power and flexibility of the containerized tool-meister and tool-data-sink in traditional pbench-agent benchmarks. Will allow for:
- Multi-node benchmarks with the pbench…
-
## Summary
Introduce a GitHub Action specifically designed for continuous benchmarking using zBench. This action will automate the process of running benchmarks on every commit/pull request, collec…
-
I'd like to optimize Elastiknn such that the Fashion Mnist benchmark performance exceeds 200 qps at 96% recall. Currently it's at 180 qps. So this would be about an 11% improvement. There are already …
-
In the documentation, there is this note under [Makefile](https://conbench.github.io/conbench/#:~:text=your%20current%20shell.-,Makefile%20targets,-%C2%B6) targets:
```
You can use Ctrl+C to term…
-
Right now, I can see only plots in the repo, which are pretty useless for the "external" comparison as such.
In particular, is HNSW still using hours for preprocessing? :) There does not seem to be…
-
Investigate image maintenance processes around:
- continuous image building (re-building of provided images over time to patch CVEs)
The expectations that we are trying to meet are:
- low activ…
-