-
A few small things that would be great to improve in benchmarking CLI:
1. We should use `structop` instead of `clap` as thats what we use for all other CLIs (e.g., [here](https://github.com/0xPolyg…
-
We need to make an evaluation script so we can benchmark our pre-trained models. I recommend creating a `src/probllms/evaluation.py` and making an evaluation script using Hydra in `src/core/experiment…
-
We should code up as many of the PerMedCoE community benchmarking simulations as possible in Chaste, and make sure that the code is running as quickly as it can (e.g. does the draft report give simula…
-
I am working on integration models and wanted to test my models integration metrics against those currently supported in the pipeline. I was wondering if there is a way to run benchmarking on custom m…
-
## 🚀 Feature
I would like to have automated benchmarks for selected models to allow for performance tracking.
### Work items
Automated benchmarking in this context means two things:
- [ ] #2…
-
Congrats on Flash Attention in the latest version, or to be precise, in having your storage limit increased on Pypi.org so you could upload the release that was weeks ago. Here are some benchmarks fo…
-
This issue captures issues related to improving Fendermint performance.
- [[IPC-5] Benchmark Fendermint](https://github.com/consensus-shipyard/ipc/issues/160)
-
A bit like `fio`. Automatically create files.
## Allow users to select:
- [ ] `lsio` IO backend
- [x] directory
- [x] number of files
- [x] chunksize
- [ ] gaps between chunks
- [ ] whether chunks a…
-
It would also be interesting to expand the benchmarking metrics to include other useful information, e.g. level of confidence on the predictions the agent is doing.
@evangriffiths @kongzii feel fre…
-
### Benchmarking
**From: Charlene WONG**
- We are planning to conduct carbon emission comparison test for real-world functional applications before and after green software patterns (carbon redu…