Benchmark Groups Have Some Minor Problems

python / pyperformance

Python Performance Benchmark Suite

http://pyperformance.readthedocs.io/

MIT License

865 stars 176 forks source link

Benchmark Groups Have Some Minor Problems #234

Open ericsnowcurrently opened 2 years ago

ericsnowcurrently commented 2 years ago

(See https://github.com/pyston/python-macrobenchmarks/pull/10.)

[x] a benchmark group "all" is allowed, but ignored
[x] duplicate groups are allowed (between manifest files) but the semantics aren't clear
[ ] a group will overwrite a tag with the same name, without warning
[x] the "list_groups" command does not show tags if any custom groups are defined
[x] tags aren't found properly in BenchmarksManifest.resolve_group()

ericsnowcurrently commented 2 years ago

Also "list -b TAG" isn't working.

ericsnowcurrently commented 2 years ago

Also, if the "default" group includes a name that does not exist then it is ignored, whether or not "-NAME". I'd expect it to fail like it does for any other group.

ericsnowcurrently commented 2 years ago

@kmod, the main problem from https://github.com/pyston/python-macrobenchmarks/pull/10 is due to how a custom group named "all" is getting ignored. I'm probably going to just disallow such a group.