pytorch / test-infra

This repository hosts code that supports the testing infrastructure for the PyTorch organization. For example, this repo hosts the logic to track disabled tests and slow tests, as well as our continuation integration jobs HUD/dashboard.
https://hud.pytorch.org/
Other
83 stars 80 forks source link

More flexible benchmark grouping #4124

Open ezyang opened 1 year ago

ezyang commented 1 year ago

Two use cases:

Right now, benchmark grouping is hard-coded for the three benchmark suites. It would be nice if there were a light way to introduce new benchmark groupings on a model-by-model basis and see how they evolve over time. Some UI design necessary for managing these groupings.

cc @huydhn

huydhn commented 1 year ago

As this is not a simple fix IMO, I have added it into our sprint so that I can be planned accordingly

ezyang commented 1 year ago

a crappy mockup (like maybe just some manually drawn screens in excalidraw) I think would help us iterate on the UI design quickly, without having to build everything first