aimhubio / aim

Aim 💫 — An easy-to-use & supercharged open-source experiment tracker.
https://aimstack.io
Apache License 2.0
5.23k stars 321 forks source link

Aggregated metrics not computed correctly #3242

Open kirschnj opened 3 weeks ago

kirschnj commented 3 weeks ago

🐛 Bug

Aggregated values (mean, median) are not displayed correctly.

To reproduce

Log several values for different contexts, where some values appear multiple times as below:

import aim
run = aim.Run(experiment="test")

for step in range(3):
    run.track(0, name="test", step=step, context=dict(context=0))
    run.track(0, name="test", step=step, context=dict(context=1))
    run.track(1, name="test", step=step, context=dict(context=2))

run.close()

In the UI, both median and mean are incorrectly shown as 0.5.

image

Expected behavior

In the example above, the mean per step is 0.33 and the median is 0.

Environment

Additional context

The issue appears to be that only unique values in the group are used to compute the mean/median. The issue doesn't appear with distinct values (e.g. 0, 0.2, 1).

mihran113 commented 2 weeks ago

Hey @kirschnj! Thanks a lot for noticing this. 🙌 I'll open the PR with a fix shortly.