stsievert / salmon

A tool to collect triplet queries
https://docs.stsievert.com/salmon/
BSD 3-Clause "New" or "Revised" License
9 stars 2 forks source link

Adaptive road test fixes #85

Closed stsievert closed 4 years ago

stsievert commented 4 years ago

What does this PR implement? It irons out some of the bumps from the adaptive road test. Specifically, it does the following:

It also does some smaller maintenance changes:

Reference issues/PRs This PR addresses #83, closes #84 and addresses https://github.com/stsievert/salmon/issues/82#issuecomment-716768050.

stsievert commented 4 years ago

I've tested 708457df262cc6314b836e742d2d3b26a2676811 manually, and am unsure how to test it without making major modifications. When running examples/num-users/sim.py with 10 users and one adaptive algorithm (RR), here's some of the output:

dl=14, DR=3. (h, l, r, w) = (26, 12, 29, 29), score=10.98876404494382
dl=9, DR=9. (h, l, r, w) = (20, 29, 11, 11), score=11.0
dl=11, DR=1. (h, l, r, w) = (12, 1, 13, 13), score=-9999
DL=2, dr=7. (h, l, r, w) = (29, 27, 22, 27), score=10.96629213483146
dl=9, DR=4. (h, l, r, w) = (15, 24, 11, 11), score=10.98876404494382

That indicates that for the third query, zpopmax didn't return anything and had to get a random query.

stsievert commented 4 years ago

Marking as a draft until the current experiments are done; I don't want to change the redirects when the docs are updated.