Adaptive algs don't work if experiment launched on a dirty database

stsievert / salmon

A tool to collect triplet queries

https://docs.stsievert.com/salmon/

BSD 3-Clause "New" or "Revised" License

9 stars 2 forks source link

Adaptive algs don't work if experiment launched on a dirty database #88

Closed stsievert closed 3 years ago

stsievert commented 4 years ago

In an experimental use case, one adaptive algorithms was launched from a clean machine. Then, two days later, Salmon launched another experiment on top of that experiment. Here are some requests that Salmon received:

Nov. 9th, 17:00: /reset
Nov. 9th, 17:01: /init_exp. (I think this worked fine).
Nov. 11th, 21:00: /init_exp (no /reset first).

Then, neither adaptive algorithm produced adaptive queries; the score was always -9999.

stsievert commented 4 years ago

Running two experiments is explicitly outside Salmon's use case: it's only designed to run one experiment. Currently, when a user visits /init_exp to relaunch an experiment, a warning is displayed that page:

This warning should be displayed when visiting /init_exp. In addition, I think an error should be raised when posting to /init_exp.

stsievert commented 3 years ago

This is closed by #91. Now, POST'ing to /init_exp will raise an error if there's already an experiment already initialized. The error recommends /reset first (which is a large barrier, and the error message gives lots of warnings).