search

nextml / NEXT

NEXT is a machine learning system that runs in the cloud and makes it easy to develop, evaluate, and apply active learning in the real-world. Ask better questions. Get better results. Faster. Automated.
http://nextml.org
Apache License 2.0
160 stars 53 forks source link

Dashboard cacheing issues when multiple experiments on the same machine #147

Closed stsievert closed 7 years ago

stsievert commented 7 years ago

When you have multiple experiments on the same machine the dashboard it shows the same dashboard for every experiment. This changes after 30 minutes, and can show a different experiments dashboard.

This is related to commit 2131c5c21b53ef863b0ea0d4cc32bf053af7bc41.

Adding the ?force_recompute (=0 or =1) flag didn't help.

kgjamieson commented 7 years ago

Whoa! That's obviously a problem. To fix, I think we just need to change stat_id to statid+''+exp_uid here: https://github.com/nextml/NEXT/blob/master/next/broker/celery_app/tasks.py#L114-L118

The issue is that the stat_id is the same for all experiments and its being overwritten

On Mon, Oct 3, 2016 at 9:29 AM, Scott Sievert notifications@github.com wrote:

When you have multiple experiments on the same machine the dashboard it shows the same dashboard for every experiment. This changes after 30 minutes, and can show a different experiments dashboard.

This is related to commit 2131c5c https://github.com/nextml/NEXT/commit/2131c5c21b53ef863b0ea0d4cc32bf053af7bc41 .

Adding the ?force_recompute (=0 or =1) flag didn't help.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nextml/NEXT/issues/147, or mute the thread https://github.com/notifications/unsubscribe-auth/ACHgB8NfB5Y4lPK-mlPtVER6_c9bH6NNks5qwS19gaJpZM4KMxlM .

kgjamieson commented 7 years ago

That is, change stat_uid in the two places in those lines to expuid+''+stat_uid. And of course it must also be changed here for the lookup: https://github.com/nextml/NEXT/blob/master/next/broker/celery_app/tasks.py#L92

On Mon, Oct 3, 2016 at 9:39 AM, Kevin Jamieson kevin.g.jamieson@gmail.com wrote:

Whoa! That's obviously a problem. To fix, I think we just need to change stat_id to statid+''+exp_uid here: https://github.com/ nextml/NEXT/blob/master/next/broker/celery_app/tasks.py#L114-L118

The issue is that the stat_id is the same for all experiments and its being overwritten

On Mon, Oct 3, 2016 at 9:29 AM, Scott Sievert notifications@github.com wrote:

When you have multiple experiments on the same machine the dashboard it shows the same dashboard for every experiment. This changes after 30 minutes, and can show a different experiments dashboard.

This is related to commit 2131c5c https://github.com/nextml/NEXT/commit/2131c5c21b53ef863b0ea0d4cc32bf053af7bc41 .

Adding the ?force_recompute (=0 or =1) flag didn't help.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/nextml/NEXT/issues/147, or mute the thread https://github.com/notifications/unsubscribe-auth/ACHgB8NfB5Y4lPK-mlPtVER6_c9bH6NNks5qwS19gaJpZM4KMxlM .

stsievert commented 7 years ago

90e4f90965417f7a8b3c211d1c880e53fa7c8d58 passed for my simple tests (it's not the easiest to test).

Ideally I'd like to see some tests that tests for these cases.