Allow clustering for GT to be done on mturk server

[misaugstad]

We are having researchers do ground truth audits using the mturk branch, by completing HITS on the sandbox, and we are working on a GT consensus tool that will work on that server. So the piece that is missing is to do the intermediate clustering and such on the server as well.

TODOs:

• [x] add tables for storing ground truth labels and results of intermediate clustering (Teja)
• [x] add the current clustering script to the code, and figure out how to run it on the server, putting results back into the table that is being created
• [x] create a simple GUI for running the clustering, maybe re-running, possibly showing some summary stats about number of agreements, etc.

[jonfroehlich]

Why are you having them do it via HITS? I don't like that mturk is serving as an unnecessary intermediary here--what's the rationale? Just quicker to get this done?

Also, I feel anxious about having two branches that we use for our research--the production code (root) and then the mturk branch. I'm worried that we will accidentally be running different code between the two that may impact our results.

On Tue, Jul 18, 2017 at 2:12 PM, Mikey Saugstad notifications@github.com wrote:

We are having researchers do ground truth audits using the mturk branch, by completing HITS on the sandbox, and we are working on a GT consensus tool that will work on that server. So the piece that is missing is to do the intermediate clustering and such on the server as well.

TODOs:

• add a table that has label id, a boolean for if the label is in the final GT or not, and maybe a cluster id
• add the current clustering script to the code, and figure out how to run it on the server, putting results back into the table that is being created
• create a simple GUI for running the clustering, maybe re-running, possibly showing some summary stats about number of agreements, etc.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/SidewalkWebpage/issues/877, or mute the thread https://github.com/notifications/unsubscribe-auth/ABi-9aPvREsOSFCNHI-sb-CP9VZeoOH4ks5sPPV5gaJpZM4ObuRS .

-- Jon Froehlich Assistant Professor Computer Science University of Maryland, College Park http://www.cs.umd.edu/~jonf/ @jonfroehlich https://twitter.com/jonfroehlich - Twitter

[misaugstad]

The biggest reason for using the mturk server is that we don't have the ability to follow fixed routes in the main tool. A less important reason is that I really like how the system of HITs works for the task of collecting ground truth data. I generate the routes, and then the GT labelers have this set of routes that they have to do, and the HIT system automatically lets each of them do each route only once, etc.

For your second point, what is an example of a scenario that you could imagine that would cause an issue?

[jonfroehlich]

Let me preface my comments by saying that I think I'm ok with the decision of using the mturk server but let me point out some concerns/questions too.

The biggest reason for using the mturk server is that we don't have the ability to follow fixed routes in the main tool.

And so you are implicitly arguing that you think it better to go with the mturk server than push this code to production? Why?

A less important reason is that I really like how the system of HITs works for the task of collecting ground truth data. I generate the routes, and then the GT labelers have this set of routes that they have to do, and the HIT system automatically lets each of them do each route only once, etc.

OK, well, so that doesn't seem like very hard code to write in a custom GT admin front end... but you just think it's easier to use the mturk HITs infrastructure for this?

For your second point, what is an example of a scenario that you could imagine that would cause an issue?

Perhaps I'm being overly paranoid or perhaps I don't fully understand how the mturk branch actually gets updated with new code from root but having two code versions running seems like it could end in disaster ("Oh sh*t, I thought we had that feature pushed to the mturk server--oh, we didn't? OK, well then we can't actually compare some production data to mturk data because of it..."--that kind of thing).

[misaugstad]

And so you are implicitly arguing that you think it better to go with the mturk server than push this code to production? Why?

@maddalihanumateja this is really for you

Perhaps I'm being overly paranoid or perhaps I don't fully understand how the mturk branch actually gets updated with new code from root but having two code versions running seems like it could end in disaster ("Oh sh*t, I thought we had that feature pushed to the mturk server--oh, we didn't? OK, well then we can't actually compare some production data to mturk data because of it..."--that kind of thing).

I definitely share this concern. It completely hinges on our ability to remember to pull commits to the main branch to the mturk branch before starting experiments or collecting ground truth. Doesn't seem hard to remember, but it is an important step.

[misaugstad]

We had an in-depth discussion about this during an in-person meeting and concluded that it is not worth the dev time, considering the CHI18 deadline, to combine the mturk and main branches, though we would like to do this post-CHI for sure.

[misaugstad]

resolved via PR #932