search

ProjectSidewalk / sidewalk-data-analysis

Holds all offline data analysis scripts for Project Sidewalk required for our forthcoming paper submission
3 stars 0 forks source link

Compute average time to label an attribute #14

Closed misaugstad closed 6 years ago

misaugstad commented 6 years ago

The idea is to figure out how long it takes someone to label an attribute in a pano. So we were thinking of computing this by looking at the difference in time between when the user stepped into a pano, and the time at which they placed a label. For multiple labels on a single pano, we would take the time just described for the first label, and the time between the first and second label to get time for the second one, and so on.

@manaswis what was the rationale for this again? Is this a stat that we really want/need for the paper?

manaswisaha commented 6 years ago

I wanted to study the time it took to label different label types. This would give us a way to talk about which label types are cognitively difficult to assess and label. We would have to consider cases where it is within a session without breaks, I assume, to accurately use labeling time as a proxy for difficulty of label types i.e. we should not consider break times when calculating labeling time. Hence, just looking at cases where we don't see any breaks. Let me know if this sounds doable and sound.

This could serve as a good compliment when talking about accuracy of different label types. It would also be interesting to see if there is any relationship between labeling time and accuracy -- do users tend to label hard-to-assess attributes correctly or incorrectly?

misaugstad commented 6 years ago

We would have to consider cases where it is within a session without breaks, I assume, to accurately use labeling time as a proxy for difficulty of label types i.e. we should not consider break times when calculating labeling time. Hence, just looking at cases where we don't see any breaks. Let me know if this sounds doable and sound.

Yes this makes sense to me. As far as engineering goes, it should be equivalent to exclude any "time to label" values that are above the threshold we use for taking a break :)

BTW did we decide on an amount of time that we want to consider "taking a break"?

manaswisaha commented 6 years ago

Let's do 60 seconds.

On Thu, May 3, 2018 at 4:21 PM, Mikey Saugstad notifications@github.com wrote:

We would have to consider cases where it is within a session without breaks, I assume, to accurately use labeling time as a proxy for difficulty of label types i.e. we should not consider break times when calculating labeling time. Hence, just looking at cases where we don't see any breaks. Let me know if this sounds doable and sound.

Yes this makes sense to me. As far as engineering goes, it should be equivalent to exclude any "time to label" values that are above the threshold we use for taking a break :)

BTW did we decide on an amount of time that we want to consider "taking a break"?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ProjectSidewalk/sidewalk-data-analysis/issues/14#issuecomment-386465532, or mute the thread https://github.com/notifications/unsubscribe-auth/ACvXgKHcWj4x75xPRtIYsQtJAjDpBNWJks5tu5DvgaJpZM4Trmgc .

-- Best Regards, Manaswi Saha Ph.D. Student Paul G. Allen School of Computer Science & Engineering University of Washington, Seattle homes.cs.washington.edu/~manaswi http://homes.cs.washington.edu/~manaswi Twitter - @manaswisaha https://twitter.com/manaswisaha