search

mozilla / aestimia

[Archived] Assessment tool
4 stars 10 forks source link

Assessment Times and Throughput #66

Open teverette opened 11 years ago

teverette commented 11 years ago

Hi Team Moz!

I'm Tre, lead dev at DYN.

In order to manage the assessment queues, we would like receive some basic details about what we will know about assessments in near real time? The background is we expect a fluctuation of submissions, and we need to know when assessment times are going poorly or are just overwhelmed and we need to scale up the number of assessors. Based on that, those managing the assessment process need to know details related to throughput.

(I started this conversation via email with @cmcavoy with his quick responses inline.)

Examples How many submissions occurred yesterday? this week?

How many submissions were assessed yesterday? this week?

Possible submission breakdown by activity owner (ie organization)

Average wait time between assessment and submission would clearly be of great value to help keep the throughput high.

Another bonus would be a view of each assessor's work: how many assessed? completed vs rejected?

Will submission date be shown on the page listing submitted artifacts?

End Goal: We need a couple of ways to know how things are going behind the scene.

chloeatplay commented 11 years ago

Hi @teverette , great points :+1: We had a progress bar idea that would solve many of these issues, might be worth revisiting with @toolness

teverette commented 11 years ago

Hello @chloeatplay, let me know if and when it's revisited. If you have a lofi (or hifi) mockup of the progress bar concept, I would love to see it.

I know there is a lot happening over the next week, so I just want this on the radar to keep assessment running smoothly.

threeqube commented 11 years ago

Hey @teverette I'm not sure if there's a more recent one but here's one version of the progress bar concept designed by @chloeatplay at an earlier iteration:

queue-view

iamjessklein commented 11 years ago

:+1:

teverette commented 11 years ago

Hi. Sorry for the slow response but there's so much going on. Have you all made additional progress on this feature already?

Is the progress bar for each badge or does it span across a certain category?

The progress bar is an excellent visual indicator for the assessors as they try to work through the queue. In order to know the activity level, I'm also looking for several more administrative metrics like badges submitted yesterday vs badge assessed yesterday vs badges waiting for assessment plus number of badge assessed by mentor xyz.

Are there multiple roles like assessor and admin or just one role on the assessing side? If there is an admin role, I imagine a page with those stats and a stats for each mentor would solve a logistical pain point.

teverette commented 11 years ago

Some additional requests from those attempting to monitor assessors:

cmcavoy commented 11 years ago

Hey Tre, there's a meeting today about data reporting / metrics across the site...we'll roll a dashboard out soon.

On Thu, Jun 20, 2013 at 1:23 PM, teverette notifications@github.com wrote:

Some additional requests from those attempting to monitor assessors:

  • Ability to see who assessed a badge at the top level.
  • Count of badges in queue
  • Count of badges assessed(rejected and awarded) overall
  • Count of badges assessed (rejected and awarded) current day.

— Reply to this email directly or view it on GitHubhttps://github.com/mozilla/aestimia/issues/66#issuecomment-19773197 .

@chmcavoy http://twitter.com/chmcavoy http://lonelylion.com

teverette commented 11 years ago

Thanks!