Should we increase the core threshold?

[ArtOfCode-]

It's occurred to me a bunch recently that 50 feedbacks might be too low of a threshold for core; it's really no effort at all to get to that. An instant strawpoll in chat said that at least some people agree, so, here's an issue.

1. Should we increase the threshold?
2. What to? 250 has been suggested; that seems somewhere around rightish to me; other suggestions? I guess it doesn't have to be rooted in feedbacks, either, though that seems to be a nice easy measure.

[Mego]

250 seems good to me. Given the average of ~10 reports per hour, that's roughly 25 hours actively watching reports.

[zalmanlew]

How about also including that you have to have less than 1% invalidated (so for 250, that would mean 2.5 posts) and if they do have over and still want to get core, they just have to feedback a bit more to equal it out.

[Undo1]

Maybe do a rolling window threshold? X feedbacks in the last Y days/months. Tweak as necessary

[thesecretmaster]

It might be good to see some data on the number of non-core people who would meet each of these suggested criteria.

[angussidney]

I've never really thought that it's a good idea to give out the core role purely based on feedbacks. It's too easy to turn up one day, install FIRE and add duplicate feedbacks to the last x reports. No attention is paid to the amount of time that you've been around, how well you know the system, whether you've been making other meaningful contributions (flagging, reporting, blacklisting, coding), etc - all of which I would consider indicators of being a 'core' part of the team.

I'd rather switch it back to the old system: you get the role when the admins think that you're a core part of the team. I know that using a subjective system isn't exactly the most appealing, but I think that it will provide us with a better answer than an arbituary number of feedbacks.

[Undo1]

I think having core based on feedback could work pretty well with a rolling window. Subjective thresholds are... hard... and usually just means we give it to everyone who knows enough to ask and isn't annoying.

I see two problems with a static threshold (250 or otherwise):

1. There's a delayed effect between when you start contributing and when the system kicks in - with 250, it'd likely be days or weeks.
2. That list is almost never reviewed. There are currently 38 folks on the core list, some of them haven't been around for a while. Do we want to dilute the flag benefits for current contributors?

[Undo1]

Some data, in a gist: https://gist.github.com/Undo1/ea554bc20052f03a9f6db66a6a03bba1

[ArtOfCode-]

@Undo1 What period is that data for?

[Undo1]

@ArtOfCode- Fresh as of this morning; that data is for all time.

[Undo1]

Preliminary analysis on rolling windows: https://gist.github.com/Undo1/4a3e8278f87b4b16a8d8e9370577e1e8

Could definitely use some tuning, but I'm drawn to a system that doesn't reward a past contribution forever and ever.

[ArtOfCode-]

So... some thoughts, now that some other people have chipped in:

We need to keep some subjectivity, even if that's just "admin's discretion". There are people like Andy, who are definitely current core contributors, but who the stats don't include; there may also be people, who I have no example for, who meet the threshold but should really not be given the role. Any automated system we build for granting/dropping the role needs to take that into account.

I like the idea of rolling windows - it means the core list actually becomes current core users. While it's good to recognise people's past contributions too, I don't think rewarding a single huge contribution ten years ago is something we should be doing. (Okay, I exaggerate. When do I not?)

With that in mind, I'd tentatively suggest 60 per month. 60 because it's not an unreasonable ask in a month - 2 feedbacks a day, or 30 feedbacks for 2 days - both are totally possible. A month because it's a long rolling window so that going on holiday for a week doesn't drop your core status. I'd be open to higher numbers, too, but probably not much higher than 100.

[ArtOfCode-]

I'm deliberately not ruling out giving the role out for non-feedback contribution, either. Angus' point is totally valid - we should pay some attention, even if it's just a sideways glace, to other factors. Defining the objective criteria in terms of feedbacks gives us the objectivity that we can apply mostly mindlessly for the easy situations; any exceptional requests can be dealt with by a human.

[Undo1]

@ArtOfCode- Added a 60/mo list to that last gist.

Definitely agree that there should be manual exemptions, but they should be for non-feedback ongoing contribution. That should be fairly rare.

[Mego]

I like the rolling window proposal. Current contributions should be values more than historical contributions. I also support considering factors other than just feedbacks - someone who has contributed a ton of code and time spent debugging code would be a good candidate for core, even if they haven't done as much feedback.

[iBug]

I have a ~super bad~ proposal: identity in the CHQ room.

If one is recognised by a lot of folks in CHQ, then they are surely a good candidate for MS Core. Otherwise, we may need some other indicates.

Note that this is only an affirmative suggestion, we cannot say one is a bad candidate just for not being identified by a lot of people. We can only say one is good for that using this (criterion).

[Undo1]

Had a thought the other day about this in conjunction with the ongoing 4/5 flags discussion. Could we change how we think about 'core' from 'people who have contributed' to 'people who are contactable for flag retractions'? There's a concern - and a legitimate one, despite the numbers saying it's minimal - about that one time we do 5-flag a false positive and it just sits there teetering on the edge.

Would having 3 or 4 of those 5 people being pingable in CHQ help alleviate that concern? Statistically, we'd probably be able to contact at least one to back a post down from 5 flags.

[AWegnerGitHub]

I don't think it's reduce concern that much. Personally, I'd see that as a weird crutch. If we trust the system so much, why do we want to be able to guarantee someone is available to back off a flag? How do you handle it when that person or people are sleeping?

On Feb 17, 2018 5:19 PM, "Undo1" notifications@github.com wrote:

Had a thought the other day about this in conjunction with the ongoing 4/5 flags discussion. Could we change how we think about 'core' from 'people who have contributed' to 'people who are contactable for flag retractions'? There's a concern - and a legitimate one, despite the numbers saying it's minimal - about that one time we do 5-flag something and it just sits there teetering on the edge.

Would having 3 or 4 of those 5 people being pingable in CHQ help alleviate that concern? Statistically, we'd probably be able to contact at least one to back a post down from 5 flags.

Just a random thought I had.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/Charcoal-SE/SmokeDetector/issues/1594#issuecomment-366478769, or mute the thread https://github.com/notifications/unsubscribe-auth/AGKb5xmqOLx4bfI5sa_LCkx96fCdqDwzks5tV15mgaJpZM4R6ZOn .

[angussidney]

I could be wrong here, but I think the main thing that everyone has in the back of their minds while we're discussing this is the major benefit of the core role: additional autoflags. I think we're forgetting the other, potentially more important abilities that core members have.

So, what if we separate the additional autoflags out of core and into a new role?

Then:

• The core role and it's associated privileges (ability to create announcements, use the data explorer, edit domain tags etc, have their name appear on /people) could be subjectively given out to people who the admins consider to be a core member of the team, which I still think is a superior solution
  • Additional autoflags can be given out on a purely objective basis (as many of you are arguing for above). Additionally, automating this would be much easier since we wouldn't need to edit the /people list on our website.
  • People who want the role purely for autoflags won't be affected
  • We won't have to balance the level of involement required for additional autoflags vs core member when deciding privilege thresholds

One possible implementation for this would to give all accounts the ability to sign up for autoflagging (à la the current 'flagger' role), change the 'flagger' role so that it grants additional autoflags (adding/revoke the role as required), and keep the 'core' role the same, minus the autoflagging privs.

[ArtOfCode-]

@Undo1 That kind of viewpoint shift makes this inherently less objectively-grantable or automatable. Not sure if that's good or bad.

@AWegnerGitHub I think is right; one 5-flagged FP in 100k isn't the sort of numbers I think we need to do anything extra to mitigate.

@angussidney Sounds like added complexity. I think you're right that subjectively granting core is a better solution, but I also don't really want to be having subjective discussions and debates every time we add someone to the core list. It's really easy to check a feedback history and say "yes, N feedbacks over X timeframe, you're in".

[magisch]

@Undo1 That would mean we should limit the "core" to people who have core and have been active in CHQ in the last X minutes, no?

[ArtOfCode-]

Done at 60 per 30 days, auto-applied by metasmoke.