Different Pending Limits based on Reviewer Experience

[kouloumos]

Currently, our review system restricts all reviewers to a maximum of 3 pending reviews. This limit helps maintain quality control by preventing inexperienced reviewers from overloading the system. However, this one-size-fits-all approach has created bottlenecks, particularly when there are delays in the evaluation process. These delays can demotivate active reviewers, as they are unable to claim new reviews and maintain their momentum.

Proposed Solution

To address this issue and better accommodate reviewers based on their experience, we can implement dynamic pending review limits:

• First-Time Reviewers: Limit to one pending review. This ensures that we will make sure that new reviewers understand the required standards before moving on with their next review.
• Experienced Reviewers: After a reviewer has 3 merged reviews, increase their pending review limit to 6. This adjustment recognizes their proven track record and encourages continued contribution.

Technical Implementation

• Move the limiting logic to the backend: Integrate the logic for dynamic review limits directly into the backend as part of the /api/transcripts/{id}/claim API endpoint.
• Frontend Changes: Remove the current static limit logic from the frontend and ensure it correctly enforces the backend’s dynamic limit criteria.

[Extheoisah]

I'm in support of this. Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

[kouloumos]

Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

You previously suggested to limit the scope of the change and think about the 24h time limit later. That's why I didn't include it here. I'm not sure what the ideal time limit is. Some of our experienced reviewers may have an opinion on this, cc @varmur, @markon1-a, @realdezzy, @davidgumberg

[nully0x]

Cool, this is a good suggestion

[Extheoisah]

Okay

Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

You previously suggested to limit the scope of the change and think about the 24h time limit later. That's why I didn't include it here. I'm not sure what the ideal time limit is. Some of our experienced reviewers may have an opinion on this, cc @varmur, @markon1-a, @realdezzy, @davidgumberg

Okay. Since that time change is in the frontend, it doesn't touch this implementation. We can proceed this on the backend

[markon1-a]

Currently, our review system restricts all reviewers to a maximum of 3 pending reviews. This limit helps maintain quality control by preventing inexperienced reviewers from overloading the system. However, this one-size-fits-all approach has created bottlenecks, particularly when there are delays in the evaluation process. These delays can demotivate active reviewers, as they are unable to claim new reviews and maintain their momentum.

Proposed Solution

To address this issue and better accommodate reviewers based on their experience, we can implement dynamic pending review limits:

• First-Time Reviewers: Limit to one pending review. This ensures that we will make sure that new reviewers understand the required standards before moving on with their next review.
• Experienced Reviewers: After a reviewer has 3 merged reviews, increase their pending review limit to 6. This adjustment recognizes their proven track record and encourages continued contribution.

Technical Implementation

• Move the limiting logic to the backend: Integrate the logic for dynamic review limits directly into the backend as part of the /api/transcripts/{id}/claim API endpoint.
• Frontend Changes: Remove the current static limit logic from the frontend and ensure it correctly enforces the backend’s dynamic limit criteria.

I really support this idea too. It will help with the quality of work done.

[davidgumberg]

+1 on the first-time reviewer limit decrease, I think there are things not obvious to someone who is reviewing a transcript for the first time about what makes a good transcript.

I haven't spent enough time in the repo to know about the increased limit for experienced reviewers, but it sounds reasonable.

[realdezzy]

Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

You previously suggested to limit the scope of the change and think about the 24h time limit later. That's why I didn't include it here. I'm not sure what the ideal time limit is. Some of our experienced reviewers may have an opinion on this, cc @varmur, @markon1-a, @realdezzy, @davidgumberg

I support the idea of increasing the time limit to 72 hrs. It would help improve the quality of reviews 👍🏾

[aassoiants]

I propose increasing to 48 hours and seeing if there's an increased demand after that

[kouloumos]

This was supposed to close automatically after the merge of the linked PR.

Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

I will open a different issue to tackle this one.

[varmur]

Do you think it makes sense to also increase the 24hrs limit in the frontend for reviewing a transcript for experienced reviewers? Maybe to 72hrs?

@kouloumos, @Extheoisah

Yes, this is a great change as well. Sometimes there are transcripts that are related, or in parts - it would be good to be able to claim these together and have enough time to review them well. 72 hours for experienced reviewers sounds right to me.