Is your feature request related to a problem? Please describe.
When using argilla for crowdsourced annotation workflows on large datasets, it would be necessary to enforce per-user limits on how many annotations can be submitted.
Describe the solution you'd like
I would like a way to enforce an upper limit on how many instances each user can annotate. This could be added as part of a DistributionStrategy, featuring a max_annotations_per_user parameter.
As a bonus, if an upper limit is set, the progress tracker could then reflect the personal progress of the user instead of the dataset progress.
Describe alternatives you've considered
A workaround is sharding the dataset into slices of max annotations size and selectively adding users to each; this creates huge overhead.
A different approach is having secondary app running that monitors user stats and removes them from the workspace if their upper limit is reached. Again, overhead.
Is your feature request related to a problem? Please describe.
When using argilla for crowdsourced annotation workflows on large datasets, it would be necessary to enforce per-user limits on how many annotations can be submitted.
Describe the solution you'd like
I would like a way to enforce an upper limit on how many instances each user can annotate. This could be added as part of a
DistributionStrategy, featuring amax_annotations_per_userparameter.As a bonus, if an upper limit is set, the progress tracker could then reflect the personal progress of the user instead of the dataset progress.
Describe alternatives you've considered
A workaround is sharding the dataset into slices of max annotations size and selectively adding users to each; this creates huge overhead. A different approach is having secondary app running that monitors user stats and removes them from the workspace if their upper limit is reached. Again, overhead.