wildcard/unassign: user activity watcher

gentlementlegen commented 1 month ago

wildcard/unassign: we should create a plugin that checks certain conditions across every assigned task to determine if the assignee is still actively working on it. If not, give them a follow up. If they abandoned the task then unassign. Make sure to punish them by subtracting XP!

See code reference

jordan-ae commented 1 month ago

/start

ubiquibot[bot] commented 1 month ago




Deadline Fri, May 17, 3:25 PM UTC

Registered Wallet
0x2F05fD58023B0a95d1866aa0A3b672cEf05945c5

Tips:

Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

Keyrxng commented 1 month ago

I can only really think of this plugin being an invoker of #9, wouldn't you agree?

If that is the case then #9 should not handle XP mods and this plugin should handle it?

Alternatively, I've setup #9 such that an invoker passes directive === "start" | "stop" and xpAmount: number and then #9 would handle the XP mods.

What's the best approach?

gentlementlegen commented 1 month ago

@Keyrxng I think this can be standalone because events would be sent from the kernel, so it doesn't matter if start | stop is invoked. I think this should be a watcher that updates the assignee status in the task according to reasons specified in the spec (time elapsed, activity, linked pr, commits and whatnot). Reached a certain point, a reminder is posted (maybe something like "are you still working on it? If not, unassign yourself before the bot does), and if nothing ensues simply unassign, and eventually apply XP penalties (still don't have any XP system so don't worry about it yet I guess).

Keyrxng commented 1 month ago

@jordan-ae is taking care of this task and yeah I understand. I made the relevant changes in start/stop thank you

0x4007 commented 3 weeks ago

Quick question for the unassign user: what would be the conditions to disqualify exactly? I was reading the code but it's a bit unclear to me @gentlementlegen via Telegram

The purpose of this capability is to disqualify contributors that neglect to complete the task. A very common scenario is that a contributor will begin a task, and then give up, which means that the task is being blocked (other people will not work on it while it is already assigned.)

Before disqualification, the manager should ask for an update. If the contributor still has no updates, then the manager should disqualify the contributor by unassigning them, and ideally, not allowing them to reassign themselves (we could check in the /start command if the contributor was ever unassigned within the issue timeline.)

In addition, the XP of the contributor should be deducted (not sure about a static amount, or if we should do it based on the price/priority of the task.)

Any user activity that shows they are working on the task should be considered valid to restart the manager's timer. Ideally we should be able to configure this.

Definitely include watching commits.
Possibly include watching comments on either the pull or the issue.

I think these are the only two types of contributor events we should watch for.

gentlementlegen commented 3 weeks ago

So basically, whenever the user is detected as active, we push the deadline by the length of the task? E.g. it's a <1 Day task, each time activity is detected, we push the deadline by 1 day?

Also technically it is allowed on GitHub to have multiple users assigned on the same issue, should any activity of any user be counted to push the deadline? The old bot has some code but I suspect it never worked because it counts the same timers for everyone so either everyone is in the deadline or outside: https://github.com/ubiquity/ubiquibot/blob/8aaf055c091cc3e7f2b42c9f1cf88f45b1d86d17/src/handlers/wildcard/unassign/assign-event-found.ts#L20

0x4007 commented 3 weeks ago

So basically, whenever the user is detected as active, we push the deadline by the length of the task? E.g. it's a <1 Day task, each time activity is detected, we push the deadline by 1 day?

Actually, this is configurable but we always did the following default settings. 3.5 days for follow up, 7 days for disqualify. I like these default settings as they allow for flexibility for holidays/weekends. The time estimate label is mostly used for pricing, but otherwise is not really leveraged in other aspects of our system.

I am skeptical about enforcing based on the time labels because this assumes the contributor is already onboarded to the codebase and ecosystem, which is not always the case.

Also technically it is allowed on GitHub to have multiple users assigned on the same issue, should any activity of any user be counted to push the deadline? The old bot has some code but I suspect it never worked because it counts the same timers for everyone so either everyone is in the deadline or outside: https://github.com/ubiquity/ubiquibot/blob/8aaf055c091cc3e7f2b42c9f1cf88f45b1d86d17/src/handlers/wildcard/unassign/assign-event-found.ts#L20

It seems like a detail I may have implemented, but in practice we never have multiple assignees to tasks. I think this could be useful for when companies/teams get involved in the future. I think it is more precise to manage on the per capita level, so we should consider each assignee individually as what appears to be implemented in your provided example.

gentlementlegen commented 3 weeks ago

In the provided example, it doesn't work because the calculation is the same for each user

const assigneesWithinGracePeriod = assignees.filter(() => now - latestAssignEventTime < taskDisqualifyDuration);

All these values do not seem to be user based, but rather all the same for every user, just based on the latest event.

Okay so to sum it up:

on user activity, add 3.5 days before the next reminder (configurable)
on user activity, add 7 days before the complete disqualify (configurable)
user based time tracking, considering that multiple users could be assigned to an issue

gentlementlegen commented 3 weeks ago

/start

ubiquibot[bot] commented 3 weeks ago




Deadline Wed, Jun 5, 8:42 AM UTC

Registered Wallet
0x0fC1b909ba9265A846b82CF4CE352fc3e7EeB2ED

Tips:

Use /wallet 0x0000...0000 if you want to update your registered payment wallet address.
Be sure to open a draft pull request as soon as possible to communicate updates on your progress.
Be sure to provide timely updates to us when requested, or you will be automatically unassigned from the task.

gentlementlegen commented 3 weeks ago

@0x4007 Might have had an idea on how to reduce load and trigger Actions only when we need to.

This reflection comes after testing the plugin, which fires like 5 builds a second due to the '*' being used.

We do not want to use Worker Alarms, and we do not want to use CRON. What about using a one time Workflow instead? For each job we want to schedule, we could create a dummy Workflow with the only purpose to trigger our main workflow with the desired values, that deletes itself upon completion.

Example, I want to send the reminder for a task tomorrow, I create a workflow with

on:
  schedule:
    - cron: '4 2 9 6 *'

which would fire tomorrow at 2am. Then this workflow deletes itself, and that's it, our main workflow is running on schedule.

What this fixes is:

the crazy amount of workflows triggered on active repos
the stale state of the reminders on inactive repos
the need to listen to * for events

0x4007 commented 3 weeks ago

This is a novel approach I never did research on in the past so I need more information before I'm convinced. But it seems a bit heavy to write a workflow file somewhere regularly and then delete regularly.

gentlementlegen commented 1 week ago

@0x4007 Interesting that the issue didn't close despite being linked with https://github.com/ubiquibot/user-activity-watcher/pull/1 ; manually closing this.

ubiquibot[bot] commented 1 week ago

+ Evaluating results. Please wait...

ubiquibot[bot] commented 1 week ago

[ 42.9 WXDAI ]

@0x4007

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	3	42.9

Conversation Incentives

Comment	Formatting	Relevance	Reward
> Quick question for the unassign user: what would be the con...	22.1 li: count: 2 score: "2" words: 15 code: count: 1 score: "1" words: 1	0.835	22.1
> So basically, whenever the user is detected as active, we p...	16.9 code: count: 2 score: "2" words: 5	0.805	16.9
This is a novel approach I never did research on in the past so ...	3.9	0.745	3.9

[ 726.8 WXDAI ]

@gentlementlegen

Contributions Overview

View	Contribution	Count	Reward
Issue	Specification	1	12.6
Issue	Task	1	600
Issue	Comment	5	114.2
Issue	Comment	5	0

Conversation Incentives

Comment	Formatting	Relevance	Reward
wildcard/unassign: we should create a plugin that checks cer...	12.6 a: count: 1 score: "1" words: 3	1	12.6
@Keyrxng I think this can be standalone because events would be ...	22.4 code: count: 1 score: "1" words: 2	0.92	22.4
So basically, whenever the user is detected as active, we push t...	22.4	0.835	22.4
In the provided example, it doesn't work because the calculation...	25.2 li: count: 3 score: "3" words: 41 code: count: 3 score: "3" words: 5 hr: count: 1 score: "1" words: 0	0.69	25.2
@0x4007 Might have had an idea on how to reduce load and trigger...	39.4 li: count: 3 score: "3" words: 27 code: count: 3 score: "3" words: 0	0.7	39.4
@0x4007 Interesting that the issue didn't close despite being li...	4.8	0.725	4.8
@Keyrxng I think this can be standalone because events would be ...	- code: count: 1 score: "0" words: 2	0.92	-
So basically, whenever the user is detected as active, we push t...	-	0.835	-
In the provided example, it doesn't work because the calculation...	- li: count: 3 score: "0" words: 41 code: count: 3 score: "0" words: 5 hr: count: 1 score: "0" words: 0	0.69	-
@0x4007 Might have had an idea on how to reduce load and trigger...	- li: count: 3 score: "0" words: 27 code: count: 3 score: "0" words: 0	0.7	-
@0x4007 Interesting that the issue didn't close despite being li...	-	0.725	-

[ 11.6 WXDAI ]

@Keyrxng

Contributions Overview

View	Contribution	Count	Reward
Issue	Comment	2	11.6

Conversation Incentives

Comment	Formatting	Relevance	Reward
I can only really think of this plugin being an invoker of #9, w...	8.4 code: count: 2 score: "2" words: 9	0.76	8.4
@jordan-ae is taking care of this task and yeah I understand. I ...	3.2 code: count: 1 score: "1" words: 2	0.865	3.2

0x4007 commented 1 week ago

I think the comment rewards are too high here. Especially with the occasional double counts

gentlementlegen commented 1 week ago

Didn't claim, so let me know.

0x4007 commented 1 week ago

Can claim I just need to adjust the settings I guess. https://github.com/ubiquibot/ubiquibot-config/commit/5ec796f09578609cd5e0836adf86559e40eba793

gentlementlegen commented 1 week ago

@0x4007 On this version the comments are evaluated like:

Issue	Comment	5	114.2
Issue	Comment	5	0

I believe that the v2 currently would give the second row (0 reward for comments).

ubiquibot / plugins-wishlist