search

dotnet / issue-labeler

An issue labeler bot for use in dotnet repositories.
MIT License
22 stars 20 forks source link

Enhancement: prioritize training over who it labeled it (human vs auto by bot) #13

Open maryamariyan opened 4 years ago

maryamariyan commented 4 years ago

Enhancement suggested by @jkotas

If an issue was labelled by human we can assume that the label is right, but if assigned by labeler automatically there is a chance of error. Therefore, training the bot on its past decisions using incorrectly labeled issues which were never updated, may lead to amplification of wrong behaviors.

In general, it may be a good idea to give higher training priority for cases where the label is assigned by human.

maryamariyan commented 4 years ago

Note for future:

I don't know of a github API to query name of users who last updated a label on an issue/PRs. I could either use that for this case or start keeping track of last updated area-* label event per issue/pr in the issue labeler instead.

danmoseley commented 2 years ago

I'm not sure this is feasible through the API but I'm also not sure there is much of a useful signal here. Sometimes humans make mistakes; also when the bot gets it wrong a human will fix it, almost always. So if a label was written by a bot, and not changed by human, it ought to be about as reliable as if it was changed by a human.