search

argoproj / argo-cd

Declarative Continuous Deployment for Kubernetes
https://argo-cd.readthedocs.io
Apache License 2.0
17.71k stars 5.4k forks source link

Automatic cleanup for stale issues #20011

Open reggie-k opened 4 weeks ago

reggie-k commented 4 weeks ago

Summary

I propose to outline criteria according to which issues would be decided on as "stale", and then to auto-close those issues.

Motivation

We have ~3k open issues and having a more manageable number (at most 2k and ideally closer to 1k) can help us work more effectively. Closing the issues manually does not seem feasible.

Proposal

Possible criteria (the numbers were produced last week but they give an estimation of how much we could close based on the criteria):

  1. updated > 1 year and comments < 5 and reactions < 5 = 1081 issues
  2. updated > 1 year and comments < 5 = 1234 issues
  3. updated > 1 year and reactions < 5 = 1226 issues

The issues can be closed with a nice respectful comment, inviting the user to re-open the issue if still relevant.

Things to consider:

crenshaw-dev commented 4 weeks ago

I think I'm an outlier in this, but I really dislike auto-close bots.

First, I think it's a bad experience for issue authors. If an issue goes unanswered, I may think, "darn, no one has time to look at this, but such is life." If something auto-closes, I'm annoyed that I now have to either re-open the issue to keep it from slipping completely off the radar or accept that it's been moved to an even less-visible state. Even if the automated "we're closing this issue" message is nicely-phrased, I know it's still just a bot.

Second, it's a bad experience for readers (both end users and maintainers). If I click a link to an issue and see the purple "closed" label at the top, now I have to scroll to the bottom to answer "okay, was it auto-closed, or was it actually closed by a human?" The "closed by a human" and "closed by a bot" states have very different connotations for the reader.

Finally, I don't think auto-closing actually adds any value. The number of open issues would come to represent "recent/active open issues," but 1) I'm not sure anyone really needs that number and 2) if they do need that number, they can get it via issue search filters. And I don't really need a bot to tell me whether an issue is old or inactive, I can just glance at the timestamps.

christianh814 commented 4 weeks ago

I only 1/2 agree with Micheal. I really do think that we shouldn't outright close the issues, but mark/label them as "stale" first, then (after X amount of time of no activity) close them.

I do feel we need to close some issues that are old but I'm not a big fan of aggressively closing them.

crenshaw-dev commented 4 weeks ago

I'm 100% in favor of a stale label. It's purely additive and doesn't complicate any existing user or maintainer workflows.

jannfis commented 4 weeks ago

As for some history, we used to have a stale bot. It used to mark issues as stale, and IIRC it even closed issues after a while an issued marked stale didn't receive any further input. But, I don't know what actually happened to it. I think it just stopped working one day.

But I also am in favor of using a stale label. It's good for filtering, and we can decide later what to do with them.

Second, it's a bad experience for readers (both end users and maintainers). If I click a link to an issue and see the purple "closed" label at the top, now I have to scroll to the bottom to answer "okay, was it auto-closed, or was it actually closed by a human?" The "closed by a human" and "closed by a bot" states have very different connotations for the reader.

I agree, happened to me too often too. I wished there would be a dedicated status for issues that go stale, something like hibernated or so.

todaywasawesome commented 3 weeks ago

@crenshaw-dev @jannfis setting the bot aside, I'd like to do a large cleanup based on what @reggie-k has found. Basically take and close all the super stale issues right now. updated > 1 year and reactions < 5 = 1226 issues to start with. That would be ~1200 issues.

Would you be opposed to that as a starting action?

crenshaw-dev commented 3 weeks ago

@todaywasawesome not really, not yet. I'd like more details on what "having a more manageable number" means, concretely. In other words, what's the actual benefit to maintainers and to users of closing these issues? I'm sure there are benefits, but I think we need to describe them in order to weigh them against the downsides I listed above.

updated > 1 year and reactions < 5

I like the reactions rule. Can you describe what "updated" means here? Does that mean "no one has commented in a year"? Can you share your search query? Would be helpful to spot-check some of the issues to see whether we'd intuitively close them.

crenshaw-dev commented 3 weeks ago

Additional question about your proposed cleanup: would you be able to add a "we're closing this because of stale-ness" message to each closed issue, or is that only available for bot closures?

Another downside to consider is the loss of relevant statistics. Today various tools (LFX Insights, devstats, OSS Insights) collect information about opened/closed issues on a weekly/monthly/etc. basis. Those stats let us estimate how we're doing in terms of managing the issue workload. Automated closures, and especially mass closures, pollute that data and make it much less useful for estimating workload vs. capacity. To put it concretely, today I can go to my bosses at Intuit and say "I need help handling issues on Argo CD, here are the stats to prove it." With automated issue closure, those stats become much more difficult to produce.

reggie-k commented 3 weeks ago

@crenshaw-dev @todaywasawesome Sharing the possible GiHub UI search queries for what is above in the description (the date is an example for GitHub UI filter and can be a rolling window using GitHub CLI): is:issue is:open updated:<2023-09-01 reactions:<5 or: is:issue is:open updated:<2023-09-01 comments:<5 or: is:issue is:open updated:<2023-09-01 comments:<5 reactions:<5

"updated", in my understanding of GitHub API, should reflect any change made to the issue (comment, reaction, status change), I can verify this on some issues with such recent activities. Yes, the closing can be done with a comment "we're closing this because of stale-ness", without a bot.

I wonder how projects like K8S that use stale bots, deal with the closed issue stats, but I think that even if we decided as a group to dedicate a week and close many stale issues manually, this would also effect those stats, and I think this is normal. Maybe those K8S bots run weekly/daily and close an amount that does not hurt the human stats badly, and maybe the first time such a bot was introduced, that bulk closure did effect the stats one-tme, but a one-time outstanding peek is probably normal and indicates massive cleanup. And for internal presentation needs I think that the bulk-closed issues can be filtered out by dedicated labels, that can be put during the cleanup, along with the comment. Also, a massive cleanup can be performed by a dedicated GitHub token, to be filtered out later.

For me, the benefits of having only relevant issues in the repo are:

agilgur5 commented 2 weeks ago

significant downsides

100% agree with Crenshaw. Auto closing without any maintainer input has no real benefits yet has many downsides -- i.e. is largely counter-productive. The only "positives" I've seen are "number is smaller", which is a false positive (see below), and "prioritize newer issues", which is not necessarily a good thing in and of itself, and can be done with existing filters already.

Auto-close bots are also commonly derided and widely hated by most users (including in downvotes and subsequent comments); see also "Stalebot considered harmful". That includes for k8s issues, so I'm not sure a k8s practice that users strongly dislike is a good example to draw from. I've never seen a user want a stalebot.

false positives

work more effectively.

  • Psychological benefit of feeling less overwhelmed by the huge 3k number

  • [...] not necessarily for the users that opened those old issues, but for new users that take a first glance at the repo and see 3k open issues

Issues that are closed as stale but not resolved are all false positives and so all those supposed benefits are false positives, while still adding many downsides such as frustrating users, making duplicate issues, making searching harder, unnecessary pings from bots, etc.

It also makes closed issues not very helpful for historical purposes either as the "closed" marker no longer holds meaning, as Jann and Crenshaw mentioned above.

"Any updates?" comments are also not helpful, but such a bot would actually encourage them. And may cause threads of "marked as stale" -> "not stale" -> repeat, which pollute a thread and cause unnecessary notifications too.

effectiveness of metrics

Regarding "metrics", please keep in mind common fallacies about those such as Goodhart's Law; if a lower number is a target, that will be achieved despite the benefits or downsides of it. Similarly, the "age" of an issue should not be a target as it is inherently neither good nor bad. Statistically, issues are opened, commented, and voted on voluntarily, so that has inherent biases as well that reduce reliability for "activity" metrics.

In the same manner, a large number of open issues or PRs is not in and of itself a bad thing. It is primarily an indication of 1. popularity and 2. ratio of users to maintainers, or to be more accurate, user time to maintainer time (and 3. sometimes ratio of bugs to recent releases, but not always, and not all issues are bugs). A stalebot inherently does not handle that.

agilgur5 commented 2 weeks ago

prior art in Argoproj

Also this has been discussed and modified in other Argoproj repos. For instance, in Workflows see https://github.com/argoproj/argo-workflows/issues/11802#issuecomment-1720117452 where I outline all scenarios for staleness and then address the few automatable scenarios with a more targeted action that requires a human label indicating that "more information from the user is needed" first: https://github.com/argoproj/argo-workflows/pull/12488 (problem/more information needed is the label name in Workflows's case)

It's also implemented in Argo UI in the same way: https://github.com/argoproj/argo-ui/pull/549

I would suggest a similar approach if CD is considering it. That is, humans do the hard, non-automatable work of filtering issues, and bots can help follow-up on them.

I also listed some simple counter-examples to auto-closure in https://github.com/argoproj/argo-workflows/pull/11836#issuecomment-1766661155 -- it's pretty common to have popular issues closed by a stalebot, as age is not necessarily correlated to correctness nor importance. (popularity itself is also not inherently good or bad either; a bug is a bug, regardless of its age or popularity)

agaudreault commented 1 day ago

+1 on adding "stale" label automatically, but manually closing stale issues for all the reasons above. There's also a need to label an issue to be non-stalable, so the stale bot does not pick it up. For instance, "feature request x" is stale, but a valid low priority request that someone might decide to implement one day.