Project maintenance/move project to marge-bot/marge-bot namespace? #295

Closed nejch closed 1 year ago

Hi all, I see this project has quite an active community for its size, but it seems to be pretty hard to get things merged. From what I see in other issues/PRs, this means a lot of people are running their private forks with their unmerged changes.

I'm not sure if this is still meant to be actively maintained or if it was something that solved the author's problem at some point but is no longer needed. But it'd be really helpful, and if some of the open PR's (from early 2020 in some cases) get merged this could be used/easily expanded to basically have quite a few advanced features/workflows in GitLab via marge-bot.

How about giving some of the more active community contributors merge access, or if that's not feasible, moving the project to its own namespace where they can help get rid of the PR backlog and open up the project for new contributions?

I know I'd have some stuff coming in because we're looking at using it for our on-prem projects and would need some changes to make it scale better for large instances instead of running an army of bots.

@tclh123 @JaimeLennox

/cc @brettdh @flaviouk @FlorianLudwig @Hi-Angel @lkm @matthiasbalke @snim2 @nirizr @qqshfox are some of the people I've seen contributing/interacting lately but there are probably more.

Just wanted to start the discussion, sorry if this seems intrusive. Thanks! :bow:

Hi @nejch

Thank you for bringing this up. I do think Marge bot needs more love and I'm glad there're new contributors like you. I think bringing in some active contributors to the Marge maintainer team is a good idea. It's not about having the rights to merge, it's about having a common vision of this project and guide us and new contributors to continually improve this project. Actually, I'm so glad that @qqshfox just became a new maintainer of this project. If anyone have passion on this just let us know. Cheers!

Thanks for the update @tclh123 great to hear!

I guess what I meant by merge access is, whatever helps getting PR's merged in a timely manner so that people are motivated to contribute. There are some really small PR's open where having a few maintainers would already help just to move things along.

Would it help you if people suggest ideas on next steps to take? If so here are just some thoughts I'd have on what would make sense to me, not necessarily in this particular order:

Add a fully native python development environment (@qqshfox is on this already, #291)
Cleanup/merge at least part of the PR backlog, especially the smaller/easier ones (like https://github.com/smarkets/marge-bot/pull/294)
Add GitHub Actions to always run existing suites and clean up tests if needed, review existing testenv/linting etc
Add HTTPS cloning support so people don't need to provide duplicate credentials (https://github.com/smarkets/marge-bot/pull/283)
Add support for codeowners & approvals: https://github.com/smarkets/marge-bot/pull/254
Add support for per-project config files (i.e. for project-specific options, read a yml from the default branch in the target project when looping projects)
Maybe use libraries to avoid reinventing the wheel for all of the above (I can take this as I already help with python-gitlab, but same would go for parsing codeowners)
Make MR processing non-blocking so marge can scale better to more projects (I haven't given this one much thought yet).
Add some integration tests against a running Gitlab container to really test the tricky stuff like failed rebases etc, that some people describe in the other issues. I have some experience there too if needed.

I'm sure others would have more ideas as well, this is just from what I've seen so far.

This is a good summary, thanks @nejch .

The "small issues" (your item 2.) do include some quite important changes, for example #180 and #169 may be useful for some users, and many would be simple to implement (or as a minimum it might be better to document some things rather than have them in the codebase). Some of the documentation / message issues such as #234 would also be very helpful.

Scale

Like you, I've also had issues with scalability. I had to extend the "wait for pipeline to resolve" setting to 10 minutes for one of our projects, but the result is that devs working on projects with faster pipelines sometimes think Marge is "broken" because they have no way of knowing that they are waiting on another merge.

The suggestion in your item 8. is a good one (issues #224 and #97 are also related), and I'd love to see that implemented.

Plugins

There are also a number of other issues around workflow, integration and configuration, that might be resolved by adding "hooks" into the core merge algorithm, so that users can write plugins to either expose the bot's current state or integrate marge into another system or their usual workflow. These issues might include:

296 Support for extra pre-merge checks
145 Use emoji reactions instead of approver systems
138 Add label on issue when MR is merged
147 Show Marge status
290 Allow Marge to re-assign to last assignee after merge

I left a comment on #138 about how this might be implemented, but it needs more thought, especially around concurrency.

The reason I mention this, is that although Marge is not huge, the core algorithm is relatively complicated, and if we want it to support a wide range of workflows, we need to avoid adding complexity in the codebase as far as possible. Allowing users to create their own workflow would give Marge that flexibility. Those workflows might well include quite arbitrary integrations, such as sending Marge's status to a Slack webhook, or setting the bot user's GitLab status...

At the very least, more complexity could be added to a plugin system (which might provide some support for posting comments on issues / MRs, adding/removing labels, etc.) which would be separate from the core Marge algorithm.

State

Marge currently doesn't save any state, I believe, but issue 1 which would be nice to have, suggests that it might need to. This isn't a high priority issues, but it's the only one I could find that might require substantial changes to the current core algorithm.

Like you, I've also had issues with scalability. I had to extend the "wait for pipeline to resolve" setting to 10 minutes for one of our projects, but the result is that devs working on projects with faster pipelines sometimes think Marge is "broken" because they have no way of knowing that they are waiting on another merge.

I think my #293 improves this. Within and outside of a single project (We have an hour long pipeline :X ).

Allowing users to create their own workflow would give Marge that flexibility.

+1 for plugins and hooks. I've added a functionality to my Marge fork that forces the "only allow merge when pipeline succeeds" and protection for the master branch, to fight high privileged users that occasionally turn those off to bypass tests :-X

I can see this is something nobody will want in Marges' codebase but will benefit few users.

Thanks @nirizr I like the idea of #293 but I think one of the problems you've highlighted is that we need some good issue triage, because there are several possible approaches here, and with some of the other feature requests.

Just wanted to follow up on this two months after. It's nice that there are ideas and direction laid out here but I still think it's more about getting real activity here and just merging in the basic stuff first, before the rest can even be addressed.

Otherwise contributors have zero confidence their PRs will even be considered, see https://github.com/smarkets/marge-bot/pull/254#issuecomment-823197267. @tclh123, can @qqshfox as the new maintainer help merge the existing backlog if others don't have time? See e.g. even trivial PRs like dependency updates not getting attention etc.

If not, I think forking to a more maintained project will start to make more sense for people as @lkm said in the other issue, I'd just try to avoid that as then you have PRs and issues all over the place.

Just wanted to check in and see if there has been any progress or update on #147.

Thanks in advance.

Hi all! I'm also interested in seeing this moving forward 🚀

marge-bot is already taken as github username, but margebot is still not, is anyone interested in joining the effort there? Is anyone from the actual maintainers team active and willing to help on the transition? Or setting a plan to help get PRs merged? 🍀 🤞🏾

I'm thinking about waiting after Christmas period is over (January 10th 2023) and if no reaction has been made, I'm willing to create a new organization here in GitHub and ask on PRs to move over there.

Hopefully before that we have reactions and comments here? Anyone would like to join, so they can be added from the first day? 🤝 🎉

I'd love moving my PR (and perhaps a few smaller changes) to a better-maintained fork. It seems current repo maintainers have other priorities nowadays and I encourage a fork if no progress can be made in the current repo.

Sorry for the late reply @gforcada. I don't want to overcommit as I'm involved in a few things atm, but I'd be open to help as well if you migrate. Maybe at least migrating the backend to python-gitlab (which I help maintain), to avoid reinventing the API client every time here (as you did in https://github.com/smarkets/marge-bot/pull/362 etc, but would also help with other issues that pop up here).

Edit for context: I've given up a bit on this since the original post almost 2 years ago as there was not much movement - many of the PRs above have not been merged - but if there is a fork with more activity I'll probably reconsider :)

@nejch What about talking about it over the beer?

I said January 10th, but work and family took me by a storm 😅 if there is still interest I will try to do it this weekend or during next week 😄 🍀

@nejch What about talking about it over the beer?

@okias sorry for the late reply, I'm definitely up for beer to talk about this if you're also in Prague!

@gforcada sure, we've been talking about this for years so no rush. I would be interested to hear what @okias has to say as I know the people at https://gitlab.freedesktop.org/ make extensive use of marge-bot too so would definitely be nice to start with a healthy community. If you catch a train down to Prague maybe we can all talk over beer :sweat_smile:

GNOME is also using it, it would indeed be interesting to get a bit of momentum again and find a place were we can collaborate 👍🏾

Any 🍻 ... I mean progress on this front? 😄

Any :beers: ... I mean progress on this front? smile

Hey @gforcada I'm probably meeting @okias later this week :beers: .. I just need to respond to your message @okias :sweat_smile:

Will keep you posted on the weekend!

Beverages aside, has there been any progress re: getting this project into a more maintainable state? 🙂

Intent to create FDO GitLab namespace

Before you ask why FDO GitLab: it's shared space for basic stones of FOSS world (GNOME and KDE, where GNOME already uses and patches Marge), also it's independent from the GitLab and running FOSS version (and that's not going to change). I'm not sure how would GitLab felt about hosting project which brings some "premium" functionality to FOSS GitLab version. Hosting on GitHub could work, but why when we have GitLab, right :wink:

We finally met up with @okias last night :wave: here's a quick summary:

hosting on GitHub.com/GitLab.com with a canonical-sounding namespace might give the new community fork more visibility, but GitLab probably makes a bit more sense since it's for GitLab :grinning:
we could try to claim https://gitlab.com/marge-bot via GitLab's name-squatting policy, I've done this in the past and the account looks dormant so I could give it a shot. That would give us https://gitlab.com/marge-bot/marge-bot.
add direct access for all the people here that are active or maintain their own forks (a recent one seems to be https://github.com/hiboxsystems/marge-bot)
for all issues/dormant PRs here that get merged in the new community fork we could inform the users this is fixed on the fork
we could maybe open a PR here if the current maintainers are willing to merge a small readme note pointing towards the new fork if this remains unmaintained here.

How does that sound /cc @gforcada @nirizr @snim2?

Edit: for now I've requested the namespace:

Your GitLab support request 389126 | 'Namespace request for gitlab.com/marge-bot via name squatting policy' has been received

Sounds good to me

On Fri, Mar 24, 2023, 11:11 Nejc Habjan @.***> wrote:

We finally met up with @okias https://github.com/okias last night 👋 here's a quick summary:

hosting on GitHub.com/GitLab.com with a canonical-sounding namespace might give the new community fork more visibility, but GitLab probably makes a bit more sense since it's for GitLab 😀

we could try to claim https://gitlab.com/marge-bot via GitLab's name-squatting policy https://about.gitlab.com/support/gitlab-com-policies/#name-squatting-policy, I've done this in the past and the account looks dormant so I could give it a shot. That would give us https://gitlab.com/marge-bot/marge-bot.

add direct access for all the people here that are active or maintain their own forks (a recent one seems to be https://github.com/hiboxsystems/marge-bot)

for all issues/dormant PRs here that get merged in the new community fork we could inform the users this is fixed on the fork

we could maybe open a PR here if the current maintainers are willing to merge a small readme note pointing towards the new fork if this remains unmaintained here.

How does that sound /cc @gforcada https://github.com/gforcada @nirizr https://github.com/nirizr @snim2 https://github.com/snim2?

— Reply to this email directly, view it on GitHub https://github.com/smarkets/marge-bot/issues/295#issuecomment-1482410419, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA3WUVYJNSPBSFZEVR7MFWLW5VJJTANCNFSM4YXUCQWA . You are receiving this because you were mentioned.Message ID: @.***>

Looks great!

Just a quick update for transparency: GitLab seems to no longer review namesquatting cases unless you have a subscription on GitLab.com. I don't know who owns marge-bot, but I've tried to reach out to another user on gitlab.com who has a similar namespace that I think may be useful, I'll update if I have any luck.

I've not had any luck with reaching out to users. Maybe just one last attempt here, /cc @pulicean just in case you happen to be the owner of https://gitlab.com/Marge and are willing to rename the username, otherwise sorry to bother you here! :bow:

For now as a workaround I created https://gitlab.com/marge-org. It sounds generic and close to gitlab's own structure, let me know what you think, I can already give access to people interested.

So we could perhaps import it as https://gitlab.com/marge-org/marge-bot or simply https://gitlab.com/marge-org/marge, @okias' preference is for marge-bot for better searchability, I also like just marge as it's clean but no hard opinions.

I've already played with the API in a local gitlab instance and was able to fully import all issues/merge requests, activity, branches from forks with opnen PRs etc. So IMO it'll be easy to continue the work/discussions started here:

PS: I also realized that the original maintainers we interacted with in this issue, potentially no longer work for smarkets, and so would not be able to help us here in any case, right @qqshfox @tclh123? (edit: since they both appear as Contributor here now, I think that's almost definitely the case).

Imported: https://gitlab.com/marge-org/marge-bot. We can still move it to https://gitlab.com/marge if we get a response, IMO.

Imported via a triage bot created for this purpose (https://gitlab.com/marge-triage-bot). We might be able to use this bot later to triage issues and MRs (via https://gitlab.com/gitlab-org/ruby/gems/gitlab-triage). Some of you will see that your activity was replicated there, as GitLab mapped usernames via emails. I hope this hasn't caused too much noise for anyone.

I say we close this and continue in https://gitlab.com/marge-org/marge-bot/-/issues/295.

See also https://gitlab.com/marge-org/marge-bot/-/issues/373 if you'd like to get involved :)

smarkets / marge-bot