GitLab support #2169

Open knocte opened 6 years ago

User Story

as a repo maintainer, I want to be able to use GitCoin if I have a repository in GitLab instead of Github

as a user, I want to be able to fund issues in GitLab repositories, not only Github ones

Why Is this Needed

Summary: It's important that Gitcoin stays agnostic to the git platform being used (otherwise it should be called githubcoin! :P ). Especially given that GitLab is opensource (as opposed to Github) and aligns better with Gitcoin's own opensource nature too.

Also, given that GitLab can host private repositories for free (as opposed to Github, which has this as a commercial service), this feature is more likely to be useful when #3449 is implemented.

Description

Type: Feature

Many users prefer GitLab to GitCoin for variety reasons.

To me it's also question of the spirit of space : decentralisation. Providing support to one then one platform is good for many reasons.

Please do provide:

support for GitLab projects or separate version of gitcoin for gitlab?

Please consider providing until it's implemented:

documentation how to use GitCoin for GitLab users, ok instructions how to brindge via GitHub (e.g. sync to GitHub, make bounty, accept pull request, and sync back to GitHub, or other operations, preserving authors signed commits (if possible and if author signs commit), etc.)

Related reddit thread: https://www.reddit.com/r/gitcoincommunity/comments/attpyx/how_to_use_gitlab_projects_with_gitcoin/

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 0.28 ETH (41.21 USD @ $147.18/ETH) attached to it.

If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
- Want to chip in? Add your own contribution here.
- Questions? Checkout Gitcoin Help or the Gitcoin Slack
- $59,065.49 more funded OSS Work available on the Gitcoin Issue Explorer

I just put about 40USD as a bounty for this. @gwpl you're welcome to vote with your wallet to this issue now, let's make this happen! (you can add extra funds to an existing bounty, last time I looked gitcoin already supported this crowdfunding-mode model)

This is nice, but what do you think of adding login with GitLab as well?

This is nice, but what do you think of adding login with GitLab as well?

Fine by me of course, but maybe can be considered a separate issue (that you're welcome to file separately). Bounties should stay as small as possible in scope so that a bounty-hunter can have a clear goal.

please review the work and list todo-list to clean the workround.

please review the work and list todo-list to clean the workround.

Sorry, what?

@knocte i suggest list a checklist to help what need implement. e.g.

Sorry but I only see 1 item in that checklist:

[ ] Make gitcoin support gitlab.

I don't see how this can be more simple really :)

I'm guessing, though, that a developer might understand this better if what I explain here is, instead of a checklist that needs to be done, an "acceptance test" that I would perform to check if the work covers the full picture. So, I would do this to test this work:

1) Fund an issue that is in gitlab.com instead of github.com Expected result: issue should get a gitcoin comment with the info about the bounty. 2) Make a worker apply to the issue. Expected result: issue should get a gitcoin comment explaining that the work is in progress. 3) Make a worker submit a pull request in gitlab.com that references the gitlab issue. Then worker should click on "submit work" in gitcoin. Expected result: funds contributor should get an email notification that he needs to review the work. 4) Approve work submission. Expected result: bounty should be payed out to the worker, and gitcoin should send a message to the github issue about this.

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 11 months, 3 weeks from now. Please review their action plans below:

1) svipal has started work.

Hello ! I've been looking at this.

From what I've gathered here's the action plan for the first part (accepting the issues).

make gitlab equivalent functions to the already existing github ones in app/git/utils.py concerned functions (only those that deal with adding issues, not the github auth ones): github_connect get_gh_issue_details get_gh_issue_state *Means there is a need for importing https://salsa.debian.org/debian/python-pygitlab
properly route gitlab issues to said equivalent gitlab functions in app/dashboard/helpers.py:issue_details
allow gitlab urls in app/assets/v2/js/shared.js:retrieveIssueDetails
change app/dashboard/templates/bounty/new.html to reflect the new dual github/gitlab policy
- change "edit on github" and "sync with github"
- remove the github logo in the style.

Questions, observations :

in the issue_details function in app/dashboard/helpers.pythere is a TODO :
- Modify the view to only use the Github API (remove BeautifulSoup).
- Simplify the view logic. What should be done ? Just update the todo to include the Gitlab API ?
I don't know yet how similar these two APIs are. I feel like a lot of the site was made with only Github in mind. Links for comparison https://python-gitlab.readthedocs.io/en/stable/api-usage.html https://pygithub.readthedocs.io/en/latest/
there's a need to separate the github auth stuff from the github issue stuff. Could anyone familiar with that side of the project tell me what's up with the gazillion GITHUB_... variables in app/app/settings.py ? there's a #TODO next to most of them.

In particular I'm not sure about the following things, should we :
add social_core.backends.gitlab.GitlabOAuth2' to the AUTHENTIFICATION_BACKENDS in app/app/settings.py
add gitlab specific settings to app/app/settings.py
add gitlab specific env variables to app/local.env At first glance, seems like we would only need GITLAB_API_TOKEN for the issues. But there's also a #TODO to it in settings, maybe it's planned future functionality.

Learn more on the Gitcoin Issue Details page.

@svipal: if you try to migrate every github occurrence that you find in the codebase, you might never finish, I mean, it could be a job that takes too long because there may be a lot of github-related features. What I proposed in #issuecomment-466855718 though, is that you take the simplest functionality, and make that work. Change your OS' hosts file to make the domain github.com point to garbage (e.g. 0.0.0.0) and try the acceptance test until you find the first problem, then fix the code to use gitlab instead and restart testing the acceptance test. Repeat this until the whole acceptance test works.

I'm not trying to blindly migrate every github occurence, don't worry. That's what I meant by "there's a need to separate the github auth stuff from the github issue stuff". I only want to deal with github issues for now, and not touch the auth unless it's needed. (it's needed at one point during the github issue validation process, that's why I was talking about oauth and the API token) I agree with the idea of taking the simplest functionality and making it work - that's why I said the first step was to accept gitlab issues.

Been doing some more digging.

There's also web/app/dashboard/models.py where every issue is assumed to be a github one , need to weave gitlab into that.
Need to change create_new_bounty function in app/dashboard/helpers.py:issue_details so that it calls the gitlab eq to get_gh_issue_details if the issue is a gitlab one.

I'm planning on short term to just make it work following the above acceptance test; then ideally what should be done is do a repo agnostic layer so that it's easy to add others on top of it such as bitbucket.

Progress : Gitlab issues are correctly synced on the new bounty page. Try out my fork to see it in action. You will need to set a personal access token for your gitlab account, then add it in the .env at GITLAB_API_TOKEN to make it work. Will work on both public issues and your personal private issues. Next steps : immediately : understand why it's hanging when I press the fund new issue button~ Then : probably need to write a gitcoinbot for gitlab.

@knocte Expect a working build in 2 weeks tops. I know I can do it, but to be honest; I think the scale's off. It's a project of such importance to Gitcoin that it's not to be done by a lone dev on a $40 bounty. Not really because of the work but because of the conventions : I have already , and will have to rename a lot of functions in a lot of parts of the project , add env variables, add conditions in existing functions, etc... I will probably end up being a mess when it comes to merge it even if it works properly on its own. Would love to get people on board with this.

I'd be happy to add some ETH to the pot.. How much do you think is warranted? Also, I'd love to know how you're abstracting the app from just Github..

@owocki you can look at the commits on my fork to see what I've been doing to fetch the issue. https://github.com/svipal/web/ basically import the gitlab API python bindings and do the same steps or so. Right now it's very dirty but you can see the idea. Concerning the ETH, I think about 200$ worth ? Since it's a collab bounty it would be nice to get multiple people working on this =) @iamonuwa suggested to do the gitlab oauth also, and it might be a good idea, but if we do so, there's gonna be a lot of work ahead !

@svipal @owocki I am ready to contribute if it the work increases and requires 2 people. I also agree the scale is off.

I checked out https://github.com/svipal/web/ , specifically this diff

I'm excited about this direction, but i think we should provide a cleaner layer of abstraction in the code.

Instead of get_issue_details implementing an if/else statement for github/gitlab, i think there should be seperate functions (if not seperate classes) to implement parallel functionality. This will make git/utils much more sustainable long term, when we add bitbucket + OSCoin and others.

Let me know feedback on the above approach. Happy to fund this if it's something you are interested in working on still.

@owocki I'm indeed working on it still. I agree - it's dirty - as I said on the long term there should be a clean repo agnostic layer before the function calls. I was just quickly making it work there =) Progress : I'm working on a gitlab gitcoinbot. I'd like to talk to someone that worked on the GH gitcoinbot about the details.

According to what I've seen : The only problem with gitlab bot integration is that you can't create an app that will automatically install itself on the repo. if you want your repo to respond to a bot you'll have to add the webhooks yourself.

Note: I'm talking about the gitlab.com server. If one is hosting his own gitlab server then it's very possible to write a gitcoin service that would do that.

Progress : Good news - the changes I made are compatible with the new gitcoin version with easy auto merge. Bad news - It is gonna be tough. It's impossible to make gitcoin 100% gitlab compatible on the issue side without also implementing oauth for gitlab, One exemple of things where it required atm : the current gitcoin bounty url format is https://gitcoin.co/issue/user/repo/issue By default it's bound to github, which isn't the problem, I thought about reserving an url like this for gl bounties : https://gitcoin.co/gitlab/issue/user/repo/issue But ! in the code, the github auth is woven into the bounty details retrievals. Meaning if we want to do this cleanly, we have to weave the (currently unexisting) gitlab auth into the gitlab bounty details retrievals. Or rethink it altogether. There's actually a lot more to it that makes it more complicated than I expected, but bottom line is : Scope is even bigger than I thought, turns out we're gonna need to actually make gitlab alternative pathways for the current github stuff. Not as simple as just making a single layer as github permeates the whole project. There's gonna be the bot, the user profile, the issue retrieval, url formatting, etc... I'm dropping this until more people come along but will stay open to questions and discussion. Come on people, jump in this boat, gonna be a long trip.

@owocki how can we move this forward? can any technical fellow from your team chime in wrt the url issues that @svipal described above? And when it comes to funding, maybe mentioning this gitcoin issue in twitter might increase pledgers? Thx

it'd be helpful to get a WIP PR up so we can see whats been done so far and how.

It's impossible to make gitcoin 100% gitlab compatible on the issue side without also implementing oauth for gitlab,

I dont see why we couldnt do this.

the current gitcoin bounty url format is https://gitcoin.co/issue/user/repo/issue

I'm fairly certainly we could keep this bounty url format and not have any namespace collisions. But even if we didnt dashboard/models.py:bounty.url() is the only place we'd need to change (in addition to urls.py)

Scope is even bigger than I thought, turns out we're gonna need to actually make gitlab alternative pathways for the current github stuff.

Respect. Yes it's a big task!

Here is a tentative recap of some gitcoin architecture points where gitlab pathways are needed :

Issue fetch : done in WIP - changes are described in the latest commit and in this thread.
Bot :
- command bot : easily done because API supports webhooks , but caveats due to GL diffs
- issue status report : haven't looked at this yet.
user profiles and Oauth. Meaning also, matching gitlab env variables.
The bounty_details function in dashboards/views.py is probs one of the worst offenders. it is called when a bounty url is accessed by a user and it assumes everything is github.
bounty urls : namespace collision is bound to happen one day. There are ton of project mirrors on gitlab (https://docs.gitlab.com/ee/workflow/repository_mirroring.html), or projects that moved by the same dev, etc.. Even if it doesn't in a reasonable time frame - to say, ok I got the bounty url now let's look if I get a github proj with that.. No doesn't work, ok let's try gitlab... isn't optimal.
naming conventions : For instance, issue urls in general are considered to be "github_urls" throughout the whole project.

Would a pull request for the sake of discussion be helpful? There's nothing more that's been implemented that works since last time, except cleanly separating the gitlab and github issue detail retrieval functions.

Would a pull request for the sake of discussion be helpful?

For the sake of collaboration (so that potential contributors can jump in and continue the work), I would say yes.

This is one of the open heavy issues - good points in here.

Assigned to me? I'm the backer not the dev :)

Closed without reason?

I still hope it will be implemented one day :)

@gwpl so this is not something we'd be able to accommodate this year! This is def something we'd need to do eventually but I'm not sure when we'd want to do it cause it would involve revisiting the whole auth flow and break it onto multiple milestones to ensure this feature can be done smooth

PS: sorry about the closing without a reason -> sad the entire day closing out old issues which were either fixed / not something we'd be able to do now.

@thelostone-mc Is there a workarounf flow for managing code using gitlab and mirroring with github to use the gitcoin api? Can I still add some funds?

all the features will work as per normal, but the only one thats really super dependant upon github is bounties, bc u have to put a github issue URL into the bounty creation form. for that, the workaround is just to create a github repo with a github issue, and in the text of the github issue describe that the repo lives on gitlab.

@owocki Thanks for the heads up. To make sure I understood you correctly. If i were to hypothetically make a bot, would this be the flow?

not something we'd be able to do now.

That still doesn't sound like a reason to close an issue. As far as I understand, gitcoin is opensource so anyone could tackle this, not just the gitcoin team. Closing, in github-speak, means "won't fix".

@knocte there was an assumption (not sure if conscious) that gitcoin was going to bee deeply tied to github apis. Changing this is almost as good as rewriting the whole thing from scratch. You can't even auth without a github account. Usually when there's this kind of issue, there's something major decision like: a v2 rewrite, a fork, or a massive overhaul. This is bigger than gitlab per se. The real issue is how do you support other git clients? gitea? gogs, bitbucket? How about svn? What about design work done? Figma or Sketch work? This involves a much higher level of abstraction with a totally new api. We look at things like end users, but they look at it like architects.

Issue Status: 1. Open 2. Cancelled

The funding of 0.28 ETH (67.85 USD @ $242.32/ETH) attached to this issue has been cancelled by the bounty submitter

Questions? Checkout Gitcoin Help or the Gitcoin Slack
$109,704.15 more funded OSS Work available on the Gitcoin Issue Explorer

i'm re-opening the issue by way of popular support

Very interesting discussion. There is a lot to think about here.

+1

Gitlab is vastly more aligned with the open source values of Gitcoin than Github, and it's honestly a better tool. Minds has completely switched and we would never go back. Happy to help.

gitcoinco / web