Add magiclink support for private GitHub / GitLab url #933

Closed crivetimihai closed 9 months ago

Using magiclink with a custom GitLab or GitHub URL (ex: private repo).

Please allow customization of the url parameter in pymdown-extensions/pymdownx/magiclink.py

Please be very specific. This request is currently too vague.

Do you just want a customizable host name? Anything beyond that I'd have to probably pass on. I'm unfamiliar with all the solutions a company or person would run locally. I know my company runs a version of bitbucket locally, but they use email addresses as usernames. Such user name requirements break the syntax. Some run older versions as well which may cause link references to be different, either that or enterprise links are just different to begin with. All of this is beyond what I'm interested in supporting. A way to customize host is probably as far as I'm willing to go right now.

This is of course assuming this kind of customization is what you are talking about. I'll give it a bit and wait for your response. If I don't hear back, I'll just close the issue.

@gir-bot remove S: triage @gir-bot add S: more-info-needed

Pinging one more time for additional info.

Hi! Just the hostname URL, for example, to use 'github.mycompany.com'. Just changing the URL seems to work great with private github deployments.

For example, adding this to the TravisCI pipeline allowed me to use a custom (private) GitHub deployment. Having the same options through mkdocs.yaml make sense (ex: being able to override the base url variable)

sed -i 's/github\.com/github\.mycompany.com/g' /home/travis/virtualenv/python3.6*/lib/python3.6/site-packages/pymdownx/magiclink.py

I don't know if this will work in all providers currently, but this was the code from my first experimentation. Like I said, I don't think it works in my company's enterprise bitbucket, but it will likely work with some providers, so we will most likely base the code off this. If a provider changes expectations of links between versions or allows more complex usernames in another version, this will of course not work, but if only the host name is changing, then this will likely work. That is all we can promise and will probably document it as such.

Anyways, the linked code needs to be rebased onto the most recent MagicLink and updated to work with the recent changes.

@gir-bot remove S: more-info-needed @gir-bot add T: feature, C: magiclink

Hi there. I'd like to second this request. We're using a self-hosted gitlab instance. There is no difference with gitlab.com with respect to format of usernames, issue references etc, only the hostname differs. If I were able to configure provider_url in addition to provider I'd expect it to work just fine.

Yeah, it is on my list to look into. I'll probably allow the user to set the host, and if it doesn't work for their provider, 🤷. I probably won't add any specialized support beyond that. If provider URLs differ or if username requirements for a given private provider differs, the user is simply out of luck.

@facelessuser I was just having a look for a possibility to define a custom provider and found your idea a while ago that moreless matches to what I had in mind and would satisfy this issue as well, I think: https://github.com/facelessuser/pymdown-extensions/issues/583#issuecomment-497981849

In our case, we'd only need to allow shortened user profile links, I see now that it is more complex when one wants to define issue/commit/PR links, which I guess is why you had in mind the "type" option to adopt existing providers URL schemes. I guess that would work in our case with Twitter, which as well only allows user profile links? Only issue is the wrong/missing shorthand icon. A custom image URL and/or unicode character is all I can currently think of.

The more flexible I make the extension, the more complex it gets behind the scenes. It's one of the reasons I haven't quite got around to addressing this yet.

I guess that would work in our case with Twitter, which as well only allows user profile links? Only issue is the wrong/missing shorthand icon. A custom image URL and/or unicode character is all I can currently think of.

@MichaIng I'm not exactly sure what you are referring to here. What do you mean wrong/missing shorthand icon?

What do you mean wrong/missing shorthand icon?

As I understand your idea, it was not to allow completely customised providers with custom icons, and an own set of associated page types, but more to allow a customised version of an existing provider "type", which shares the same set of page types (like user profile, repo, pull request, issues, commits for type: github), and probably the same icons (like Octocat icon for shortened GitHub repo URLs), but with custom base URL to allow private or business GitHub repositories or links to a custom GitLab instance. That should serve the need in this issue and the one I linked.

I personally could need a custom forum provider. But since it's about user profile links only, with the above approach, type: twitter should do fine. Forget my sentence about the icon, I thought there would be a Twitter icon shown with shortened Twitter URLs, but it's an @ only, so that is perfectly fine.

The more flexible I make the extension, the more complex it gets behind the scenes.

Definitely, I just wanted to throw my vote inside and place the reference to the older issue. You will know best whether its worth your time to implement or not. I just started to deal with Python code here and there a little, so cannot be of great help yet, I guess 😅.

Just one argument: Since platforms may come and go, a flexible provider backend (if not already there) would make magiclink future-proof, allowing you to easier add and remove providers when they become important enough. I can only guess that, given that, an options frontend to allow feeding custom providers won't make it that much more complicated?

Anyway, many thanks for your awesome Markdown extensions btw, they have become an essential part of our MkDocs-based documentation website. Tabbed alone is a huge enhancement 👍 🙏.

Just one argument: Since platforms may come and go, a flexible provider backend (if not already there) would make magiclink future-proof, allowing you to easier add and remove providers when they become important enough.

If it were completely abstracted, I could definitely agree. Maybe one of these days 🙂.

I thought there would be a Twitter icon shown with shortened Twitter URLs, but it's an @ only, so that is perfectly fine.

Yeah, there are no icons by default. Any you see in the documentation is due to CSS playing off the classes.

Yeah, there are no icons by default. Any you see in the documentation is due to CSS playing off the classes.

I just recognised this now on reviewing a build of our docs with magiclink enabled. Makes sense as adding either additional fonts or a set of images is probably too much for a Markdown extension 😉.

Maybe one of these days 🙂.

99% of requests on our project, even for very good and reasonable + own ones, the time is by far not enough time to keep up 😄.

I'd also love to be able to configure a hosted gitlab instance. For now, I will try to patch the PROVIDER_INFO for gitlab.

I haven't looked at the difference between links for private and public repos for the main, official service. Can you post the differences? If reasonable, I could consider supporting the official service's private repos, assuming it's fairly inline with the public convention.

I guess, I kind of assumed private and public repos had the same convention already, is this not true? The main thing that has prevented me from implementing this so far is that often the more custom setups (where a company or individual is running their own instance of GitLab, etc.) may use drastically different hosts and, at times and drastically different user pattern requirements: normal user names vs corporate emails for user names. Those usually arise when a company is running its own instance of GitLab etc.

I know my company has a custom Bitbucket service running, and it follows different conventions for user names that conflict with the current user name pattern.

We have two instances of Gitlab at our company, one private and one public (https://gitlab.cc-asp.fraunhofer.de).

I'm not sure about what the specific criteria are for the username, but we mostly use 3-4 letter alphanumeric abbreviations of our first and last time. My username on both those instances is "mkj".

I used the following snippet to patch magiclink to use our private Gitlab instance:

import pymdownx.magiclink
base_url = "https://git01.iis.fhg.de"
pymdownx.magiclink.PROVIDER_INFO["gitlab"].update({
    "url": base_url,
    "issue": "%s/{}/{}/issues/{}" % base_url,
    "pull": "%s/{}/{}/merge_requests/{}" % base_url,
    "commit": "%s/{}/{}/commit/{}" % base_url,
    "compare": "%s/{}/{}/compare/{}...{}" % base_url,
})

I have added this to my docs/macros.py, as it seems to be loaded early enough for this to work.

So, this is not private repos on Gitlab, but private and public repos on a self-hosted Gitlab, correct?

correct, its a self-hosted gitlab instance! :)

edit: ahh, now I see that the issue description mentions "private repo". From the "private url" part of the title I assumed it was about private self-hosted instances of gitlab. Should I move my comments into a new issue?

Okay, self-hosted is far more complicated, generally. I know for BitBucket, there is a number of different ways in which this could potentially not work correctly. I imagine it is the same for GitLab. At one time we also self-hosted a GitLab service at my company, but I'm not sure we still do.

What I am hesitant about is potentially adding support, formally, and it not working for various different people/organizations and having to deal with the influx of issues, potentially surfacing various scenarios where this won't work.

In your case, it seems that only the base URL needed an override. It is possible that maybe we can expose overriding the URL with a statement that if you need more than that, too bad 🤷🏻.

Or maybe we could even allow users to create a custom variant with a custom name. For instance, a user could create a variant based on GitLab and provide a different base. I'm not sure I'm keen on supporting custom issue URLs that don't follow the default.

I will at least consider this. I tried going down this road one time before, and for instance, my work's BitBucket instance was very different than the public BitBucket. The amount of complexity needed was more than I was willing to commit to., but maybe if we are only customizing the base URL, it might not be too bad.

That sounds very reasonable.

I feel like by providing a way to override the base url, the majority of the use cases might be addressed. Of course, my feeling might be wrong here, as I only have ever worked with a couple of self-hosted instances of Gitlab and BitBucket myself.

I don't think there's much of a difference between public GitLab and self-hosted instances. As far as I know there is no separate documentation for self-hosted instances, which means that API and repo endpoints are all in the same place and work the same, which seems to hold so far as I compare GitLab.com and a self-hosted GitLab instance.

I suspect you'd be able to cover 99% of self-hosted GitLab use cases by allowing users to change the base URL.

Changing the host is probably about as far as I would go. I know my work allows emails as user names, and that would definitely complicate the username requirement. The main goal was always to make this compatible with public versions, but I'm probably okay with allowing the host to be changed for self-hosted systems that don't deviate from how the public ones operate.

Is there a schedule for when a GitLab self-hosted can be used?

I don't know, I don't have an immediate plan. I don't really have a self-hosted gitlab environment to test on anymore. At most, I am considering allowing the host to be configurable, that is about as far as I would consider expanding things. I just haven't decided if that is something I want to support yet.

I can provide you with a public project on my Gitlab self-hosted server. It runs in the standard configuration, username without an @ and via https. What do you think about? In case of doubt, it is enough for me, for example, that the URL can be easily configured.

It would make it easier, so that would be great.

We have two instances of Gitlab at our company, one private and one public (https://gitlab.cc-asp.fraunhofer.de).

I'm not sure about what the specific criteria are for the username, but we mostly use 3-4 letter alphanumeric abbreviations of our first and last time. My username on both those instances is "mkj".

I used the following snippet to patch magiclink to use our private Gitlab instance:
import pymdownx.magiclink
base_url = "https://git01.iis.fhg.de"
pymdownx.magiclink.PROVIDER_INFO["gitlab"].update({
    "url": base_url,
    "issue": "%s/{}/{}/issues/{}" % base_url,
    "pull": "%s/{}/{}/merge_requests/{}" % base_url,
    "commit": "%s/{}/{}/commit/{}" % base_url,
    "compare": "%s/{}/{}/compare/{}...{}" % base_url,
})
I have added this to my docs/macros.py, as it seems to be loaded early enough for this to work.

@jannismain you mention docs/macros.py. Is this from some plugin?

Correct, it's the macro file loaded when using fralau/mkdocs_macros_plugin.

Thank you. I installed the plugin and I am trying to set it up as you did, but it doesn't see to be reading the docs/macros.py file. Did you have to more to configure it?

Thank you. I installed the plugin and I am trying to set it up as you did, but it doesn't see to be reading the docs/macros.py file. Did you have to more to configure it?

Yeah, you have to tell the plugin where to find your macros file. The documentation for mkdocs-macros should help you out there 👍

I really should get around to allowing people to alter the host .

thank you @jannismain I found why it wasn't working for me. I am doing it for Github and while the values are being changed, there are a few other places where the host name is hardcoded:

@facelessuser it would be great to be able to customize it, yes.

This will be coming in the next version. I have it far enough along now that I am confident we will have a working solution.

Changes have been committed to main. It will now be possible to create a custom repository provider based on either github, gitlab, or bitbucket. A custom provider will be specified with the name you provide, and instead of generating classes such as magiclink-github, it will generate classes of magiclink-myname so that you can style them differently than the default. The documentation will go over the details. Outside the name of your provider, what type of repository it is a variant of, a label for tooltips, and your custom host URL, nothing else will be configurable. This is about as custom as we will allow.

facelessuser / pymdown-extensions

Add magiclink support for private GitHub / GitLab url #933