Support for GitLab Pages #22

Open eduardoboucas opened 7 years ago

Anyone using GitLab Pages that would be interested in using Staticman?

cc/ @andrewbanchich ?

Thanks for the mention! This looks amazing! I will definitely try it out soon, probably this weekend.

(To be clear, only GitHub is supported at the moment. The issue is to gauge the interest in adding support for GitLab and ideally find someone to help make it happen.)

@eduardoboucas Ah, okay. I would definitely be interested. GitLab is amazing and has been blowing past GitHub in most ways from what I can tell. I know several others who feel the same way. I actually just use GitLab and mirror my repos to GitHub.

I have my site's code hosted on GitLab and it'd be great to use staticman with it!

I don't have a static site yet, but I'd love to use Staticman and Gitlab with it, especially since Gitlab supports any static site generator for doing continuous deployment.

Agreed. I switched to gitlab and would love to continue this.

Thanks for the feedback, y'all.

This won't be a quick job, as a lot of the codebase was made to use the GitHub API, which will need to be reworked to support GitLab as well. I'll be on the lookout for someone that wishes to chip in, otherwise I'll (slowly) work on it whenever I get a chance.

+1 currently using gitlab pages(due to github pages block baidu spider)

using github webhooks and gitlab triggers make it work for gitlab pages see zongren/comment and my blog

For this we should add the git provider domain to the request as a path element. so it would look like /v3/entry/<github|gitlab>/<vendor>/<repo>/<branch>/<configKey>

But because one can have it's own GitLab on it's own domain, we should also handle that. And GitLab Pages are now available on GitLab Community Edition

Required Node libs for future reference: gitlab gitlab-webhook

What do you think @eduardoboucas?

The first step in making this work will be to decouple all the logic that interacts with GitHub, creating a generic storage class that can then be plugged in to GitHub or GitLab. This will be quite a substantial amount of work.

Regarding the endpoint, I think we're adding too many levels to the URL. We either rename entry to github and gitlab in v3, or we add a options[repo] field that contains the full URL to the repository, which would also address the point you made about GitLab instances running on custom domains (which also stands for GitHub Enterprise).

I definitely want to get this done, but I'd rather get some of the low-hanging fruit out of the way first (like #42) before attempting to build this.

I would implement a common git interface and then instantiate a github/gitlab connector based on the initialisation object.

But yes, there might be other features that would have bigger benefit than this. We should have a simple poll to see where people want to use their pages.

Hi,

unfortunatelly I am not able to provide any help on this, but this is the only reason why I am not using Staticman yet; all my pages are hosted on Gitlab. Despite the lack of technical knowledge to help on any PR, I would still help on testing, documenting and giving any feedback needed in order to make this happen. (:

@lsrdg as @zongren commented, it's possible to use Gitlab using Gitlab triggers and Github webhooks. See his links above for reference. In a nutshell, you would need to instruct your .gitlab-ci.yml file to clone your GitHub repo with the comments (and just the comments) inside your GitLab repository, and make the necessary changes in the templates to render them. Also, you'd need to add a GitHub webhook which triggers a GitLab build when the GitHub repository receives a push event.

Thanks for taking the time, @sukiletxe! When I first saw his comment a couple of weeks ago, I take a look at his links and enjoyed what I saw over there, but I got hooked on the webhook stuff (bad punch..), which I have no idea how to approach. Maybe I should give it another try. Thanks to your comment, it does seem to be achievable. But I don't think it is worth to bother you guys here with my lack of knowledge on the issue. (: I'll give it a try and take notes, hopefully I can come up with some sort of guidelines to help another lost soul around there. Or, if anyone stumbles on any kind of tutorial around there, please share.

Thanks again, @sukiletxe

@sukiletxe yes, but that's a workaround.

@zburgermeiszter: I know, but until Staticman supports it, it can be used successfully.

Github is great, but it is nice to have an alternative such as Gitlab, which provide different features.

There are many engines providing comments to static websites, yet I really like the idea of Staticman, and the fact that Staticman is not only an alternative, but it also provides different possibilities and a different approach to a common problem.

Under this perspective of: alternatives + different paths to be followed, I would like to see native support for Gitlab pages, and not only an workaround.

Yet, my comments here are not meant to press the development, and I do believe that those working on the development should decide how and when things should be done. Anyway, I'll keep watching this great project, and I would love contribute as possible. (:

Any update for this issue? Can we assume it is coming soon? I am desperately waiting for it.

@eduardoboucas

The issue is to gauge the interest

I think it's clear that there's a fair amount of interest in this. I recently moved my blog to GitLab Pages because GitHub doesn't support HTTPS for custom domains. I was using Staticman and would like to continue doing so.

I couldn't make it work, but if someone cares to interprete and translate what @zongren did to get Staticman to work with Gitlab, I would be glad to test it and help to document a how to (so that @eduardoboucas doesn't need to tweak the code, but users could still make it work...).

The issue isn’t about gauging interest anymore, people clearly want it. But the same people also need to understand that is is a non-profit open-source project, that depends on people (mainly myself) donating their time for free to keep it going.

I have made it clear that I don’t have the capacity to look into this at the moment, but left the issue open hoping to get someone else to contribute with their time to make it happen.

I looked at @zongren and @sukiletxe comments, as well as the links @zongren provided, but none of that made sense to me. First of all, I don't want to maintain my blog on GitHub Pages anymore (due to lack of HTTPS as previously mentioned), so I don't understand how it's supposed to receive push notifications from Staticman which apparently is one part of making it work with GitLab. Secondly, his blog appears to be in Chinese which I don't read.

For the record, I understand the lack of time for open-source contributors, so this comment is not an attempt to press for a quick solution.

FWIW Netlify supports custom HTTPS and has no restrictions on Jekyll's plugins. You just have to link your GitHub repo to Netlify to be able to use Staticman without GitHub Pages restrictions. 🎉

@asarkar

The target of comment form is https://api.staticman.net/v2/entry/zongren/comment/master/,so there will be not https issues.
I only maintain my comments on github,and whenever there is a comment pushed to github repository by staticmanapp,it triggers gitlab to run page runner

The key is gitlab clones from github comment repository

- npm install hexo-cli -g -s 
- npm install -s
- git clone "https://github.com/zongren/comment.git"
- mv comment/_data source/
- hexo clean --silent
- hexo algolia --silent true --vscode false
- hexo deploy --silent --vscode false

and uses them in theme file

for(i in site.data){
    var nameArray = i.split('/');
        var postName = nameArray[0];
            if(postName == slug){
               site.data[i]["fileName"] = nameArray[1];
                  commentArray.push(site.data[i]);
                }
         }

Any update on this?

https://donboulton.com/staticman/2018/03/05/post-Staticman-with-Gitlab.html

@ysolis 404 Page not found

http://zongren.me/2016/09/23/use-staticman-in-gitlab-pages/ (in Simplified Chinese)

thanks @VincentTam ... corrected URL now:

https://donboulton.com/2018-02-26-post-Staticman-with-Gitlab.html

Hi @eduardoboucas, thanks for your work on this project. GitLab is definitely something I want to see support for so I'd like to step up and volunteer to take point on this one :smile:

I've already given it a nosy and looks like it shouldn't be too hard once the main pieces are in place. Here's my proposal:

API changes

To allow users to choose between GitHub or GitLab, I think we need to introduce a breaking change to the /entry endpoint. Specifically, we need to allow users the ability to choose between GitHub and GitLab like so:

https://api.staticman.net/v3/entry/{SERVICE}/{USERNAME}/{REPOSITORY}/{BRANCH}/{PROPERTY (optional)}

Where SERVICE would be a new parameter that could take the following as options:

github
gitlab

This is also relevant for the /auth endpoint as well:

https://api.staticman.net/v3/auth/{SERVICE}/{USERNAME}/{REPOSITORY}/{BRANCH}/{PROPERTY (optional)}

This is not quite a breaking change as the endpoint is already in the correct structure. I think bumping up the entire API to v3 seems like the way to go anyway?

Allowing configuration of GitLab site URLs

Currently, the Staticman API uses a hard-coded GitHub URL (http://api.github.com) to make its API calls. Unfortunately this is too inflexible if users want to use Staticman in conjunction with a self-hosted version of GitHub or GitLab (most likely GitLab).

As such, the following configuration options could be introduced to the config to allow for this kind of flexibility:

  githubUrl: {
    doc: 'URL of the hosted GitHub service.',
    format: String,
    default: 'https://api.github.com',
    env: 'GITHUB_URL'
  },
  gitlabUrl: {
    doc: 'URL of the hosted GitLab service.',
    format: String,
    default: 'https://gitlab.com',
    env: 'GITLAB_URL'
  },

Obviously with the hosted Staticman instance, this would just use the default URLs.

Updating the minimum NodeJS version

I found it somewhat confusing what was the 'true' minimum version of Node that we intend to support. Currently the README suggests that 4.8.3 is the minimum version, however, it appeared like the provided Docker environment and tests actually only really support version 6+.

As 4.8.3 seems pretty barebones in terms of providing the complete ES2015+ feature set, I would suggest that whilst we are also upgrading the API version, we also take the opportunity to bump up to at least 6. This would provide features such as classes (which could be used for implementing the GitLab functionality) and would allow further quality-of-life refactoring across the rest of the codebase.

I would personally suggest we just bump it up to the latest and greatest stable release - 8.11.3. Obviously this is at your discretion :wink:

If all of the above seem amenable, then I think I could have it hacked together pretty soon (next week or so).

Hi @ntsim, thanks for that. I'm happy that someone has the change to work on adding support for GitLab, given that it's unquestionably the most popular feature request.

Your plan is sound. I'm not against introducing a breaking change to the API (that's why it is versioned, after all), but I wonder if we could avoid it by allowing the service to be part of the repository parameter, defaulting to github.

https://api.staticman.net/v3/entry/{USERNAME}/{REPOSITORY}@{SERVICE (optional)}/{BRANCH}/{PROPERTY (optional)}

So, these URLs would use GitHub:

And this one would use GitLab:

- https://api.staticman.net/v3/entry/eduardoboucas/staticman@gitlab/master/comments

I'm in two minds about it. In the one hand, avoiding a breaking change is always tempting, but on the other hand it seems a bit hacky to cram the service in the repository parameter.

What do you think? What does everyone else think?

Thanks again.

@eduardoboucas if I had to choose, I would go with the breaking change so that the URLs can be better aligned. I'm mostly thinking about the /auth endpoint which actually looks like /auth/github/{USERNAME}/{REPOSITORY}/{BRANCH} currently (and would be consistent with the breaking change).

Okay, let's go with that!

@thercast @sukiletxe @mdxprograms @lsrdg Thanks to #219, I've managed to set up a demo Hugo site (Source) powered by Staticman v3's native GitLab support.

screenshot from 2018-09-17 04-06-36

There's a setup guide in #219, but GitLab repo config and API server config are mixed up, contrary to Staticman's README. A mistake is hidden somewhere between me conversation with the developer. As a result, I've decided to write my own GitLab repo setup guide. A custom Staticman server setup guide is upcoming.

Hi! Thanks to @VincentTam's blog post I was able to setup my gitlab page with staticman and creating moderated comments works. Creating a mailing list also works, but sending notification e-mails does not. I suspect this is because the webhook part isn't yet converted to use the GitServiceFactory.

Can someone please help me out a bit understanding some specifics? Because I'm really new to Node.js.

server.js initializes the webhook handler, but still uses express-github-webhook as GithubWebHook. So the first order of business would be to replace that with something git service agnostic or switch by service used. What's the best practice here?

This then leads to controllers/handlePR.js that needs to use the GitServiceFactory instead of GitHub. So on top set const gitFactory = require('./GitServiceFactory'), then use const git = gitFactory.create(<service>, {<options>}). How do I populate the arguments service and options in this case? The currently used GitHub constructor gets this from repository: data.repository.name and the likes, but what exactly is needed for the GitServiceFactory? I don't know where the data reference is coming from and what's inside.

If we did replace all the github stuff with our gitFactory instance, git.getReview kicks off staticman.processMerge in lib/Staticman.js , which in turn handles the subscriptions.

This seems fairly simple, but unfortunately I don't know the language yet.

Any news?

@OlafHaag Maybe @ntsim can help, since he has authored #219 and #231.

@eduardoboucas,

would it be possible to have simple documentation (similar to the docs for GitHub) for GitLab?

Many thanks for your help.

Hi! Thanks to @VincentTam's blog post I was able to setup my gitlab page with staticman and creating moderated comments works. Creating a mailing list also works, but sending notification e-mails does not. I suspect this is because the webhook part isn't yet converted to use the GitServiceFactory.

@OlafHaag I've submitted a PR that addresses this - https://github.com/eduardoboucas/staticman/pull/388

@hispanic Thanks for contributing! In the meantime the e-mail list provider (mailgun) that I was using changed their pricing/free plan, and it wasn't clear how staticman would handle the GDPR requirements for e-mail notifications (need consent/confirmation to be added to the list), so I disabled e-mails completely for now. But it's nice to see there's some progress.

@OlafHaag I know that Mailgun dropped their free plan, but the terms of their flex plan are fine for me:

If you send fewer than 1,250 email messages in a given month while you are on the Flex plan, you will receive an invoice for $0, and there will be no balance rolled over to the next month. If you send more than 1,250 emails in a given month you will be charged $1.00 at 1,251, and $0.80 per thousand for anything beyond 1,251.

I'm not sure I understand your GDPR-related concerns. On my blog, I intend to provide a checkbox that the potential subscriber needs to affirmatively check in order to signal their permission for me to send them email. The only thing I can imagine Staticman needs to provide is support for capturing some audit info (possibly in Mailgun):

What they were told at the time of consent
How they consented (e.g., during checkout, via Facebook form, etc.)
Whether they have withdrawn consent ~ https://www.litmus.com/blog/5-things-you-must-know-about-email-consent-under-gdpr/

@hispanic Thanks for the info! I might reconsider.

On my blog, I intend to provide a checkbox that the potential subscriber needs to affirmatively check in order to signal their permission for me to send them email.

I could easily add anyone's e-mail address and check the box. I outlined my GDPR related thoughts in #277. Maybe we should take the discussion there.

@OlafHaag Will do. Thanks for the pointer.

eduardoboucas / staticman