Badge request: Gitlab CI and Gitlab

[tankerkiller125]

It would be awesome if Shields supported Gitlab and Gitlab CI. They have a Decent API and it should be 100% feasible.

[espadrine]

Thanks for the suggestion! I believe they detail their API here, although I haven't looked into what kind of badges we can provide.

[tankerkiller125]

Unfortantly based on the documentation a authentication key is required no matter what. Would it be possible to get a key that Shields can use? Say for example a Shields account on gitlab just for this purpose?

[espadrine]

We use secrets.json for authentication (for instance, we use it with GitHub). As long as it isn't a per-user authentication, it works.

[tankerkiller125]

They only seem to be user specific for private projects (not a suggested thing to badge) internal projects and public projects can be viewed by anyone so long as they have an API key.

[ishitatsuyuki]

+1: any plans on this?

[moeabdol]

:+1:

[sergeysova]

Gitlab now supports CI badge!

[sergeysova]

[sergeysova]

Maybe add Gitlab issues count?

[gluons]

@LestaD But I still prefer Shields badge style. 😅

[paulmelnikow]

PR #841 has gone stale and needs work. If anyone would like to adopt it, please do!

[sum01]

The v4 API doesn't require authentication for most things. Unsure if that's what's stalling this or not.

[LVMBDV]

I wanted to poke around the API on https://gitlab.com before I began working on this and I must say it's weird. The documentation is lacking and in some cases, misleading (wrong?).

The docs for issue and merge request state that every API call to these functions must be authenticated but I can access them without any authentication on public projects. Try it yourselves:

curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/issues'
curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/merge_requests'

This is good for making badges but still, weird documentation.

Also, pipeline status(es) can be accessed only by the project owner, as per its documentation but there is a but. Despite the documentation (again), all commits of a public project can be accessed and the pipeline status of the last commit can be retrieved through the commits API. So we can make a badge for it after all.

Moreover, implementing a user token cycling subsystem like it was done for GitHub would be a security risk as it might allow other users to access private repository data due to the lack of a fine-grained permission system for application tokens (or personal access tokens). I haven't looked much into it but are we sure this is not a problem with private repositories on GitHub as well?

TL;DR We can make it happen but I wish GitLab had better permissions.

P.S. I hope GitLab isn't authenticating me through some black magic without a token. Please try the command above to confirm my findings.

[Snuggle]

~ ➜ curl -v 'https://gitlab.com/api/v4/projects/coldnight%2fci-test/issues'
*   Trying 52.167.219.168...
* TCP_NODELAY set
* Connected to gitlab.com (52.167.219.168) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: OU=Domain Control Validated; OU=PositiveSSL Multi-Domain; CN=gitlab.com
*  start date: May 11 00:00:00 2018 GMT
*  expire date: May 11 23:59:59 2019 GMT
*  subjectAltName: host "gitlab.com" matched cert's "gitlab.com"
*  issuer: C=GB; ST=Greater Manchester; L=Salford; O=COMODO CA Limited; CN=COMODO RSA Domain Validation Secure Server CA
*  SSL certificate verify ok.
> GET /api/v4/projects/coldnight%2fci-test/issues HTTP/1.1
> Host: gitlab.com
> User-Agent: curl/7.60.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Date: Thu, 14 Jun 2018 21:00:35 GMT
< Content-Type: application/json
< Content-Length: 7699
< Cache-Control: max-age=0, private, must-revalidate
< Etag: W/"71530dc535d8812ea57aeb5f6d419c91"
< Link: <https://gitlab.com/api/v4/projects/coldnight%2Fci-test/issues?id=coldnight%2Fci-test&order_by=created_at&page=1&per_page=20&sort=desc&state=all>; rel="first", <https://gitlab.com/api/v4/projects/coldnight%2Fci-test/issues?id=coldnight%2Fci-test&order_by=created_at&page=1&per_page=20&sort=desc&state=all>; rel="last"
< Vary: Origin
< X-Content-Type-Options: nosniff
< X-Frame-Options: SAMEORIGIN
< X-Next-Page: 
< X-Page: 1
< X-Per-Page: 20
< X-Prev-Page: 
< X-Request-Id: 16c93385-31dc-4c40-a829-4e8a68685956
< X-Runtime: 0.188505
< X-Total: 9
< X-Total-Pages: 1
< Strict-Transport-Security: max-age=31536000
< RateLimit-Limit: 600
< RateLimit-Observed: 1
< RateLimit-Remaining: 599
< RateLimit-Reset: 1529010095
< RateLimit-ResetTime: Fri, 14 Jun 2018 21:01:35 GMT
< 
[{"id":11752626,"iid":12,"project_id":5168279,"title":"Epale","description":"Epale","state":"opened","created_at":"2018-06-07T01:37:54.903Z","updated_at":"2018-06-07T01:37:54.903Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2421176,"name":"Yamel Senih","username":"yamelsenih","state":"active","avatar_url":"https://secure.gravatar.com/avatar/e8270c029a61298fee5373903a889c8f?s=80\u0026d=identicon","web_url":"https://gitlab.com/yamelsenih"},"assignee":null,"user_notes_count":1,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/12","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10959931,"iid":11,"project_id":5168279,"title":"test issue","description":"il faut changer l'image d'accueil","state":"opened","created_at":"2018-05-22T12:19:21.217Z","updated_at":"2018-05-22T12:21:00.293Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2327051,"name":"Matthieu Jourdain","username":"mjourdai","state":"active","avatar_url":"https://secure.gravatar.com/avatar/4c59cb12bccffb8f6459e423f1607b4f?s=80\u0026d=identicon","web_url":"https://gitlab.com/mjourdai"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/11","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10299621,"iid":7,"project_id":5168279,"title":"SuperR(@superr)","description":"hello","state":"opened","created_at":"2018-04-13T08:08:24.121Z","updated_at":"2018-04-13T08:08:24.121Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/7","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":10298307,"iid":6,"project_id":5168279,"title":"superr","description":"hello","state":"closed","created_at":"2018-04-13T06:35:12.496Z","updated_at":"2018-04-13T06:55:47.239Z","closed_at":"2018-04-13T06:53:57.240Z","closed_by":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"labels":[],"milestone":null,"assignees":[],"author":{"id":2209888,"name":"huangzhiping","username":"yeren86125","state":"active","avatar_url":"https://assets.gitlab-static.net/uploads/-/system/user/avatar/2209888/avatar.png","web_url":"https://gitlab.com/yeren86125"},"assignee":null,"user_notes_count":2,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/6","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":9712207,"iid":5,"project_id":5168279,"title":"My first sample issue","description":"Steps to repro:\n\nThis is my first sample issue.Please ingnore ![4hit_URL]\n(/uploads/6c5962b228919a9fe8f1bbd088c41c98/4hit_URL.jpg)","state":"closed","created_at":"2018-03-14T05:28:48.887Z","updated_at":"2018-03-14T05:36:17.614Z","closed_at":"2018-03-14T05:36:17.609Z","closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":2105368,"name":"Prabhu","username":"prabhuamtex","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5f6b6125e41f4053da8839c780d2ae11?s=80\u0026d=identicon","web_url":"https://gitlab.com/prabhuamtex"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/5","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":9694027,"iid":4,"project_id":5168279,"title":"jjj","description":"![DXkgU83XkAY99NM](/uploads/5c8b13977802a87850c9d474f683200f/DXkgU83XkAY99NM.jpg)","state":"opened","created_at":"2018-03-13T08:51:58.143Z","updated_at":"2018-03-13T08:51:58.143Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":727716,"name":"ganggang","username":"ganggang","state":"active","avatar_url":"https://secure.gravatar.com/avatar/8d2e85d5fd0de2a495fa7bc197918cb6?s=80\u0026d=identicon","web_url":"https://gitlab.com/ganggang"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/4","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8859575,"iid":3,"project_id":5168279,"title":"cryptocoopx-ccx","description":"","state":"closed","created_at":"2018-01-26T02:32:22.344Z","updated_at":"2018-01-26T02:32:26.864Z","closed_at":"2018-01-26T02:32:26.860Z","closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":1963760,"name":"Naif ALhaider","username":"Naif711","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5e77d108fa5a43b854cd75703a56e98f?s=80\u0026d=identicon","web_url":"https://gitlab.com/Naif711"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/3","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8859520,"iid":2,"project_id":5168279,"title":"cryptocoopx-ccx","description":"","state":"opened","created_at":"2018-01-26T02:24:43.413Z","updated_at":"2018-02-14T19:38:43.360Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"author":{"id":1963760,"name":"Naif ALhaider","username":"Naif711","state":"active","avatar_url":"https://secure.gravatar.com/avatar/5e77d108fa5a43b854cd75703a56e98f?s=80\u0026d=identicon","web_url":"https://gitlab.com/Naif711"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/2","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null},{"id":8786090,"iid":1,"project_id":5168279,"title":"issur","description":"BUg","state":"opened","created_at":"2018-01-23T07:06:29.246Z","updated_at":"2018-04-13T18:11:38.445Z","closed_at":null,"closed_by":null,"labels":[],"milestone":null,"assignees":[],"auth* Connection #0 to host gitlab.com left intact
or":{"id":1953843,"name":"waqar ahmad","username":"waqar92","state":"active","avatar_url":"https://secure.gravatar.com/avatar/79f866cdc934803a60a2cfded672b430?s=80\u0026d=identicon","web_url":"https://gitlab.com/waqar92"},"assignee":null,"user_notes_count":0,"upvotes":0,"downvotes":0,"due_date":null,"confidential":false,"discussion_locked":null,"web_url":"https://gitlab.com/coldnight/ci-test/issues/1","time_stats":{"time_estimate":0,"total_time_spent":0,"human_time_estimate":null,"human_total_time_spent":null},"weight":null}]%     ```

[chris48s]

Thanks for doing some work on this @LVMBDV - appreciate that even answering the question "is there an API endpoint we can use for this badge?" can take some effort sometimes.

I hope GitLab isn't authenticating me through some black magic without a token. Please try the command above to confirm my findings.

I tried the commands you posted and I am getting a response back. Looking at the response headers, it contains keys:

RateLimit-Limit
RateLimit-Observed
RateLimit-Remaining
RateLimit-Reset
RateLimit-ResetTime

so it looks like unauthenticated use is rate-limited. Shields has the potential to generate quite high API traffic if we are proxying use for many users of a popular service, so rate limited access may not be suitable for this. We would probably need to authenticate to stay under rate limits/keep consistent service, even if authentication is not needed for repo permissions.

Additionally if the behaviour you intend to rely on does not match the documentation, maybe post on https://gitlab.com/gitlab-com/support-forum/issues to clarify. I think its better to ensure the behaviour isn't going to change out from under us before submitting a PR for a badge that relies on it.

[LVMBDV]

I guess we can do the GitHub token cycling thing with a GitLab application token that doesn't ask for any permissions (just authentication). I tried to create one and I was able to.

Additionally if the behaviour you intend to rely on does not match the documentation, maybe post on https://gitlab.com/gitlab-com/support-forum/issues to clarify.

I should, maybe tomorrow. Will post a link here when I do.

P.S. My authenticated queries came back with the same rate limit (600) so that's another bit of information.

[LVMBDV]

I made some progress but I would like to push them and open a PR after I clean it up a bit. I got these working without implementing token cycling yet:

• forks
• stars
• contributors

What I plan to also implement:

• last commit's pipeline status
• open & closed issues
• open & closed merge requests
• last activity date
• last commit date
• total commits
• repo size

And of course, token cycling.

[paulmelnikow]

Hey, I'm just tuning back into issues here after a little break.

Just a heads up that I rewrote and fixed the Github token cycling logic in #1205. It includes a service-agnostic TokenPool with tests. It would be interesting to see how well the abstraction carries over to another use case. And it would be great to get some more eyes on that in general. Though sadly, I don't think I'll be comfortable merging it until I get access to server logs.

[LVMBDV]

The pooling token provider seems like a better solution to the problem at hand but I want to get GitLab support deployed as soon as possible with the smallest maintenance requirements possible.

I could wait for #1205 to be merged to resume working on GitLab badges built on its token pools or finish it with its own token pool helpers and leave refactoring it after #1205 is merged to someone else.

[paulmelnikow]

Yea, I don't see merging #1205 any time soon. At the same time, I don't like adding tech debt / deferred maintenance to get in new features. How about this: I'll open a new PR with the helper classes from #1205, and get that merged, and then you merge in master, and update this to use those classes? That way we're moving the ball forward on both fronts.

[LVMBDV]

That would work for me. Thanks.

[LVMBDV]

Now that token pools are in master, I will start working on GitLab badges. While I was waiting for it, I made a little badge service of my own in Crystal.

[jhgoebbert]

Just a short comment: Using the project-name in the url fails as GitLab API requires the url to be encoded and shields.io eats that encoding:

https://img.shields.io/badge/dynamic/json.svg?label=issues&url=https://gitlab.com/api/v4/projects/coldnight%2fci-test&query=$.forks_count&colorB=brightgreen

It is required to use the project-id instead.

https://img.shields.io/badge/dynamic/json.svg?label=issues&url=https://gitlab.com/api/v4/projects/5168279&query=$.forks_count&colorB=brightgreen

[paternal]

Since http://gitlab.com is not the only instance of gitlab (see for example http://framagit.org), bonus points if URL of the gitlab instance can be passed as an argument to this badge.

Thanks for your work!

[opensorceror]

I second @paternal's comment. Since many organizations have an on-prem hosted instance of GitLab, it would be great to be able to specify a GitLab URL to an internally hosted Shields instance.

[aldomann]

It would be ideal if License badges were also supported (as they are in GitHub).

[paulmelnikow]

The very first GitLab badge went live today on https://shields.io/.

It works on gitlab.com and any public repo on public-facing GitLab instances. If you host your own GitLab inside a firewall you can run your own Shields inside the firewall, too.

This provides the same information as the default badge, with a few more formatting options.

This one was the easiest one to do first. More to come!

[gluons]

@paulmelnikow Thank you very much.

Now, I can use https://img.shields.io/gitlab/pipeline/%{project_path}/%{default_branch}.svg?style=flat-square for GitLab Badges on repo page. 😎

[noraj]

Look at what I did here: https://gitlab.com/rawsec/rawsec-cybersecurity-list/blob/master/js/sweetalert2.js

With Gitlab API, you can (while being UNAUTHENTICATED) grab forks number and stars number.

[noraj]

@paulmelnikow Nice to see GitLab badges were added. But pasting a gitlab project link like https://gitlab.com/rawsec/rawsec-cybersecurity-list on Suggests badges returns only the twitter badge. So one need to go in Build.

[paulmelnikow]

I just opened a new issue for the suggest issue.

@noraj Isn't there a rate-limit issue we'll run into on unauthenticated requests to the public API?

Note: #1838 was closed with some useful implementation bits which could be revived and ported to the new service architecture, once we sort out how to handle the authorization.

If someone wants to work on this, it would be very much appreciated! We have a pretty good toolset which is manageable even if you don't write a lot of JavaScript. The other maintainers and I are happy to lend support here or in Discord.

[paulmelnikow]

It's clear that Gitlab badges would create a lot of value for the community. I've reached out to Gitlab to ask if we can get an API key with a high limit so we can make these badges reliably.

Our workaround for GitHub is to create an OAuth app with a pool of many users' tokens. However this is a lot of work to set up and maintain! A mega-token for Shields (or a handful of static tokens) would be much simpler!

It wasn't clear what the best channel was for this within Gitlab, so I sent a tweet to start: https://twitter.com/paulmelnikow/status/1118613535079194624.

If anyone knows what the best channel is, and/or connect me with a person or department who can help, I'd appreciate it!

[YoshiRulz]

FWIW private repos on GitLab.com can still use the built-in badges, but they only render if you're logged in (first-party cookies) and have permissions.

[Bl4d3s]

One Question: I'm currently trying to get the badge working with a public repo inside a group. It states only not found. How can I use this with a group project? (Both are public, https://gitlab.com/TheEmpire/Backend/)

Edit: Fixed it. Apparently there is an additional setting under Settings > CI/CD > General Pipelines to set the Pipelines public. With that set, all works well.

[paulmelnikow]

Would you like to add that to the badge documentation? It seems like it could be useful to others! We have something similar on eg the Discord badge.

[jhgoebbert]

you can find my example project for CI/CD (incl. badges) here: https://gitlab.version.fz-juelich.de/vis/jusense-cicd

[Bl4d3s]

@paulmelnikow Will do so gladly. I looked at the discord service. Is it correct I will only have to provide a const string named documentation with the doc?

Edit: I went ahead and wrote a PR, I hope I got everthing right. See #4218

[flxholle]

Until now, I did not find any other badges than build status for GitLab. So I created a repository with examples, that use the basic API information without any access tokens: https://gitlab.com/asdoi/gitlab-badges As there is now token required maybe some of them can be integrated in shields.io

[calebcartwright]

@asdoi - Thank you for sharing. However, there's still outstanding issues around Gitlab's API rate limits (which are IP based AIUI).

The hope was that Gitlab would be able to offer Shields an increased rate limit threshold if the Shields requests were authorized with a known/agreed-to account; it wasn't really about whether the data could be retrieved anonymously (GitHub supports anonymous API calls too, but Shields can't use them due to rate limiting).

IIUC the current thresholds on gitlab.com are 10 requests/second https://docs.gitlab.com/ee/user/gitlab_com/index.html#gitlabcom-specific-rate-limits

Even during slow/non-peak windows, the Shields servers are issuing well over 1,000 requests/minute to the GitHub APIs, so Shields would almost certainly exceed the Gitlab rate limits given the popularity of the platform.

We won't be able to expand Shields' support for Gitlab badges until this rate limiting issue has been solved.

[noraj]

@calebcartwright Is there an issue open on GitLab tracker?

[calebcartwright]

I don't know of any @noraj. I know @paulmelnikow had started a dialog with the Gitlab folks but believe the fizzled out.

[ENDrain]

But does Shield have to use its own access token? Can user access token be passed to endpoint as a parameter, so that Shield makes an API request with user/group/whatnot token?

[PyvesB]

But does Shield have to use its own access token? Can user access token be passed to endpoint as a parameter, so that Shield makes an API request with user/group/whatnot token?

This would be acceptable, as long as it strictly follows the conditions outlined by our contributing guidelines:

Badges may require users to specify a token in the badge URL as long it is scoped only to fetching information and doesn't expose any sensitive information. Generating a token with the correct scope must be clearly documented.

In some cases, the scoping options of an API simply don't allow only fetching information or not exposing sensitive information, I'm unsure whether this is the case with GitLab. However, having an access token would be an overall better solution, as it makes it easier for users and is more consistent with most of our other badges, in particular the GitHub ones.

@paulmelnikow did you ever get round to opening a feature request on GitLab, as was suggested on Twitter?

[guillermin]

According to the documentation, the limits that would apply to Shields if it were to have its own access token would be:

• 2,000 requests per minute for Authenticated API traffic (for a given user)
• 2,000 requests per minute for All traffic (from a given IP address)

@calebcartwright mentions that for the GitHub APIs Shields is "issuing well over 1,000 requests/minute", but I think 2,000 requests per minute could be a good starting point for GitLab (considering it is less popular than GitHub). If it gets to the point where Shields calls are rejected because of the limit then we could try to ask GitLab for a higher limit, but from my point of view it seems like it would make sense to start with what we have and see if it is enough.

[noraj]

There is already an official gitlab badge for pipeline status so it should not be concerned by any rate limit:

[![pipeline status](https://gitlab.com/rawsec/rawsec-cybersecurity-list/badges/master/pipeline.svg)](https://gitlab.com/rawsec/rawsec-cybersecurity-list/commits/master)

This one if already included in shields.io so I guess we should say the CI part is already done

But about generic gitlab badges it's true there is nothing yet in shields.io but it should be feasible since badgen is already doing it (see https://badgen.net/gitlab):

[PyvesB]

• 2,000 requests per minute for Authenticated API traffic (for a given user)

That's significantly higher than the number I gathered from previously reading through this thread. According to the documentation, the limits also seem to have gone up by quite a bit last January, giving even more headroom.

Having a quick look at our dashboards, we peak at ~4000 requests per minute for our GitHub badges, which are by far our most popular offering. Unless I'm missing something, 2000 requests per minute with a single Gitlab token should realistically cover us for the foreseeable future.

Given these new numbers that have come to light, I would personally be happy to green light an implementation that relies on a single Gitlab user token (similar to our Discord, YouTube and several other badges). @badges/shields-maintainers any objections?

[calebcartwright]

@noraj

This one if already included in shields.io

Yes, for some time we've had badges in the handful of cases where Gitlab make it possible to get the data without hitting the rate limiting constraints

but it should be feasible since badgen is already doing it

"Service XYZ is doing it" arguments don't really guide the work we do here in Shields.io. Shields runs on a very different scale and we have very different operational concerns than some of the smaller services

@guillermin - thanks for the bump, though two important pieces of context I feel were missing from your note that I want to emphasize

According to the documentation, the limits that would apply to Shields if it were to have its own access token would be

On 2021-01-18 Gitlab apparently changed the limit from 600 requests/minute to 2,000 requests/minute which is indeed a sizeable change that could perhaps unblock us

@calebcartwright mentions that for the GitHub APIs Shields is "issuing well over 1,000 requests/minute",

The previous comment from me that you are quoting was referring to the floor of what we see for GitHub during our absolute slowest traffic hours. As P-Y noted in a subsequent comment, we often see much, much higher traffic which is why we wouldn't want to use that floor as a measuring stick for the total traffic profile of a given set of services like Gitlab. My original point was that our absolute lowest volume for GitHub traffic was nearly double the absolute limit of Gitlab, and that we'd essentially have been guaranteed to hit the Gitlab limits during our more typical traffic load periods.

@PyvesB

Given these new numbers that have come to light, I would personally be happy to green light an implementation that relies on a single Gitlab user token (similar to our Discord, YouTube and several other badges). @badges/shields-maintainers any objections?

I definitely think Gitlab's >3x rate limit increase can potentially unblock us here. I don't have any explicit objections, but am still somewhat cautious (though cautiously optimistic now). I feel like both we as maintainers and our users been bitten hard in the past with rate limit driven issues, and I'd really like to avoid getting into a proverbial case of having to try to rebuild the plane midflight.

I'm open to starting the work to move forward, but my preference would be that we start small (wire up auth and add one or two badges) and observe for a bit before opening the floodgates.

[noraj]

@calebcartwright

"Service XYZ is doing it" arguments don't really guide the work we do here in Shields.io. Shields runs on a very different scale and we have very different operational concerns than some of the smaller services

My thought was more "it's an open-source service so we may take a look at what they do, they may have found a way to bypass that limit".

I definitely think Gitlab's >3x rate limit increase can potentially unblock us here. I don't have any explicit objections, but am still somewhat cautious (though cautiously optimistic now). I feel like both we as maintainers and our users been bitten hard in the past with rate limit driven issues, and I'd really like to avoid getting into a proverbial case of having to try to rebuild the plane midflight.

Have you tried to open an issue on gitlab.com bug tracker. They set a limit to avoid abuse but they may be ok to set shields.io in a whitelist to avoid you the rate limit since you're a free libre and open source project. I'm sure they we'll be happy to have gitlab supported in a community project like shields.io.

[PyvesB]

Have you tried to open an issue on gitlab.com bug tracker

This question has been asked multiple times already, we're basically going round in circles at this point and creating noise for whoever has subscribed to this issue. Shields.io is a community project, a much more constructive stance would be to help out, open the issue yourself, and report back once you've done so. The Gitlab folks on Twitter have given a few pointers on submitting the feature request. :)

I'm open to starting the work to move forward, but my preference would be that we start small (wire up auth and add one or two badges) and observe for a bit before opening the floodgates.

Makes sense. @guillermin if I reopen #6389, would you be happy to extend it a little by adding the API token support that we have in other services (Discord or Twitch are probably a decent examples)? Depending on the popularity of this unique new badge and on Gitlab's response to the feature request that @noraj is hopefully going to open, we can then make an informed decision on whether to add more Gitlab badges or not. 😉

[noraj]

Gitlab issue created: https://gitlab.com/gitlab-org/gitlab/-/issues/331960