Gitlab webhooks not installed

[bartlangelaan]

This is a kind of follow up on issue renovatebot/renovate#2807.

What Renovate type are you using? Renovate App for Gitlab

Describe the bug When I enable a repository trough te Renovate Dashboard, the Renovate user is correctly added to the repository. The only thing is: the webhook is not installed. Even after waiting for the project to show up on the left side, and opening the project list, it fails to add the webhook.

Screenshots This part works perfectly:

👆 This returns an empty array

Then it tries to make a call to the Renovate API to add the webhooks:

It's a project that is in a subgroup by the way, in case that matters.

If there's anything I can do to help debugging, please let me know.

[ChristianIvicevic]

@rarkins I am having the same issue right now and my project isn't even appearing in the sidebar after enabling it. The bot account was successfully added to the project, but no Webhook was installed. I can imagine the API having changed or auth being wrong.

[ChristianIvicevic]

Another small tidbit I just noticed. The dashboard is performing an API call to https://gitlab.com/api/v4/projects/12141547/members/1263493. The project ID 12141547 is my "real" project since I created a personal fork to test Renovate in the fork. I think the dashboard is trying to go upstream instead of taking the fork. The Webhook API call is indeed against the fork: https://v1.renovateapi.com/gitlab/repos/12177324/webhooks.

[rarkins]

It checks against every repo you have access to. You should see both the origin as well as fork being queried but the origin returns 404

[ChristianIvicevic]

To test possible causes, I removed the fork relation and the Webhook still isn't installed. I will be testing this with a fresh repository now. Is it possible to manually register the Webhook in the meantime until the automation of the Dashboard works again?

[ChristianIvicevic]

Alright, even with a fresh repository and no fork relation at all the webhooks endpoint on your end fails due to Gitlab most likely.

The initial OPTIONS request at https://v1.renovateapi.com/gitlab/repos/12177648/webhooks returns 204 with a cache miss from cloudfront. The second POST request to the same endpoint returns a 404 error from cloudfront.

[rarkins]

Is it a regular repo, or grouped, or sungrouped? And do you definitely have admin privileges?

[ChristianIvicevic]

The repos I tried so far were regular personal repositories. And yes, I am Maintainer for my own repos so I have admin priviledges.

[rarkins]

The post creates a verifiable webhook secret and registers it with GitLab using your admin privileges, which are only kept for the duration of the transaction. So it’s not actually possible to set up manually unless either you disclose me your admin token manually or I disclose you the service-wise secret seed term. I’ll see if the logs give enough hint as to why it’s failing, or if we can add additional debug logs

[rarkins]

Do you have the same username on GitLab as here?

[ChristianIvicevic]

On Gitlab my username is NearAutomata, you'll see it in the activity log of the Renovate Bot over there that I attempted to hook my (forked) monorepo multiple times to see whether the hook gets installed.

[ChristianIvicevic]

I just noticed that the bot has created a MR in my test repository 2 days ago without me noticing it and without and hooks. Furthermore the bot has updated the onboarding MR. I guess it would be working properly on a schedule when I finish onboarding, but it would not react to me updating the configuration as no hooks are installed.

[ChristianIvicevic]

@rarkins Any updates on this issue?

[esetnik]

I have the exact same problem as @ChristianIvicevic. No webhooks are installed.

[MadLittleMods]

Same issue when trying to use with the Gitter webapp project (it's in a subgroup), https://gitlab.com/gitlab-org/gitter/webapp

• No webhooks created, https://gitlab.com/gitlab-org/gitter/webapp/-/settings/integrations)
• The project isn't listed in the Renovate dashboard sidebar after enabling, https://renovatebot.com/dashboard

POST https://v1.renovateapi.com/gitlab/repos/3601513/webhooks -> 404

[rarkins]

Sorry for the inconvenience everyone. I will look into this again.

As a reminder, if you manually add user @renovate-bot to projects then you should see it running hourly (or at worst every 2-3 hours if there’s high load). Repos will show up in the dashboard once there exists at least one log for it, ie not immediately.

Addition of webhooks is to provide “reactiveness”, Eg merge an MR and see other conflicted ones rebased immediately, or tick a rebase checkbox and see the bot run immediately. If I’m still unable to reproduce myself and need any client-side logs from you I’ll reach out soon with details.

[ChristianIvicevic]

This issue has been stale for a while and so far I have been using a custom instance of renovate invoked on my server and I'd like to move over to your automated bot. Are there any updates in respect to this issue?

[SantiMA10]

I have the same issue if I can provide more information to help to solve it, let me know.

POST https://v1.renovateapi.com/gitlab/repos/11131635/webhooks -> 404

[monochromata]

Same issue here. It would be nice if you could reproduce and fix this.

[esetnik]

I still am having this. @rarkins is it on your radar?

[esetnik]

Is there a way to figure out what the webhook url is supposed to be so we can just manually add it to gitlab under the integrations tab?

[rarkins]

The webhook URL secret is unique per-repo so that people can't retrieve it out of gitlab.com and then go nuts spoofing webhooks at us. This means you can't add it manually.

[esetnik]

What if you expose it in the renovate dashboard so that we can grab it from there and then add it to gitlab?

[rarkins]

Yes, it's a possibility but that also involves adding a new API + UI to do so, so it's not something quickly possible. I'm going to see if I can get more debugging out of the backend first.

[rarkins]

Can you try uninstalling/reinstalling a repo now, so I can see what logs I can get dumped?

[esetnik]

@rarkins I just uninstalled/reinstalled a repo and got 404 from https://v1.renovateapi.com/gitlab/repos/14873529/webhooks. I'd be happy to jump on a slack or communicate directly if you need anything further from me to help debug this issue.

[esetnik]

❯ curl 'https://v1.renovateapi.com/gitlab/repos/14873529/webhooks' -X POST -H 'Accept: application/json, text/javascript, */*; q=0.01' -H 'Referer: https://app.renovatebot.com/dashboard' -H 'Origin: https://app.renovatebot.com' -H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36' -H 'Authorization: Bearer REDACTED' -H 'Sec-Fetch-Mode: cors' --compressed -v
*   Trying 13.249.188.23...
* TCP_NODELAY set
* Connected to v1.renovateapi.com (13.249.188.23) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=*.renovateapi.com
*  start date: Mar  8 00:00:00 2019 GMT
*  expire date: Apr  8 12:00:00 2020 GMT
*  subjectAltName: host "v1.renovateapi.com" matched cert's "*.renovateapi.com"
*  issuer: C=US; O=Amazon; OU=Server CA 1B; CN=Amazon
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fb4ea003a00)
> POST /gitlab/repos/14873529/webhooks HTTP/2
> Host: v1.renovateapi.com
> Accept-Encoding: deflate, gzip
> Accept: application/json, text/javascript, */*; q=0.01
> Referer: https://app.renovatebot.com/dashboard
> Origin: https://app.renovatebot.com
> User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.120 Safari/537.36
> Authorization: Bearer REDACTED
> Sec-Fetch-Mode: cors
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 404
< content-type: application/json; charset=utf-8
< content-length: 35
< x-powered-by: Express
< access-control-allow-origin: *
< etag: W/"23-qhMAT2LgUMCXT9whBWXDOvX2rpc"
< date: Thu, 17 Oct 2019 18:29:13 GMT
< x-cache: Error from cloudfront
< via: 1.1 dc39434a8fa09d1811be19e737658745.cloudfront.net (CloudFront)
< x-amz-cf-pop: BOS50-C2
< x-amz-cf-id: i8yU1nKmCyqUrLaR0gHtYfxpOvTXUBL-jctsqoCuWoa34rEpdpIWoQ==
<
* Connection #0 to host v1.renovateapi.com left intact
{"message":"404 Project Not Found"}* Closing connection 0

[rarkins]

Thanks, I think I can now reproduce it, although it's pretty confusing so I'm not sure when it can be fixed.

[rarkins]

I have fixed one scenario that was affecting my own account, I now see this in every project I've installed:

Please confirm if you see the same: (a) after logging out then back in, and (b) after uninstalling/reinstalling

[esetnik]

Working great for me now. Thanks @rarkins.

[monochromata]

Works for me too, yay :-)