Azure Devops Webhook Test gets 400 Bad Request from Atlantis

[Richard-Payne]

I've setup an Atlantis server, got it configured with Github and tested. Now I'm trying to switch it to our Azure Devops server.

When configuring the Service Hook, using the Test function produces a 400 Bad Request error from Atlantis. The test event being sent is:

Method: POST
URI: https://myserver.com:4141/events
HTTP Version: 1.1
Headers:
{
  Authorization: Basic ********
  Content-Type: application/json; charset=utf-8
}
Content:
{
  "subscriptionId": "5937ed81-5e86-4d5b-9169-23475f1953b4",
  "notificationId": 21,
  "id": "2ab4e3d3-b7a6-425e-92b1-5a9982c1269e",
  "eventType": "git.pullrequest.created",
  "publisherId": "tfs",
  "message": {
    "text": "Jamal Hartnett created a new pull request",
    "html": "Jamal Hartnett created a new pull request",
    "markdown": "Jamal Hartnett created a new pull request"
  },
  "detailedMessage": {
    "text": "Jamal Hartnett created a new pull request\r\n\r\n- Merge status: Succeeded\r\n- Merge commit: eef717(https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/eef717f69257a6333f221566c1c987dc94cc0d72)\r\n",
    "html": "Jamal Hartnett created a new pull request\r\n<ul>\r\n<li>Merge status: Succeeded</li>\r\n<li>Merge commit: <a href=\"https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/eef717f69257a6333f221566c1c987dc94cc0d72\">eef717</a></li>\r\n</ul>",
    "markdown": "Jamal Hartnett created a new pull request\r\n\r\n+ Merge status: Succeeded\r\n+ Merge commit: [eef717](https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/eef717f69257a6333f221566c1c987dc94cc0d72)\r\n"
  },
  "resource": {
    "repository": {
      "id": "4bc14d40-c903-45e2-872e-0462c7748079",
      "name": "Fabrikam",
      "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079",
      "project": {
        "id": "6ce954b1-ce1f-45d1-b94d-e6bf2464ba2c",
        "name": "Fabrikam",
        "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/projects/6ce954b1-ce1f-45d1-b94d-e6bf2464ba2c",
        "state": "wellFormed",
        "visibility": "unchanged"
      },
      "defaultBranch": "refs/heads/master",
      "remoteUrl": "https://fabrikam.visualstudio.com/DefaultCollection/_git/Fabrikam"
    },
    "pullRequestId": 1,
    "status": "active",
    "createdBy": {
      "displayName": "Jamal Hartnett",
      "url": "https://fabrikam.vssps.visualstudio.com/_apis/Identities/54d125f7-69f7-4191-904f-c5b96b6261c8",
      "id": "54d125f7-69f7-4191-904f-c5b96b6261c8",
      "uniqueName": "fabrikamfiber4@hotmail.com",
      "imageUrl": "https://fabrikam.visualstudio.com/DefaultCollection/_api/_common/identityImage?id=54d125f7-69f7-4191-904f-c5b96b6261c8"
    },
    "creationDate": "2014-06-17T16:55:46.589889Z",
    "title": "my first pull request",
    "description": " - test2\r\n",
    "sourceRefName": "refs/heads/mytopic",
    "targetRefName": "refs/heads/master",
    "mergeStatus": "succeeded",
    "mergeId": "a10bb228-6ba6-4362-abd7-49ea21333dbd",
    "lastMergeSourceCommit": {
      "commitId": "53d54ac915144006c2c9e90d2c7d3880920db49c",
      "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/53d54ac915144006c2c9e90d2c7d3880920db49c"
    },
    "lastMergeTargetCommit": {
      "commitId": "a511f535b1ea495ee0c903badb68fbc83772c882",
      "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/a511f535b1ea495ee0c903badb68fbc83772c882"
    },
    "lastMergeCommit": {
      "commitId": "eef717f69257a6333f221566c1c987dc94cc0d72",
      "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/eef717f69257a6333f221566c1c987dc94cc0d72"
    },
    "reviewers": [
      {
        "reviewerUrl": null,
        "vote": 0,
        "displayName": "[Mobile]\\Mobile Team",
        "url": "https://fabrikam.vssps.visualstudio.com/_apis/Identities/2ea2d095-48f9-4cd6-9966-62f6f574096c",
        "id": "2ea2d095-48f9-4cd6-9966-62f6f574096c",
        "uniqueName": "vstfs:///Classification/TeamProject/f0811a3b-8c8a-4e43-a3bf-9a049b4835bd\\Mobile Team",
        "imageUrl": "https://fabrikam.visualstudio.com/DefaultCollection/_api/_common/identityImage?id=2ea2d095-48f9-4cd6-9966-62f6f574096c",
        "isContainer": true
      }
    ],
    "commits": [
      {
        "commitId": "53d54ac915144006c2c9e90d2c7d3880920db49c",
        "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/commits/53d54ac915144006c2c9e90d2c7d3880920db49c"
      }
    ],
    "url": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/pullRequests/1",
    "_links": {
      "web": {
        "href": "https://fabrikam.visualstudio.com/DefaultCollection/_git/Fabrikam/pullrequest/1#view=discussion"
      },
      "statuses": {
        "href": "https://fabrikam.visualstudio.com/DefaultCollection/_apis/git/repositories/4bc14d40-c903-45e2-872e-0462c7748079/pullRequests/1/statuses"
      }
    }
  },
  "resourceVersion": "1.0-preview.1",
  "resourceContainers": {
    "collection": {
      "id": "c12d0eb8-e382-443b-9f9c-c52cba5014c2"
    },
    "account": {
      "id": "f844ec47-a9db-4511-8281-8b63f4eaf94e"
    },
    "project": {
      "id": "be9b3917-87e6-42a4-a549-2bc06a7a878f"
    }
  },
  "createdDate": "2021-01-07T12:54:06.7915094Z"
}

on the Atlantis docker logs:

2021/01/07 12:47:19+0000 [INFO] server.go:528 server: Atlantis started - listening on port 4141
2021/01/07 12:54:06+0000 [DBUG] middleware.go:35 server: POST /events – from <ip redacted>:53637
2021/01/07 12:54:06+0000 [DBUG] events_controller.go:137 server: Ignoring request
2021/01/07 12:54:06+0000 [DBUG] middleware.go:37 server: POST /events – respond HTTP 400

The Devops version is 16.131.28226.3 if that matters.

[Richard-Payne]

Oh, on a hunch, and because it was overdue anyway, we upgrade Devops to the latest 2020 version.

This improved things a little. Now, the Test button (during webhook setup) works. Atlantis ignores comments (due to not being a command comment) and errors during the PR create because the repo the test function uses doesn't exist. However, it is at least parsing and handling the incoming payloads.

However, when doing operations on an actual repo, the Atlantis server ignores the request again, returning 400.

My first thought was to setup wireshark and check the payloads to see what is different. However, Atlantis requires TLS when using Devops and I have no way to configure TLS decryption from the MS libraries.

Is there any way to make Atlantis output the json payload of the requests it receives? If not, how do I go about debugging this?

[Richard-Payne]

So I've managed to work around this by using an haproxy in front of Atlantis and inserting a Request-Id header with a uuid value.

Atlantis is now responding to the messages, and this has revealed a possible explanation for the discrepancy. It appears that Atlantis has been hard coded to expect a cloud hosted Azure Devops platform. We're running on-prem. Not sure the cloud version is producing additional headers.

[gunnertwin]

Running into the exact same issue from our self hosted devops instance

[gunnertwin]

Request id insertion worked for me too, thans @Richard-Payne

server {
        listen 80;
        listen [::]:80;
        server_name localhost;
        add_header Request-ID $request_id;
        location / {
                proxy_pass http://127.0.0.1:4141;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header Host $host;
                proxy_set_header Request-ID $request_id;
        }
}