How to receive json payload from GitLab webhook using plugin? #185

Closed JaniszewskiMarcin closed 3 years ago

In the section when we can test webhooks, we can also see received json payloads that are POST to TeamCity for example from GitLab, my question is can we somehow get to this payload to read it in build steps or there is more easiest approach to get the whole informations about merge request becouese I can not found any help in documentation or another plugin that will make it possible. Thank you for your help and time in advance.

Hi @JaniszewskiMarcin

Sorry for the late reply. If I understand correctly, you want an endpoint for receiving webhooks that you can interact with programmatically.

Currently, the testing endpoint does not support this. I would think the best way would be to query gitlab for the PR details as part of your build.

I think there is a plug-in in TeamCity that simplifies making REST calls to other services. I'll have a look and see what I can find.

Otherwise adding the testing endpoint to the rest API in tcWebhooks should be relatively easy, but you'd still need to query it somehow from your build.

I did write a separate springboot app that does exactly what I think you want. It listens as an endpoint for webhooks and has a rest API to query for stored payloads. I use it internally for testing that my webhooks are being delivered correctly.

This plug-in supports making REST requests as a step in your build. You can then use groovy to extract values and make them available as variables to your build.

https://plugins.jetbrains.com/plugin/9310-rest-runner

Hi @netwolfuk,

I am really glad that you respond to my issue!

The main problem is that we want to trigger builds by GitLab for example if PR has received comment "/deploy" we want to trigger deployment job on TeamCity with parameters special to our PR. And now we can't use GET in build step because it requires merge number. And TeamCity receive PR parameters only when webhook is activated by Push events. We want to be able to trigger jobs on TeamCity through GitLab comment or button on PR view and still be able to get the merge number parameter, and later on in build steps make a call through API to receive more parameters or approvers list that we want ^^ Currently we shooting a webhook by comment on PR to add2Queue specified job, it is starting but we don't know how to get to the json payload that is coming with it, it would be perfect to somehow extract this parameters from json payload to TeamCity parameters and use it in build steps.

Thank you for your time and all explanations ^^

Are you able to share the payload for a gitlab webhook?n it might be easier to write a separate plugin for TeamCity to trigger a build when that event happens.

I looked again at the trigger plugins. That looks like the best solution for you. Did you get an example of the GitLab webhook payload.

{
  "object_kind": "note",
  "event_type": "note",
  "user": {
    "id": 232,
    "name": "",
    "username": "",
    "avatar_url": "",
    "email": ""
  },
  "project_id": 12,
  "project": {
    "id": 12,
    "name": "Project",
    "description": "",
    "web_url": "",
    "avatar_url": null,
    "git_ssh_url": "",
    "git_http_url": "",
    "namespace": "DevOps",
    "visibility_level": 0,
    "path_with_namespace": "",
    "default_branch": "master",
    "ci_config_path": "",
    "homepage": "",
    "url": "",
    "ssh_url": "",
    "http_url": ""
  },
  "object_attributes": {
    "attachment": null,
    "author_id": 23,
    "change_position": null,
    "commit_id": null,
    "created_at": "2021-09-16 08:38:01 UTC",
    "discussion_id": "sfwr2232wq",
    "id": 222,
    "line_code": null,
    "note": "Hello",
    "noteable_id": 234,
    "noteable_type": "MergeRequest",
    "original_position": null,
    "position": null,
    "project_id": 234,
    "resolved_at": null,
    "resolved_by_id": null,
    "resolved_by_push": null,
    "st_diff": null,
    "system": false,
    "type": null,
    "updated_at": "2021-09-16 08:38:01 UTC",
    "updated_by_id": null,
    "description": "Hello",
    "url": ""
  },
  "repository": {
    "name": "Name",
    "url": "",
    "description": "",
    "homepage": ""
  },
  "merge_request": {
    "assignee_id": 333,
    "author_id": 333,
    "created_at": "2021-07-30 11:36:26 UTC",
    "description": "",
    "head_pipeline_id": 333,
    "id": 33,
    "iid": 22,
    "last_edited_at": null,
    "last_edited_by_id": null,
    "merge_commit_sha": null,
    "merge_error": null,
    "merge_params": {},
    "merge_status": "cannot_be_merged",
    "merge_user_id": null,
    "merge_when_pipeline_succeeds": false,
    "milestone_id": null,
    "source_branch": "master",
    "source_project_id": 33,
    "state_id": 1,
    "target_branch": "develop",
    "target_project_id": 3333,
    "time_estimate": 0,
    "title": "Master",
    "updated_at": "2021-09-16 08:38:01 UTC",
    "updated_by_id": null,
    "url": "",
    "source": {
      "id": 333,
      "name": "Name",
      "description": "",
      "web_url": "",
      "avatar_url": null,
      "git_ssh_url": "",
      "git_http_url": "",
      "namespace": "DevOps",
      "visibility_level": 0,
      "path_with_namespace": "",
      "default_branch": "master",
      "ci_config_path": "",
      "homepage": "",
      "url": "",
      "ssh_url": "",
      "http_url": ""
    },
    "target": {
      "id": 33,
      "name": "Name",
      "description": "",
      "web_url": "",
      "avatar_url": null,
      "git_ssh_url": "",
      "git_http_url": ".git",
      "namespace": "DevOps",
      "visibility_level": 0,
      "path_with_namespace": "",
      "default_branch": "master",
      "ci_config_path": "",
      "homepage": "",
      "url": "",
      "ssh_url": "",
      "http_url": ".git"
    },
    "last_commit": {
      "id": "2222",
      "message": "Update .gitlab-ci.yml file",
      "title": "Update .gitlab-ci.yml file",
      "timestamp": "2021-08-20T15:20:35+00:00",
      "url": "/-/commit/232",
      "author": {
        "name": "Some Name",
        "email": "sfes@gmail.com"
      }
    },
    "work_in_progress": false,
    "total_time_spent": 0,
    "time_change": 0,
    "human_total_time_spent": null,
    "human_time_change": null,
    "human_time_estimate": null,
    "assignee_ids": [
      2344
    ],
    "state": "opened"
  }
}

Here it's sample payload from GitLab repository we want to read. Is it easy to make plugin that will get this message read this JSON and assign values to parameters in the job that is triggered, just like in Jenkins

Yes, I think it's relatively easy.

I started looking at it the other day. I was thinking it would be good to use json-path to define a location in the JSON structure and assign that to a variable.

Eg my_branch=$.project.default_branch

This would define a build parameter called my_branch with the value obtained from the JSON payload.

Yes, json-path would be the best option but is there any simple way that we can for example store all the payload that is sent from GitLab with URL add2Queue in some kind of variable in job that could be even as a String, after that we could extract everything by code. Becouse the worst thing is that we don't know how to receive that payload it's only triggering the job but there is no endpoint for that json. Is there any way to add to your plugin something like a variable that will store all the payload?

Good idea. Something like payload which contains the whole payload in case you need it. I'll see what TeamCity will support there. Failing that, maybe a URL that will be set eg, payload-url from which you can download the payload that triggered the build as part of the build.

In Gitlab, how much control do you have over the URL that the Webhook is sent to? Can you use variables in the URL? eg, branch_name, or project_name?

I wonder if the listener should listen on specific or generic URLs.

eg,

/app/webhook-trigger and the plugin has to try to determine which build to trigger
or /app/webhook-trigger/{teamcity_project_id}/{teamcity_build_id} and the plugin can easily check if there is a trigger configured for that build.
or app/webhook-trigger/{uuid} and then the plugin maintains some sort of mapping from uuid to build_id.

You can search it in the REST API documentation. We used something like that to trigger the job by adding it to queue https://teamcityserver.pl/action.html?add2Queue=TestParametersJob and if we could even set one parameter from URL like https://teamcityserver.pl/action.html?add2Queue=TestParameters&env.MERGE_NUMBER=$.merge_request.iid that would be perfect because we could after that use cURL in the job step and get information about this merge request that URL requires the merge number id. So the generic URL with job id to trigger would be enough If we could receive the whole payload in some variable if not sending even one parameter through URL will also work for us ^^

Today I think I tried everything to set the parameter through URL Webhook in GitLab and that's impossible. I don't even know where is the endpoint in TeamCity to even see this payload... It was only possible on your endpoint for testing on tcWebhooks. All we need is to pass only one parameter from GitLab and I am running out of ideas :/

I made good progress this evening. Hopefully I'll have a plug-in for you to test within a week.

Hi @JaniszewskiMarcin . I can't see a commit ID in the payload. The build will just run against the newest code in the source_branch. Is that ok?

I looked up the Jenkins plugin and have identified the following requirements...

Parameters resolved by the trigger plugin override any values defined on the build.
A parameter should be able to be marked as required or optional.
If optional, a default value could be defined so that that value is used for the build if the value is not found in the payload.
If optional and no default is defined, the parameter will be ignored and not passed to the build.
Parameter values must be able to be validated against a RegEx. If the parameter values do not match their associated regexs, the build is not invoked.
It makes no sense to have a parameter marked as optional if it has a regex.
If all parameters are optional, and are not resolved, then the build will run without any customisation
The branch should be resolved if possible. If resolved, the build will run against that build.
If the commit id (hash for git, number for SVN, etc) can be determined, we must run the build at this point in the branch history.

Some questions.

What happens if we cannot determine the branch? Do we invoke the build against the default branch?
Can a parameter be an empty string? Is that valid? It could even be validated by the regex as empty. I am assuming that the path being not found is "unset", but a parameter that exists in the document and can be found by the json-path or x-path but resolves to null or an empty string is a valid parameter, at which point it would need to be validated by the regex for it to prevent the build from running.

Hi @netwolfuk! The latest commit hash you can find after $.merge_request.last_commit.id ^^I guess you looked on Jenkins notifier plugin which is working really well, yes we can filter parameters by regEx, set parameters to anything in a string by URL "buildWithParameters", If parameters will be null or empty there is no problem they can be for example set in TeamCity to the default value and if you try to pass null value or an empty string to them we will simply check and validate for that in build scripts straight from TeamCity. Also, a great way to check the payload is to set test webhook in the test repository at GitLab, then pass the URL to the testing endpoint, and after you click on Test "Note event" (payload when triggered by a comment) all the details will appear under "Recent Deliveries" and when you click "Vie details" you will have all information you want ^^

If the branch cannot be determined we just throw an error and do not invoke any build, but it can be optional functionality that may be useful in some cases :)

Yep. This all makes sense to me.

Just one point to make you aware of. If a value is not set in this plugin, but is defined by the normal build properties, then the normal property will be used by the build.

However, it won't be possible to construct a new property from the one from the standard build.

An example.

foo=$.foo <- resolves bar=123 <- defined in the build fred=${foo} is a friend of ${bar}

fred will be undefined because bar is not available in the context that the plugin runs in.

So in summary, If we will have parameters defined in the build the parameters from plugin will be ignored?

The parameters from the plugin will only be ignored if they are not able to be resolved AND they are not considered "required".

If required but are empty, the build won't trigger. If not required but are empty, they are not set.

I hope to have something for you to test tomorrow

That's fine, thank you for your help and work! 😃 I will be glad to test this thing out ^^

https://github.com/tcplugins/tcWebHookTrigger

It's looking awesome, really great job so far 😄 What I understood the URL endpoint when we have to POST webhook to trigger the current job is still in work, right?

Oops. Sorry. /app/webhook-trigger/buildConfigId

Build config id is defined in your build configuration. Typically something like ProjectName_BuildName

Hi @JaniszewskiMarcin A new release with fixes for two more issues is now available. See https://github.com/tcplugins/tcWebHookTrigger/releases I'll close this ticket now. I would suggest staring the tcWebHookTrigger project on Github, so that you get updates. Any further discussion can happen on that issue board.

tcplugins / tcWebHooks

How to receive json payload from GitLab webhook using plugin? #185