search

jenkins-x / jx

Jenkins X provides automated CI+CD for Kubernetes with Preview Environments on Pull Requests using Cloud Native pipelines from Tekton
https://jenkins-x.io/
Apache License 2.0
4.57k stars 787 forks source link

webhook on Github always fails We couldn’t deliver this payload: Failure when receiving data from the peer #5633

Closed dove-young closed 3 years ago

dove-young commented 5 years ago

Summary

webhook on Github always fails. Error msg said We couldn’t deliver this payload: Failure when receiving data from the peer

Steps to reproduce the behavior

Expected behavior

webhook should works well

Actual behavior

webhook on Github always fails. Error msg said We couldn’t deliver this payload: Failure when receiving data from the peer

Jx version

The output of jx version is:

NAME               VERSION
jx                 2.0.758
jenkins x platform 2.0.1247
Kubernetes cluster v1.13.5+icp-ee
kubectl            v1.13.5+icp-ee
helm client        Client: v2.12.3+geecf22f
git                2.23.0
Operating System   Red Hat Enterprise Linux Server release 7.6 (Maipo)

verifying packages

Jenkins type

Kubernetes cluster

IBM Cloud Private 3.2.0

Operating system / Environment

RHEL 7.6

kubectl get svc,ep
NAME                                  TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)    AGE
service/deck                          ClusterIP   10.0.201.214   <none>        80/TCP     44m
service/heapster                      ClusterIP   10.0.5.104     <none>        8082/TCP   40m
service/hook                          ClusterIP   10.0.223.151   <none>        80/TCP     44m
service/jenkins-x-chartmuseum         ClusterIP   10.0.220.196   <none>        8080/TCP   40m
service/nexus                         ClusterIP   10.0.128.133   <none>        80/TCP     40m
service/pipelinerunner                ClusterIP   10.0.145.181   <none>        80/TCP     44m
service/tekton-pipelines-controller   ClusterIP   10.0.212.128   <none>        9090/TCP   44m
service/tekton-pipelines-webhook      ClusterIP   10.0.134.175   <none>        443/TCP    44m
service/tide                          ClusterIP   10.0.1.131     <none>        80/TCP     44m

NAME                                    ENDPOINTS                            AGE
endpoints/deck                          10.1.127.199:8080,10.1.212.72:8080   44m
endpoints/heapster                      10.1.94.195:8082                     40m
endpoints/hook                          10.1.212.73:8888,10.1.229.8:8888     44m
endpoints/jenkins-x-chartmuseum         10.1.77.139:8080                     40m
endpoints/nexus                         10.1.48.139:8081                     40m
endpoints/pipelinerunner                10.1.94.196:8080                     44m
endpoints/tekton-pipelines-controller   10.1.77.137:9090                     44m
endpoints/tekton-pipelines-webhook      10.1.229.9:8443                      44m
endpoints/tide                          10.1.127.202:8888                    44m

kubectl get ing
NAME          HOSTS                               ADDRESS       PORTS   AGE
chartmuseum   chartmuseum.jx.<my-ip-addr>.nip.io   <my-ip-addr>   80      40m
deck          deck.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      40m
hook          hook.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      53s
nexus         nexus.jx.<my-ip-addr>.nip.io         <my-ip-addr>   80      40m
tide          tide.jx.<my-ip-addr>.nip.io          <my-ip-addr>   80      109s

And also there is no place to debugging.

print logs from pod hook-5b79bdd4cb-658t8 and tekton-pipelines-webhook-76c6bd5b6-29cpl do not help debugging.

arghyadeep-k commented 5 years ago

I'm facing the same issue too.

vamsilella commented 5 years ago

i'm facing the same issue with freestyle project.

vamsilella commented 5 years ago

i have changed the IP of jenkins to public ip in webhook url and it worked.

arghyadeep-k commented 5 years ago

i have changed the IP of jenkins to public ip in webhook url and it worked.

That works. Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

pmuir commented 5 years ago

Looks like we need a FAQ for this one.

dsanders23 commented 4 years ago

to public ip in webhook url and it worked.

Looks a nice solution, thanks! Could you share a little bit more about how can we do the same? Thanks!

arghyadeep-k commented 4 years ago

to public ip in webhook url and it worked.

Looks a nice solution, thanks! Could you share a little bit more about how can we do the same? Thanks!

Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

srafiqa commented 4 years ago

Hi , I would really appreciate if anyone could share me the syntax of the payload with public IP address. I am also getting the error message "We couldn’t deliver this payload: Failure when receiving data from the peer" , even after provided public ip of jenkins server.

arghyadeep-k commented 4 years ago

Hi , I would really appreciate if anyone could share me the syntax of the payload with public IP address. I am also getting the error message "We couldn’t deliver this payload: Failure when receiving data from the peer" , even after provided public ip of jenkins server.

Use the address in the following format: http://104.211.218.105:8080/github-webhook/

Replace the ip with the public ip of your jenkins server and the port on which your jenkins is running which is by default 8080.

srafiqa commented 4 years ago

Thank you for your response , i updated in the same way , however still getting the same issue. Not sure it is because the github enabled with SSO .

binbrayer commented 4 years ago

Try to use application/x-www-form-urlencoded not application/json content type.

preconceptstech84 commented 4 years ago

I'm seeing the same issue for our GHE. We too have SSO enabled but we are using a token so that shouldn't be an issue. Changing to the ip of the loadbalancer in front of our jenkins doesn't help

adeoke commented 4 years ago

I'm facing the same issue by the looks of it. Has anyone managed to find the solution?

narayanan-n commented 4 years ago

any updates on this?

mmreddy1582 commented 4 years ago

Having same issue ,Please kindly post/share how to fix this issue .

himanshush13 commented 4 years ago

Any luck about this issue. Even though i have used the public ip in the webhook url, but no changes

himanshush13 commented 4 years ago

@vamsilella , if you can check

vamsilella commented 4 years ago

@himanshush13 can you provide more details like webhook url / payload is SSO enabled? is your jenkins hosted on any container? or is it on ec2? This would help debugging the issue.

mmreddy1582 commented 4 years ago

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server" Github won't be able to communicate with Jenkins which is running local /private IP address . You need to specify an IP address GitHub can contact over the internet not the actual IP address . Steps: 1.Download the https://ngrok.com/download server where it is running with Jenkins . 2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins . 3.http://<......>ngrok.io eg http://3b2db437.ngrok.io 4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ .. ====>Don't give port 8080 as we already passed /ngrok http 8080 5.Final test the github webhook status .

mmreddy1582 commented 4 years ago

step 4 to update the github webhook URL .

himanshush13 commented 4 years ago

Jenkins is not on any container, it is working on a linux vm independently. I tried giving the github credentials to jenkins configuration.

this is the url, I am using http://frparvm-blkchn:8082/github-webhook/ I also used this url http://193.57.249.2:8082/github-webhook/

I followed this blog-- https://oncomputingwell.princeton.edu/2018/01/triggering-a-jenkins-build-every-time-changes-are-pushed-to-a-git-branch-on-github/

is it an issue because, we are in our company network ?

vamsilella commented 4 years ago

you can use the above ip http://193.57.249.2:8082/ to access jenkins dashboard within your company using VPN. But github needs an IP ( public ) which can be accessible through internet so your payload should be

http://publicip:8082/github-webhook/

and using http://frparvm-blkchn:8082 as it's not a public domain its just the hostname so github doesn't know what it is... as github is not hosted in your company network ;) just kidding.

shyamshinde008 commented 4 years ago

@vamsilella I am also facing same issue and i am also in company network. But i am not sure how can i get public IP for a machine in private network. I tried ngrok but that is not working. i found smee and smee-client topic but i am not sure how to configure smee-client on my windows machine. will you please help me in triggering jenkins using webhook. Thanks in advance!!!

vamsilella commented 4 years ago

@shyamshinde008 its the IT job to assign a static IP to the server where your Jenkins master is running and hostname should be set in that server (get a sub domain from IT) and they have to whitelist either bitbucket/github for ingress. thats it.

JohnYu500 commented 4 years ago

I have the same issue. In my case, my Jenkins is running on an EC2 in AWS and it does have a public IP address. I have opened up port 8080 for HTTP traffic from github.com. When I pasted the payload URL directly to a browser I got this java error: java.lang.Exception: Method POST required. This at least indicates the URL hit the Jenkins server.

So I really have no idea why the payload could not be delivered.

Any help is appreciated.

Bheemalingareddyh commented 4 years ago

Install GitHub Integration plugin and check.

Pratishvg commented 4 years ago

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server" Github won't be able to communicate with Jenkins which is running local /private IP address . You need to specify an IP address GitHub can contact over the internet not the actual IP address . Steps: 1.Download the https://ngrok.com/download server where it is running with Jenkins . 2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins . 3.http://<......>ngrok.io eg http://3b2db437.ngrok.io 4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ .. ====>Don't give port 8080 as we already passed /ngrok http 8080 5.Final test the github webhook status .

This actually worked for me. I am working locally on my machine. Don't know how big companies work this issue out. Thanks a lot. Just that the I downloaded the ngrok and unzipped the file. Opened the command window from the folder the ngrok.exe was stored and executed command 'ngrok http 8080' as stated earlier and had to wait. I got 2 addresses(names) - one with 'http' and the other with 'https'. I used the 'https://......ngrok.io/github-webhook/' address for webhook in github and it worked.

JohnYu500 commented 4 years ago

Actually I was able to make it work. The issue I had before was because I was using the wrong IP address. Later I found out that github webhook has its own IP addresses. I was using the IP of the github.com. Once I added the three IP addresses of Github webook to my AWS security group, the payload works like a charm. Thanks.

arghyadeep-k commented 4 years ago

I don't think this is an issue anymore. Can we close this?

shadoath commented 4 years ago

Actually I was able to make it work. The issue I had before was because I was using the wrong IP address. Later I found out that github webhook has its own IP addresses. I was using the IP of the github.com. Once I added the three IP addresses of Github webook to my AWS security group, the payload works like a charm. Thanks.

Those can be found https://api.github.com/meta

"hooks": [
"192.30.252.0/22",
"185.199.108.0/22",
"140.82.112.0/20"
]
JohnYu500 commented 4 years ago

Yes. That is right.

sittuanand007 commented 4 years ago

Ngrok will be temporary solution right because after the session ends, we will further have to get it configured from the beginning. I am working on my company's VM which is in private network. My doubt is that I have configured Jenkins on localhost in the vm that is in private network. As we know that localhost generally runs on http. So if I want my Jenkins to have https, so what will be the approach for that ?? When I will get a static IP assigned to the VM and when the IT dept will give me a sub-domain, then will it be http or https ?

shyamshinde008 commented 4 years ago

@sittuanand007 Ngrok is just for POC not permanent solution. you have to make your Jenkins server public if you are in private network (company network) so that git is able to connect and deliver webhook payload to your system. Check with your network team they might help you in this.

darenjacobstellic commented 4 years ago

i have changed the IP of jenkins to public ip in webhook url and it worked.

That works. Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

Could you please state this more clearly? Is this operation happening in Jenkins, is it happening in Github? Please specify the exact steps

dineshdevopsshare commented 4 years ago

Try to use application/x-www-form-urlencoded not application/json content type.

Thanks mate! this saved my lot of time!!!

prashant-bande commented 4 years ago

I have updated the IP with the current public IP of the server. It works like a charm.

Cloud server often gets the new public IP and it happens that webhook has an old IP and updating it to current public IP will solve the issue,

jenkins-x-bot commented 3 years ago

Issues go stale after 90d of inactivity. Mark the issue as fresh with /remove-lifecycle stale. Stale issues rot after an additional 30d of inactivity and eventually close. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle stale

mattmattox commented 3 years ago

I ran into this issue when using CloudFlare in Proxy mode.

jenkins-x-bot commented 3 years ago

Stale issues rot after 30d of inactivity. Mark the issue as fresh with /remove-lifecycle rotten. Rotten issues close after an additional 30d of inactivity. If this issue is safe to close now please do so with /close. Provide feedback via https://jenkins-x.io/community. /lifecycle rotten

Adercley commented 3 years ago

Are you using HTTPS?? If so, try to use without it, just http.

So, instead of using: https://yourdomain.com:8099/github-webhook/, use

http://yourdomain.com:8099/github-webhook/

and you can use application/json as CONTENT TYPE.. it will work.

Good luck!

jenkins-x-bot commented 3 years ago

Rotten issues close after 30d of inactivity. Reopen the issue with /reopen. Mark the issue as fresh with /remove-lifecycle rotten. Provide feedback via https://jenkins-x.io/community. /close

jenkins-x-bot commented 3 years ago

@jenkins-x-bot: Closing this issue.

In response to [this](https://github.com/jenkins-x/jx/issues/5633#issuecomment-763866234): >Rotten issues close after 30d of inactivity. >Reopen the issue with `/reopen`. >Mark the issue as fresh with `/remove-lifecycle rotten`. >Provide feedback via https://jenkins-x.io/community. >/close Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [jenkins-x/lighthouse](https://github.com/jenkins-x/lighthouse/issues/new?title=Command%20issue:) repository.
jarahul1209 commented 3 years ago

ngrok solution worked for me... very cool!! Thanks all

SELVAKUMAR-SD commented 3 years ago

i have changed the IP of jenkins to public ip in webhook url and it worked.

That works. Use the public IP of your server in the url instead of IP provided under Git in Configure Settings section of Jenkins.

Im using like that but im facing the same issue

tpham210 commented 3 years ago

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server" Github won't be able to communicate with Jenkins which is running local /private IP address . You need to specify an IP address GitHub can contact over the internet not the actual IP address . Steps: 1.Download the https://ngrok.com/download server where it is running with Jenkins . 2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins . 3.http://<......>ngrok.io eg http://3b2db437.ngrok.io 4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ .. ====>Don't give port 8080 as we already passed /ngrok http 8080 5.Final test the github webhook status .

You're the best, bro. Worked for me! (When you run ngrok in the terminal, keep it there, don't close it, so it can "introspected tunnels to localhost") Not sure if this is the original repo: https://github.com/inconshreveable/ngrok But you can see the diagram there to make sense out of this, if you're a beginner like me

saurav12345 commented 1 year ago

you can use the above ip http://193.57.249.2:8082/ to access jenkins dashboard within your company using VPN. But github needs an IP ( public ) which can be accessible through internet so your payload should be

http://publicip:8082/github-webhook/

and using http://frparvm-blkchn:8082 as it's not a public domain its just the hostname so github doesn't know what it is... as github is not hosted in your company network ;) just kidding.

@vamsilella how to find the public ip of jenkins hosted on an azure vm. I can access jenkins using http://ip:8080, if this ip is not a public ip then how can I get it?

karthikinfo2001 commented 1 year ago

Actually am using in virtual box centos.jenkins is running but it shows connection failed to the host.please give a solution.am using the inet ip ipv4

KiritiAryal commented 11 months ago

Please follow the below steps to fix the "Jenkins+Github: We couldn’t deliver this payload: Couldn't connect to server" Github won't be able to communicate with Jenkins which is running local /private IP address . You need to specify an IP address GitHub can contact over the internet not the actual IP address . Steps: 1.Download the https://ngrok.com/download server where it is running with Jenkins . 2../ngrok http 8080 ,it will give one http URL ..wait for some time ..some time taking 2 to 3 mins . 3.http://<......>ngrok.io eg http://3b2db437.ngrok.io 4.http://<...>/github-webhook/ eg http://3b2db437.ngrok.io/github-webhook/ .. ====>Don't give port 8080 as we already passed /ngrok http 8080 5.Final test the github webhook status .

This works for testing purposes