search

softdevteam / mattermost-github-integration

GitHub integration for Mattermost
Other
79 stars 50 forks source link

Not implemented return response from Github testing #8

Closed remimarenco closed 8 years ago

remimarenco commented 8 years ago

Hi!

Thanks for this project, it is awesome to be able to get the github notifications :).

Although, I encounter a few issues while trying to set it up.

My config.py is like this:

USERNAME = "rmarenco"
ICON_URL = "octodex.github.com/images/octobiwan.jpg"
MATTERMOST_WEBHOOK_URLS = {
    'default' : ("mattermost.remionramp.com/hooks/hook1", "off-topic"),
    'remimarenco/gonramp' : ("mattermost.remionramp.com/hooks/hook2", "Github"),
    'remimarenco' : ("mattermost.remionramp.com/hooks/hook3", "town-square")
}
SECRET = "mysecret"
SHOW_AVATARS = True

But I am not sure about what to insert into USERNAME...

My webhook on github side is like this:

Payload URL: http://mattermost.remionramp.com:5000
Content type: application/json
Secret: mysecret

And:

When I try to test with Github feature, I have a return response 400 and a body like this:

Not implemented

Do you know why? How can I investigate on my side?

Also, I have nginx on mattermost.remionramp.com

Thanks!

Remi

ptersilie commented 8 years ago

Hi @remimarenco! The username is not really important. It is only used to post the messages into Mattermost (use "GithubBot" or anything you like).

About the "Not Implemented" message. This usually means that the notification you are trying to send hasn't been implemented (yet). There are quite a few notifications that Github creates that we didn't find very important and thus didn't forward to Mattermost (especially since some notifications are reduandant).

Can you try to test this with a notification like a comment in an issue or a commit and tell me what happens then?

ptersilie commented 8 years ago

Ah I think I see what is happening now. You checked "Just the push event", which is something I admittedly didn't try as in our setup we selected individual events. If you post the event details (json) that Github is sending I can have a look how I have to handle these type of events.

remimarenco commented 8 years ago

Hi @ptersilie! Thanks for answering so quickly :).

So I selected individual event to try it through the testing Gihub interface, and not much :/. Here is the json:

Payload

{
  "zen": "Practicality beats purity.",
  "hook_id": 8994512,
  "hook": {
    "type": "Repository",
    "id": 8994512,
    "name": "web",
    "active": true,
    "events": [
      "push"
    ],
    "config": {
      "content_type": "json",
      "insecure_ssl": "0",
      "url": "http://mattermost.remionramp.com:5000"
    },
    "updated_at": "2016-07-05T00:49:19Z",
    "created_at": "2016-07-05T00:49:19Z",
    "url": "https://api.github.com/repos/remimarenco/gonramp/hooks/8994512",
    "test_url": "https://api.github.com/repos/remimarenco/gonramp/hooks/8994512/test",
    "ping_url": "https://api.github.com/repos/remimarenco/gonramp/hooks/8994512/pings",
    "last_response": {
      "code": null,
      "status": "unused",
      "message": null
    }
  },
  "repository": {
    "id": 62431522,
    "name": "gonramp",
    "full_name": "remimarenco/gonramp",
    "owner": {
      "login": "remimarenco",
      "id": 2152858,
      "avatar_url": "https://avatars.githubusercontent.com/u/2152858?v=3",
      "gravatar_id": "",
      "url": "https://api.github.com/users/remimarenco",
      "html_url": "https://github.com/remimarenco",
      "followers_url": "https://api.github.com/users/remimarenco/followers",
      "following_url": "https://api.github.com/users/remimarenco/following{/other_user}",
      "gists_url": "https://api.github.com/users/remimarenco/gists{/gist_id}",
      "starred_url": "https://api.github.com/users/remimarenco/starred{/owner}{/repo}",
      "subscriptions_url": "https://api.github.com/users/remimarenco/subscriptions",
      "organizations_url": "https://api.github.com/users/remimarenco/orgs",
      "repos_url": "https://api.github.com/users/remimarenco/repos",
      "events_url": "https://api.github.com/users/remimarenco/events{/privacy}",
      "received_events_url": "https://api.github.com/users/remimarenco/received_events",
      "type": "User",
      "site_admin": false
    },
    "private": false,
    "html_url": "https://github.com/remimarenco/gonramp",
    "description": "Repository to gather all the G-OnRamp specific resource",
    "fork": false,
    "url": "https://api.github.com/repos/remimarenco/gonramp",
    "forks_url": "https://api.github.com/repos/remimarenco/gonramp/forks",
    "keys_url": "https://api.github.com/repos/remimarenco/gonramp/keys{/key_id}",
    "collaborators_url": "https://api.github.com/repos/remimarenco/gonramp/collaborators{/collaborator}",
    "teams_url": "https://api.github.com/repos/remimarenco/gonramp/teams",
    "hooks_url": "https://api.github.com/repos/remimarenco/gonramp/hooks",
    "issue_events_url": "https://api.github.com/repos/remimarenco/gonramp/issues/events{/number}",
    "events_url": "https://api.github.com/repos/remimarenco/gonramp/events",
    "assignees_url": "https://api.github.com/repos/remimarenco/gonramp/assignees{/user}",
    "branches_url": "https://api.github.com/repos/remimarenco/gonramp/branches{/branch}",
    "tags_url": "https://api.github.com/repos/remimarenco/gonramp/tags",
    "blobs_url": "https://api.github.com/repos/remimarenco/gonramp/git/blobs{/sha}",
    "git_tags_url": "https://api.github.com/repos/remimarenco/gonramp/git/tags{/sha}",
    "git_refs_url": "https://api.github.com/repos/remimarenco/gonramp/git/refs{/sha}",
    "trees_url": "https://api.github.com/repos/remimarenco/gonramp/git/trees{/sha}",
    "statuses_url": "https://api.github.com/repos/remimarenco/gonramp/statuses/{sha}",
    "languages_url": "https://api.github.com/repos/remimarenco/gonramp/languages",
    "stargazers_url": "https://api.github.com/repos/remimarenco/gonramp/stargazers",
    "contributors_url": "https://api.github.com/repos/remimarenco/gonramp/contributors",
    "subscribers_url": "https://api.github.com/repos/remimarenco/gonramp/subscribers",
    "subscription_url": "https://api.github.com/repos/remimarenco/gonramp/subscription",
    "commits_url": "https://api.github.com/repos/remimarenco/gonramp/commits{/sha}",
    "git_commits_url": "https://api.github.com/repos/remimarenco/gonramp/git/commits{/sha}",
    "comments_url": "https://api.github.com/repos/remimarenco/gonramp/comments{/number}",
    "issue_comment_url": "https://api.github.com/repos/remimarenco/gonramp/issues/comments{/number}",
    "contents_url": "https://api.github.com/repos/remimarenco/gonramp/contents/{+path}",
    "compare_url": "https://api.github.com/repos/remimarenco/gonramp/compare/{base}...{head}",
    "merges_url": "https://api.github.com/repos/remimarenco/gonramp/merges",
    "archive_url": "https://api.github.com/repos/remimarenco/gonramp/{archive_format}{/ref}",
    "downloads_url": "https://api.github.com/repos/remimarenco/gonramp/downloads",
    "issues_url": "https://api.github.com/repos/remimarenco/gonramp/issues{/number}",
    "pulls_url": "https://api.github.com/repos/remimarenco/gonramp/pulls{/number}",
    "milestones_url": "https://api.github.com/repos/remimarenco/gonramp/milestones{/number}",
    "notifications_url": "https://api.github.com/repos/remimarenco/gonramp/notifications{?since,all,participating}",
    "labels_url": "https://api.github.com/repos/remimarenco/gonramp/labels{/name}",
    "releases_url": "https://api.github.com/repos/remimarenco/gonramp/releases{/id}",
    "deployments_url": "https://api.github.com/repos/remimarenco/gonramp/deployments",
    "created_at": "2016-07-02T03:09:26Z",
    "updated_at": "2016-07-03T23:00:22Z",
    "pushed_at": "2016-07-03T23:15:14Z",
    "git_url": "git://github.com/remimarenco/gonramp.git",
    "ssh_url": "git@github.com:remimarenco/gonramp.git",
    "clone_url": "https://github.com/remimarenco/gonramp.git",
    "svn_url": "https://github.com/remimarenco/gonramp",
    "homepage": "",
    "size": 3,
    "stargazers_count": 0,
    "watchers_count": 0,
    "language": null,
    "has_issues": true,
    "has_downloads": true,
    "has_wiki": true,
    "has_pages": false,
    "forks_count": 0,
    "mirror_url": null,
    "open_issues_count": 2,
    "forks": 0,
    "open_issues": 2,
    "watchers": 0,
    "default_branch": "master"
  },
  "sender": {
    "login": "remimarenco",
    "id": 2152858,
    "avatar_url": "https://avatars.githubusercontent.com/u/2152858?v=3",
    "gravatar_id": "",
    "url": "https://api.github.com/users/remimarenco",
    "html_url": "https://github.com/remimarenco",
    "followers_url": "https://api.github.com/users/remimarenco/followers",
    "following_url": "https://api.github.com/users/remimarenco/following{/other_user}",
    "gists_url": "https://api.github.com/users/remimarenco/gists{/gist_id}",
    "starred_url": "https://api.github.com/users/remimarenco/starred{/owner}{/repo}",
    "subscriptions_url": "https://api.github.com/users/remimarenco/subscriptions",
    "organizations_url": "https://api.github.com/users/remimarenco/orgs",
    "repos_url": "https://api.github.com/users/remimarenco/repos",
    "events_url": "https://api.github.com/users/remimarenco/events{/privacy}",
    "received_events_url": "https://api.github.com/users/remimarenco/received_events",
    "type": "User",
    "site_admin": false
  }
}
ptersilie commented 8 years ago

Is this a ping event? Those are (currently) not forwarded by the plugin. Can you try with any of the following events and tell me what the response is: Issue comment, Commit comment, Push, Issues (open, delete, etc), Repository (open, delete, etc), Create (branch, tag).

remimarenco commented 8 years ago

Sure! I tried with Issue comment and the plugin crashed:

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
  "http://www.w3.org/TR/html4/loose.dtd">
<html>
  <head>
    <title>MissingSchema: Invalid URL 'mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h': No schema supplied. Perhaps you meant http://mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h? // Werkzeug Debugger</title>
    <link rel="stylesheet" href="?__debugger__=yes&amp;cmd=resource&amp;f=style.css"
        type="text/css">
    <!-- We need to make sure this has a favicon so that the debugger does
         not by accident trigger a request to /favicon.ico which might
         change the application state. -->
    <link rel="shortcut icon"
        href="?__debugger__=yes&amp;cmd=resource&amp;f=console.png">
    <script src="?__debugger__=yes&amp;cmd=resource&amp;f=jquery.js"></script>
    <script src="?__debugger__=yes&amp;cmd=resource&amp;f=debugger.js"></script>
    <script type="text/javascript">
      var TRACEBACK = 139653275215376,
          CONSOLE_MODE = false,
          EVALEX = true,
          EVALEX_TRUSTED = false,
          SECRET = "N9Xa4QxXKIizeYArn2B1";
    </script>
  </head>
  <body>
    <div class="debugger">
<h1>requests.exceptions.MissingSchema</h1>
<div class="detail">
  <p class="errormsg">MissingSchema: Invalid URL 'mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h': No schema supplied. Perhaps you meant http://mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h?</p>
</div>
<h2 class="traceback">Traceback <em>(most recent call last)</em></h2>
<div class="traceback">
remimarenco commented 8 years ago

I have more details if you want, on Github side. And on server side:

Traceback (most recent call last):
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/remi/mattermost-github-integration/server.py", line 77, in root
    post(msg, url, channel)
  File "/home/remi/mattermost-github-integration/server.py", line 90, in post
    r = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/sessions.py", line 461, in request
    prep = self.prepare_request(req)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/sessions.py", line 394, in prepare_request
    hooks=merge_hooks(request.hooks, self.hooks),
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/models.py", line 295, in prepare
    self.prepare_url(url, params)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/models.py", line 355, in prepare_url
    raise MissingSchema(error)
MissingSchema: Invalid URL 'mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h': No schema supplied. Perhaps you meant http://mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h?
ptersilie commented 8 years ago

Oh, interesting. Can you try and add "http://" to the webhooks in your config.py?

remimarenco commented 8 years ago

Just did, trying :)

remimarenco commented 8 years ago 
Traceback (most recent call last):
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 2000, in __call__
    return self.wsgi_app(environ, start_response)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1991, in wsgi_app
    response = self.make_response(self.handle_exception(e))
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1567, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1988, in wsgi_app
    response = self.full_dispatch_request()
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1641, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1544, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1639, in full_dispatch_request
    rv = self.dispatch_request()
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/flask/app.py", line 1625, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/remi/mattermost-github-integration/server.py", line 77, in root
    post(msg, url, channel)
  File "/home/remi/mattermost-github-integration/server.py", line 90, in post
    r = requests.post(url, headers=headers, data=json.dumps(data), verify=False)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/api.py", line 111, in post
    return request('post', url, data=data, json=json, **kwargs)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/api.py", line 57, in request
    return session.request(method=method, url=url, **kwargs)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/sessions.py", line 475, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/sessions.py", line 606, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/home/remi/mattermost-github-integration/venv/lib/python2.7/site-packages/requests/sessions.py", line 113, in resolve_redirects
    raise TooManyRedirects('Exceeded %s redirects.' % self.max_redirects, response=resp)
TooManyRedirects: Exceeded 30 redirects.
remimarenco commented 8 years ago

Reading this: http://stackoverflow.com/questions/30574314/web-crawler-toomanyredirects-exceeded-30-redirects-python

Could this help? (Sorry if it seems obvious...just googled heh)

ptersilie commented 8 years ago

Seems similar to what I found here: http://stackoverflow.com/questions/27766515/toomanyredirects-error-with-requests-module

ptersilie commented 8 years ago

If I understood this correctly the reason is that your website doesn't like the useragent (or lack of it). Seems to be fixable by either faking the useragent. It also might be fixable on the server side.

remimarenco commented 8 years ago

I don't find in your code where you are handling the User-Agent from Github, will try to add it to the headers?

ptersilie commented 8 years ago

This is where I set the headers before sending stuff to the mattermost hook: https://github.com/softdevteam/mattermost-github-integration/blob/master/server.py#L90

ptersilie commented 8 years ago

Hmm, I just noticed that the stackoverflow posts are about receiving requests request.get, while your error happens when sending them request.post. Not sure the solution from those threads will work for this case. Something is odd and it seems as if your mattermost hook url redirects indefinitely.

remimarenco commented 8 years ago

Does it come from my nginx set-up maybe? What could trigger this behavior?

ptersilie commented 8 years ago

I'm not sure. I am using nginx as well and never ran into this issue. Let's see if the mattermost webhooks work at all (without having to go through Github each time). The following simple script should post a simple message to the specified mattermost webhook. Put that in a python script and run it and see what happens.

import requests
import json

MATTERMOST_WEBHOOK_URL = "your hook url"

data = {}
data['text'] = "Let's see if this works"
data['channel'] = "town-square"
data['username'] = "Bot"
data['icon_url'] = ""

headers = {'Content-Type': 'application/json'}
r = requests.post(MATTERMOST_WEBHOOK_URL, headers=headers, data=json.dumps(data), verify=False)
if r.status_code is not requests.codes.ok:
    print 'Encountered error posting to Mattermost URL %s, status=%d, response_body=%s' % (MATTERMOST_WEBHOOK_URL, r.status_code, r.json())
ptersilie commented 8 years ago

I don't really know what could cause the TooManyRedirects error. It might just be a typo in your webhook url. Or it could be something more complicated in your nginx setup that redirects urls.

remimarenco commented 8 years ago

Ok thanks will try to investigate that :).

ptersilie commented 8 years ago

Let me know what you find out. :) I have to go to sleep now, though, so will only be able to reply tomorrow.

remimarenco commented 8 years ago

Cool thanks! Have a good night then ;)

remimarenco commented 8 years ago

Trying to add allow_redirects=False, got me to another error:

  File "/usr/lib/python2.7/json/decoder.py", line 382, in raw_decode
    raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
remimarenco commented 8 years ago

So I had a r.json exception because there:

if r.status_code is not requests.codes.ok:
    print 'Encountered error posting to Mattermost URL %s, status=%d, response_body=%s' % (url, r.status_code, r.json())

json() was not good. But we entered in a case which was not exactly true, because it was a redirection, not really an erorr (the status code is 307).

So I got this working by doing this:

if r.status_code is not requests.codes.ok:
        if r.status_code == 307:
            print 'Warning: Received a redirection posting to Mattermost URL %s, status=%d' % (url, r.status_code)
        else:
            print 'Encountered error posting to Mattermost URL %s, status=%d, response_body=%s' % (url, r.status_code, r.json())

I receive a 200 respoonse on GitHub side, but nothing is updated in Mattermost although the print is:

Warning: Received a redirection posting to Mattermost URL http://mattermost.remionramp.com/hooks/iu5gzygu53bizkkf78fciwam5h, status=307
192.30.252.34 - - [05/Jul/2016 21:53:14] "POST / HTTP/1.1" 200 -
remimarenco commented 8 years ago

And it did work your example!! Ok I will try to figure out what is wrong based on this

remimarenco commented 8 years ago

Found it! The difference was the channel...I used a uppercase (as it was in my Mattermost client) whereas it should be lower_case. I will release a fix for that

remimarenco commented 8 years ago

xref https://github.com/softdevteam/mattermost-github-integration/pull/9

ptersilie commented 8 years ago

Good find and thanks a lot for the PR.

remimarenco commented 8 years ago

Thanks for the project :)