HTTP check fails to handle https://wnpp.debian.net/ ? (raw data "None")

[hartwork]

Hi!

I have a nuch of HTTP checks that all work well. One case is interesting — URL https://wnpp.debian.net/ with no HTTP auth and default check config —, as it works perfectly with wget and the browser but Cabot shows status "Failed", raw data "None" and no helpful information at all. See:

Would be cool if the issue could be fixed in Cabot, either by being successful or by providing error details. Thanks!

PS: I'm running Git HEAD with docker-compose here.

[dbuxton]

What does it say on the console, if anything?

If you go into the console (docker-compose run --rm web ./manage.py shell) and simply do import requests; requests.get('https://wnpp.debian.net/') do you get an error?

Just a guess but maybe something to do with certificates package? That would be worth fixing if so.

[hartwork]

Interestingly, requests is happy on the shell:

# docker-compose run --rm web ./manage.py shell
Starting cabot-git_redis_1 ... done
Starting cabot-git_db_1    ... done
Operations to perform:
  Apply all migrations: admin, auth, cabot_alert_email, cabot_alert_hipchat, cabot_alert_slack, cabot_alert_twilio, cabotapp, contenttypes, sessions, sites
Running migrations:
  No migrations to apply.
Found another file with the destination path 'admin/js/jquery.init.js'. It will be ignored since only the first encountered file is collected. If this is not what you want, make sure every static file has a unique path.

0 static files copied to '/code/cabot/.collectstatic', 401 unmodified.
Python 2.7.14 (default, Feb 22 2018, 22:26:54) 
Type "copyright", "credits" or "license" for more information.

IPython 5.6.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.

In [1]: import requests; requests.get('https://wnpp.debian.net/')
Out[1]: <Response [200]>

[hartwork]

Debugging it more now, have a first clue now...

[hartwork]

Please check #610 for a fix.

For the full traceback:

In [7]: from cabot.cabotapp.models.base import HttpStatusCheck

In [8]: check = HttpStatusCheck.objects.get(endpoint='https://wnpp.debian.net/')

In [9]: check._run()
/usr/lib/python2.7/site-packages/requests/packages/urllib3/connectionpool.py:852: InsecureRequestWarning: Unverified HTTPS request is being made. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecureRequestWarning)
---------------------------------------------------------------------------
UnicodeDecodeError                        Traceback (most recent call last)
<ipython-input-9-800e429ac9be> in <module>()
----> 1 check._run()

/code/cabot/cabotapp/models/base.pyc in _run(self)
    783                 result.raw_data = resp.content
    784             elif self.text_match:
--> 785                 if not self._check_content_pattern(self.text_match, resp.content):
    786                     result.error = u'Failed to find match regex /%s/ in response body' % self.text_match
    787                     result.raw_data = resp.content

/code/cabot/cabotapp/models/base.pyc in _check_content_pattern(self, text_match, content)
    753     @classmethod
    754     def _check_content_pattern(self, text_match, content):
--> 755         content = content if isinstance(content, unicode) else unicode(content, "UTF-8")
    756         return re.search(text_match, content)
    757

/usr/lib/python2.7/encodings/utf_8.pyc in decode(input, errors)
     14
     15 def decode(input, errors='strict'):
---> 16     return codecs.utf_8_decode(input, errors, True)
     17
     18 class IncrementalEncoder(codecs.IncrementalEncoder):

UnicodeDecodeError: 'utf8' codec can't decode bytes in position 956177-956178: invalid continuation byte

[hartwork]

Seems fixed, closing.