Sentry responded with an error: EOF occurred in violation of protocol

jerneju commented 6 years ago

We are having this issue for almost three months. It is the same issue as posted in another thread: https://github.com/getsentry/raven-python/issues/702#issuecomment-336833338

ERROR sentry.errors base.py Sentry responded with an error: <urlopen error [Errno 8] _ssl.c:504: EOF occurred in violation of protocol> (url: https://sentry.io/api/XXXXXX/store/) Traceback (most recent call last): File "/srv/envs/orange-web/local/lib/python2.7/site-packages/raven/transport/threaded.py", line 172, in send_sync super(ThreadedHTTPTransport, self).send(url, data, headers) File "/srv/envs/orange-web/local/lib/python2.7/site-packages/raven/transport/http.py", line 43, in send ca_certs=self.ca_certs, File "/srv/envs/orange-web/local/lib/python2.7/site-packages/raven/utils/http.py", line 66, in urlopen return opener.open(url, data, timeout) File "/usr/lib/python2.7/urllib2.py", line 401, in open response = self._open(req, data) File "/usr/lib/python2.7/urllib2.py", line 419, in _open '_open', req) File "/usr/lib/python2.7/urllib2.py", line 379, in _call_chain result = func(*args) File "/srv/envs/orange-web/local/lib/python2.7/site-packages/raven/utils/http.py", line 46, in https_open return self.do_open(ValidHTTPSConnection, req) File "/usr/lib/python2.7/urllib2.py", line 1181, in do_open raise URLError(err)

ghost commented 6 years ago

I've got something similar?

Traceback (most recent call last):                                            
  File "/home/app/v0.3.4/venv/lib/python3.5/site-packages/raven/transport/threaded.py", line 172, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)               
  File "/home/app/v0.3.4/venv/lib/python3.5/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,                                                   
  File "/home/app/v0.3.4/venv/lib/python3.5/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)                                    
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open              
    response = self._open(req, data)                                          
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open             
    '_open', req)                                                             
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain          
    result = func(*args)                                                      
  File "/home/app/v0.3.4/venv/lib/python3.5/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)                            
  File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open          
    raise URLError(err)                                                       
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:645)>

UPDATE: @ashwoods

certifi==2017.7.27.1

OpenSSL 1.0.2g 1 Mar 2016 built on: reproducible build, date unspecified platform: debian-amd64

ashwoods commented 6 years ago

Can you post your versions of certifi and openssl?

jerneju commented 6 years ago

OpenSSL 1.0.1t 3 May 2016 and no certifi installed.

jerneju commented 6 years ago

certifi==2017.07.27.1 but still does not work.

ashwoods commented 6 years ago

Our current guess it has to do with the default timeout. Will Be Rolling out a fix today. On Thu, 19 Oct 2017 at 15:14, Jernej Urankar notifications@github.com wrote:

certifi==2017.07.27.1 but still does not work.

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/getsentry/raven-python/issues/1109#issuecomment-337903929, or mute the thread https://github.com/notifications/unsubscribe-auth/AAW4tSPjUSb0qA3uxOLofscYaf1LvvgSks5st0tFgaJpZM4P9U8W .

ghost commented 6 years ago

any update?

ashwoods commented 6 years ago

We raised the default timeout in 6.3.0. @xliiv still having issues?

ghost commented 6 years ago

It works, thanks :)

ashwoods commented 6 years ago

Closing this for now, although further improvements are needed.

jerneju commented 6 years ago

We have upgraded raven to 6.3.0 but this error still persists.

jerneju commented 6 years ago

In my opinion this issue needs to be reopened since above mentioned error exceptions are still occurring.

ghost commented 6 years ago

Unfortunately, I second that. Recently I got exception again, and wsgi process took much of CPU until I restarted it. And exception wasn't pushed to sentry.

ashwoods commented 6 years ago

this issue has been kinda tricky to reproduce and I think there are several problems here. Could you provide some info on your stack?

ghost commented 6 years ago

Traceback (most recent call last):
  File "/usr/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/home/app/v0.6.0/venv/lib/python3.5/site-packages/raven/utils/http.py", line 38, in connect
    sock, ca_certs=ca_certs, cert_reqs=ssl.CERT_REQUIRED)
  File "/usr/lib/python3.5/ssl.py", line 1069, in wrap_socket
    ciphers=ciphers)
  File "/usr/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/app/v0.6.0/venv/lib/python3.5/site-packages/raven/transport/threaded.py", line 172, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/home/app/v0.6.0/venv/lib/python3.5/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/home/app/v0.6.0/venv/lib/python3.5/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/home/app/v0.6.0/venv/lib/python3.5/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:645)>

This is all i've got.

dhrp commented 6 years ago

I get the same, and am very surprised.

Our setup:

Running in Docker container on K8S, no limits set
Host has plenty of RAM available
Reaching https hosts from the very same container (curl), and even using python request.urlopen gives no issues.
This probably started happening after our upgrade from raven 6.1.0 to 6.4.0, but I'm not 100% sure, as I can't really tell what events I have not received.

More info:

$ pip freeze |grep raven
raven==6.4.0

request.urlopen('https://sentry.<ourcompany>/')
<http.client.HTTPResponse object at 0x7f9630c70278>

ERROR:sentry.errors:Sentry responded with an error: <urlopen error EOF occurred in violation of protocol (_ssl.c:719)> (url: https://sentry.<ourcompany>/api/10/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/urllib/request.py", line 1254, in do_open
    h.request(req.get_method(), req.selector, req.data, headers)
  File "/usr/local/lib/python3.5/http/client.py", line 1107, in request
    self._send_request(method, url, body, headers)
  File "/usr/local/lib/python3.5/http/client.py", line 1152, in _send_request
    self.endheaders(body)
  File "/usr/local/lib/python3.5/http/client.py", line 1103, in endheaders
    self._send_output(message_body)
  File "/usr/local/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/root/venv/lib/python3.5/site-packages/raven/utils/http.py", line 38, in connect
    sock, ca_certs=ca_certs, cert_reqs=ssl.CERT_REQUIRED)
  File "/usr/local/lib/python3.5/ssl.py", line 1077, in wrap_socket
    ciphers=ciphers)
  File "/usr/local/lib/python3.5/ssl.py", line 760, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.5/ssl.py", line 996, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.5/ssl.py", line 641, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLEOFError: EOF occurred in violation of protocol (_ssl.c:719)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/venv/lib/python3.5/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/root/venv/lib/python3.5/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/root/venv/lib/python3.5/site-packages/raven/utils/http.py", line 66, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.5/urllib/request.py", line 466, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.5/urllib/request.py", line 484, in _open
    '_open', req)
  File "/usr/local/lib/python3.5/urllib/request.py", line 444, in _call_chain
    result = func(*args)
  File "/root/venv/lib/python3.5/site-packages/raven/utils/http.py", line 46, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/local/lib/python3.5/urllib/request.py", line 1256, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error EOF occurred in violation of protocol (_ssl.c:719)>

update: I am using a wildcard certificate for my sentry deployment.

im-n1 commented 6 years ago

Hi guys, I'm getting similar error:

[pht] out: b'Sentry responded with an error: <urlopen error [Errno 0] Error> (url: https://sentry.io/api/216427/store/)'
[pht] out: ERROR | 2018-05-22 20:19:43,638 | base | Sentry responded with an error: <urlopen error [Errno 0] Error> (url: https://sentry.io/api/216427/store/)
[pht] out: Traceback (most recent call last):
[pht] out:   File "/usr/lib/python3.6/urllib/request.py", line 1318, in do_open
[pht] out:     encode_chunked=req.has_header('Transfer-encoding'))
[pht] out:   File "/usr/lib/python3.6/http/client.py", line 1239, in request
[pht] out:     self._send_request(method, url, body, headers, encode_chunked)
[pht] out:   File "/usr/lib/python3.6/http/client.py", line 1285, in _send_request
[pht] out:     self.endheaders(body, encode_chunked=encode_chunked)
[pht] out:   File "/usr/lib/python3.6/http/client.py", line 1234, in endheaders
[pht] out:     self._send_output(message_body, encode_chunked=encode_chunked)
[pht] out:   File "/usr/lib/python3.6/http/client.py", line 1026, in _send_output
[pht] out:     self.send(msg)
[pht] out:   File "/usr/lib/python3.6/http/client.py", line 964, in send
[pht] out:     self.connect()
[pht] out:   File "/usr/local/lib/python3.6/dist-packages/raven/utils/http.py", line 38, in connect
[pht] out:     sock, ca_certs=ca_certs, cert_reqs=ssl.CERT_REQUIRED)
[pht] out:   File "/usr/lib/python3.6/ssl.py", line 1149, in wrap_socket
[pht] out:     ciphers=ciphers)
[pht] out:   File "/usr/lib/python3.6/ssl.py", line 814, in __init__
[pht] out:     self.do_handshake()
[pht] out:   File "/usr/lib/python3.6/ssl.py", line 1068, in do_handshake
[pht] out:     self._sslobj.do_handshake()
[pht] out:   File "/usr/lib/python3.6/ssl.py", line 689, in do_handshake
[pht] out:     self._sslobj.do_handshake()
[pht] out: OSError: [Errno 0] Error
[pht] out: 
[pht] out: During handling of the above exception, another exception occurred:
[pht] out: 
[pht] out: Traceback (most recent call last):
[pht] out:   File "/usr/local/lib/python3.6/dist-packages/raven/transport/threaded.py", line 165, in send_sync
[pht] out:     super(ThreadedHTTPTransport, self).send(url, data, headers)
[pht] out:   File "/usr/local/lib/python3.6/dist-packages/raven/transport/http.py", line 43, in send
[pht] out:     ca_certs=self.ca_certs,
[pht] out:   File "/usr/local/lib/python3.6/dist-packages/raven/utils/http.py", line 66, in urlopen
[pht] out:     return opener.open(url, data, timeout)
[pht] out:   File "/usr/lib/python3.6/urllib/request.py", line 526, in open
[pht] out:     response = self._open(req, data)
[pht] out:   File "/usr/lib/python3.6/urllib/request.py", line 544, in _open
[pht] out:     '_open', req)
[pht] out:   File "/usr/lib/python3.6/urllib/request.py", line 504, in _call_chain
[pht] out:     result = func(*args)
[pht] out:   File "/usr/local/lib/python3.6/dist-packages/raven/utils/http.py", line 46, in https_open
[pht] out:     return self.do_open(ValidHTTPSConnection, req)
[pht] out:   File "/usr/lib/python3.6/urllib/request.py", line 1320, in do_open
[pht] out:     raise URLError(err)

Sentry worked percertly and now I'm getting this. Today I've upgraded raven but the error remains. What to do?

kstromeiraos commented 6 years ago

We are having the same error with Raven 6.8.0, is there any update on this?

lmajowka commented 6 years ago

Also having this problem :(

dmyerscough commented 6 years ago

I was able to reproduce this problem when running a Django application behind Istio Service Mesh:-

./manage.py shell --settings=mirussh.settings.staging
Python 3.6.5 (default, Jun 27 2018, 08:18:52) 
[GCC 6.3.0 20170516] on linux
Type "help", "copyright", "credits" or "license" for more information.
(InteractiveConsole)
>>> from raven.contrib.django.raven_compat.models import client
>>> 
>>> client.captureException()

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 824, in captureException
    'raven.events.Exception', exc_info=exc_info, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/contrib/django/client.py", line 303, in capture
    result = super(DjangoClient, self).capture(event_type, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 635, in capture
    elif not self.should_capture(exc_info):
  File "/usr/local/lib/python3.6/site-packages/raven/base.py", line 828, in should_capture
    exc_name = '%s.%s' % (exc_type.__module__, exc_type.__name__)
AttributeError: 'NoneType' object has no attribute '__module__'
>>> 
2018-08-11 08:05:39,009 sentry.errors ERROR    Sentry responded with an error: <urlopen error [Errno 0] Error> (url: https://sentry.io/api/1231405/store/)
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/urllib/request.py", line 1318, in do_open
    encode_chunked=req.has_header('Transfer-encoding'))
  File "/usr/local/lib/python3.6/http/client.py", line 1239, in request
    self._send_request(method, url, body, headers, encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1285, in _send_request
    self.endheaders(body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1234, in endheaders
    self._send_output(message_body, encode_chunked=encode_chunked)
  File "/usr/local/lib/python3.6/http/client.py", line 1026, in _send_output
    self.send(msg)
  File "/usr/local/lib/python3.6/http/client.py", line 964, in send
    self.connect()
  File "/usr/local/lib/python3.6/site-packages/raven/utils/http.py", line 39, in connect
    sock, ca_certs=ca_certs, cert_reqs=ssl.CERT_REQUIRED)
  File "/usr/local/lib/python3.6/ssl.py", line 1149, in wrap_socket
    ciphers=ciphers)
  File "/usr/local/lib/python3.6/ssl.py", line 814, in __init__
    self.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/local/lib/python3.6/ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
OSError: [Errno 0] Error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/raven/transport/threaded.py", line 165, in send_sync
    super(ThreadedHTTPTransport, self).send(url, data, headers)
  File "/usr/local/lib/python3.6/site-packages/raven/transport/http.py", line 43, in send
    ca_certs=self.ca_certs,
  File "/usr/local/lib/python3.6/site-packages/raven/utils/http.py", line 67, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/local/lib/python3.6/urllib/request.py", line 526, in open
    response = self._open(req, data)
  File "/usr/local/lib/python3.6/urllib/request.py", line 544, in _open
    '_open', req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 504, in _call_chain
    result = func(*args)
  File "/usr/local/lib/python3.6/site-packages/raven/utils/http.py", line 47, in https_open
    return self.do_open(ValidHTTPSConnection, req)
  File "/usr/local/lib/python3.6/urllib/request.py", line 1320, in do_open
    raise URLError(err)
urllib.error.URLError: <urlopen error [Errno 0] Error>
b'Sentry responded with an error: <urlopen error [Errno 0] Error> (url: https://sentry.io/api/1231405/store/)'
2018-08-11 08:05:39,011 sentry.errors.uncaught ERROR    ["AttributeError: 'NoneType' object has no attribute '__module__'", '  File "python3.6/code.py", line 91, in runcode', '  File "<console>", line 1, in <module>', '  File "raven/base.py", line 824, in captureException', '  File "raven/contrib/django/client.py", line 303, in capture', '  File "raven/base.py", line 635, in capture', '  File "raven/base.py", line 828, in should_capture']
b'["AttributeError: \'NoneType\' object has no attribute \'__module__\'", \'  File "python3.6/code.py", line 91, in runcode\', \'  File "<console>", line 1, in <module>\', \'  File "raven/base.py", line 824, in captureException\', \'  File "raven/contrib/django/client.py", line 303, in capture\', \'  File "raven/base.py", line 635, in capture\', \'  File "raven/base.py", line 828, in should_capture\']'

I modifying the raven/transport/http.py to use requests instead and the exception went away:-

    def send(self, url, data, headers):
        """
        Sends a request to a remote webserver using HTTP POST.
        """
        #req = urllib2.Request(url, headers=headers)

        try:
            response = requests.post(
                url=url,
                data=data,
                timeout=self.timeout,
                verify=self.verify_ssl,
                headers=headers
            )
#            response = urlopen(
#                url=req,
#                data=data,
#                timeout=self.timeout,
#                verify_ssl=self.verify_ssl,
#                ca_certs=self.ca_certs,
#            )

getsentry / raven-python

Sentry responded with an error: EOF occurred in violation of protocol #1109