Prevent 500 errors when service is down

[andyzinsser]

This is only an issue when the API Ingestion is offline – which is hardly ever. When making any call using this library, the request hangs and then eventually times out.

It would be awesome if I could just call mixpanel.track() and not have to worry about whether or not the tracking event will cause our server to hang.

Example request

mixpanel = Mixpanel(MIXPANEL_KEY)
mixpanel.track(tracking_id, 'Test Tracking')

Stack trace

2015-01-28T22:56:21.775103+00:00 app[web.2]:   File "/app/accounts/views.py", line 279, in post
2015-01-28T22:56:21.775104+00:00 app[web.2]:     mp.track(tracking_id, 'Test Tracking')
2015-01-28T22:56:21.775099+00:00 app[web.2]:     response = self.handle_exception(exc)
2015-01-28T22:56:21.775094+00:00 app[web.2]:     return wrapped(*args, **kwargs)
2015-01-28T22:56:21.775105+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/site-packages/mixpanel/__init__.py", line 74, in track
2015-01-28T22:56:21.775106+00:00 app[web.2]:     self._consumer.send('events', json.dumps(event, separators=(',', ':')))
2015-01-28T22:56:21.775108+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/site-packages/mixpanel/__init__.py", line 336, in send
2015-01-28T22:56:21.775110+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/site-packages/mixpanel/__init__.py", line 357, in _write_request
2015-01-28T22:56:21.775109+00:00 app[web.2]:     self._write_request(self._endpoints[endpoint], json_message, api_key)
2015-01-28T22:56:21.775113+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 127, in urlopen
2015-01-28T22:56:21.775114+00:00 app[web.2]:     return _opener.open(url, data, timeout)
2015-01-28T22:56:21.775111+00:00 app[web.2]:     response = urllib2.urlopen(request).read()
2015-01-28T22:56:21.775122+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/site-packages/newrelic-2.30.0.27/newrelic/hooks/external_urllib2.py", line 15, in _nr_wrapper_opener_director_open_
2015-01-28T22:56:21.775125+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 404, in open
2015-01-28T22:56:21.775126+00:00 app[web.2]:     response = self._open(req, data)
2015-01-28T22:56:21.775123+00:00 app[web.2]:     return wrapped(*args, **kwargs)
2015-01-28T22:56:21.775127+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 422, in _open
2015-01-28T22:56:21.775130+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 382, in _call_chain
2015-01-28T22:56:21.775129+00:00 app[web.2]:     '_open', req)
2015-01-28T22:56:21.775131+00:00 app[web.2]:     result = func(*args)
2015-01-28T22:56:21.775132+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1222, in https_open
2015-01-28T22:56:21.775134+00:00 app[web.2]:     return self.do_open(httplib.HTTPSConnection, req)
2015-01-28T22:56:21.775135+00:00 app[web.2]:   File "/app/.heroku/python/lib/python2.7/urllib2.py", line 1184, in do_open
2015-01-28T22:56:21.775136+00:00 app[web.2]:     raise URLError(err)
2015-01-28T22:56:21.775138+00:00 app[web.2]: URLError: <urlopen error [Errno 104] Connection reset by peer>

[joeatwork]

This makes sense- perhaps we could have a setting on construction ("Either throw exceptions or just log errors")

[lkiss80]

We recently ran into this issue as well. Should there also be a configurable timeout, so we have control over how long we want to wait before gracefully moving on?

[joeatwork]

For timeouts, the answer is probably to use a custom consumer, so you can use your own settings (and maybe even your own library) at the HTTP client level- that means the fix for this should probably also have a demo of how that should be done.

In the mean time, that's probably the workaround for folks concerned about this issue - you can provide your own Consumer to your tracker, which can wrap the existing consumer, perhaps something like this

class LoggingFailuresConsumer(mixpanel.Consumer):
    def send(self, *args, **kwargs):
        try:
             super(LoggingFailuresConsumer, self).send(*args, **kwargs)
        except:
            logAboutMixpanelErrors("Tracking failed", e)

[ivy-derm]

Just ran into this issue yesterday because MixPanel was down again yesterday and brought our entire app down with it. Surprised to see this has been an open issue since 2015.

[jarekwg]

Our app stayed up, but we ended up chewing through a lot of Sentry event quota because of all these exceptions. Maybe raise the exception just once instead of for every single failure, or handle the problem in some other, more graceful manner?

[dacevedo12]

For real, it is valuable to set a reasonable default timeout

[Culpable]

I've also been unable to debug my Python app when not connected to the internet for the same reason:

mixpanel.MixpanelException
mixpanel.MixpanelException: HTTPSConnectionPool(host='api.mixpanel.com', port=443): Max retries exceeded with url: /engage (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x16bafd9a0>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Can we please have an easy way to stop Mixpanel from preventing the app from starting when it can't connect to the internet?

Apologies if there's an easy solution to this that I'm missing.