Yelp / Testify

A more pythonic testing framework.
Other
306 stars 67 forks source link

testify doesn't deal well with network congestion #248

Closed bukzor closed 10 years ago

bukzor commented 10 years ago

If we try to start up a testify server/reporter while the network is under heavy load, we often see the below error. A simple backoff-retry loop around the connect() should be a Good Enough solution.

Traceback (most recent call last):
  File "/usr/lib/python2.6/runpy.py", line 122, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.6/runpy.py", line 34, in _run_code
    exec code in run_globals
  File "/usr/lib/python2.6/dist-packages/testify/test_program.py", line 343, in <module>
    main()
  File "/usr/lib/python2.6/dist-packages/testify/test_program.py", line 339, in main
    sys.exit(not TestProgram().run())
  File "/usr/lib/python2.6/dist-packages/testify/test_program.py", line 270, in run
    self.other_opts, self.test_runner_args['plugin_modules']
  File "/usr/lib/python2.6/dist-packages/testify/test_program.py", line 236, in get_reporters
    reporters += plugin.build_test_reporters(options)
  File "/usr/lib/python2.6/dist-packages/testify/plugins/sql_reporter.py", line 329, in build_test_reporters
    return [SQLReporter(options)]
  File "/usr/lib/python2.6/dist-packages/testify/plugins/sql_reporter.py", line 54, in __init__
    self.conn = self.engine.connect()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 2316, in connect
    return self._connection_cls(self, **kwargs)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 872, in __init__
    self.__connection = connection or engine.raw_connection()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/base.py", line 2402, in raw_connection
    return self.pool.unique_connection()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 169, in unique_connection
    return _ConnectionFairy(self).checkout()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 371, in __init__
    rec = self._connection_record = pool._do_get()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 758, in _do_get
    return self._create_connection()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 174, in _create_connection
    return _ConnectionRecord(self)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 256, in __init__
    self.connection = self.__connect()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/pool.py", line 316, in __connect
    connection = self.__pool._creator()
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/strategies.py", line 80, in connect
    return dialect.connect(*cargs, **cparams)
  File "/usr/lib/pymodules/python2.6/sqlalchemy/engine/default.py", line 280, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/usr/lib/python2.6/dist-packages/MySQLdb/__init__.py", line 81, in Connect
    return Connection(*args, **kwargs)
  File "/usr/lib/python2.6/dist-packages/MySQLdb/connections.py", line 193, in __init__
    super(Connection, self).__init__(*args, **kwargs2)
sqlalchemy.exc.OperationalError: (OperationalError) (2013, "Lost connection to MySQL server at 'reading authorization packet', system error: 0") None None
bukzor commented 10 years ago

Closed by #249