SQLite ProgrammingError during make test

[alex-dot]

While reinstalling for my other issue (just making sure - my stomach tells me I ran into some module installation problems) I get the following error when running make test after make build. Seems to me that some resources in /usr/lib/python2.7/ are used which I guess is not desired.

thanks, a.

make test
# Basic syntax and sanity checks.
local/bin/flake8 ./syncserver
# Testcases from syncstorage app
local/bin/nosetests -s syncstorage.tests
..................................................................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS..SSSSSSSSSSSSSSSS...SS...............
----------------------------------------------------------------------
Ran 233 tests in 60.883s

OK (SKIP=147)
# Testcases from tokenserver app; broken due to incorrect file paths
#local/bin/nosetests -s tokenserver.tests
# Live tests against a running server.
./local/bin/pserve syncserver/tests.ini & SERVER_PID=$! ; sleep 2 ; ./local/bin/python -m syncstorage.tests.functional.test_storage --use-token-server http://localhost:5000/token/1.0/sync/1.5 ; kill $SERVER_PID
Starting server in PID 28043.
serving on 0.0.0.0:5000 view at http://127.0.0.1:5000
No handlers could be found for logger "sqlalchemy.pool.QueuePool"
Traceback (most recent call last):
  File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/path/to/syncserver/local/lib/python2.7/site-packages/syncstorage/tests/functional/test_storage.py", line 1270, in <module>
    res = run_live_functional_tests(TestStorage, sys.argv)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tests/functional/support.py", line 178, in run_live_functional_tests
    creds = authenticate_to_token_server(url, opts.email, opts.audience)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tests/functional/support.py", line 110, in authenticate_to_token_server
    r.raise_for_status()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/requests/models.py", line 765, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: Internal Server Error
----------------------------------------
Exception happened during processing of request from ('127.0.0.1', 47280)
Traceback (most recent call last):
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/paste/httpserver.py", line 1068, in process_request_in_thread
    self.finish_request(request, client_address)
  File "/usr/lib/python2.7/SocketServer.py", line 323, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/SocketServer.py", line 638, in __init__
    self.handle()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/paste/httpserver.py", line 442, in handle
    BaseHTTPRequestHandler.handle(self)
  File "/usr/lib/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/paste/httpserver.py", line 437, in handle_one_request
    self.wsgi_execute()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/paste/httpserver.py", line 287, in wsgi_execute
    self.wsgi_start_response)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/router.py", line 251, in __call__
    response = self.invoke_subrequest(request, use_tweens=True)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/router.py", line 227, in invoke_subrequest
    response = handle_request(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/tokenserver/tweens.py", line 15, in set_x_timestamp_header_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/mozsvc/tweens.py", line 115, in fuzz_backoff_headers_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/mozsvc/tweens.py", line 71, in log_uncaught_exceptions_tween
    return handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/mozsvc/tweens.py", line 37, in catch_backend_errors_tween
    return handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tweens.py", line 120, in convert_non_json_responses_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tweens.py", line 96, in convert_cornice_errors_to_respcodes_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tweens.py", line 52, in set_default_accept_header_tween
    return handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/syncstorage/tweens.py", line 29, in set_x_timestamp_header_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/tweens.py", line 21, in excview_tween
    response = handler(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/router.py", line 161, in handle_request
    response = view_callable(context, request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/config/views.py", line 588, in __call__
    return view(context, request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/config/views.py", line 321, in attr_view
    return view(context, request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/config/views.py", line 297, in predicate_wrapper
    return view(context, request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/config/views.py", line 347, in rendered_view
    result = view(context, request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/pyramid/config/views.py", line 493, in _requestonly_view
    response = view(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/cornice/service.py", line 514, in wrapper
    response = view_(request)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 154, in __call__
    return self._real_call(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 151, in _real_call
    return replacement(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/mozsvc/metrics.py", line 150, in metlog_call
    return self._fn(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 154, in __call__
    return self._real_call(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 151, in _real_call
    return replacement(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/stats.py", line 50, in metlog_call
    result = self._fn(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 154, in __call__
    return self._real_call(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/base.py", line 151, in _real_call
    return replacement(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/metlog/decorators/stats.py", line 36, in metlog_call
    return self._fn(*args, **kwargs)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/tokenserver/views.py", line 186, in return_token
    user = backend.get_user(service, email)
  File "/path/to/syncserver/syncserver/staticnode.py", line 110, in get_user
    res = self._engine.execute(_GET_USER_RECORDS, **params)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1598, in execute
    return connection.execute(statement, *multiparams, **params)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 664, in execute
    return meth(self, multiparams, params)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/sql/elements.py", line 282, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 761, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 828, in _execute_context
    None, None)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1013, in _handle_dbapi_exception
    self._autorollback()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 556, in _autorollback
    self._rollback_impl()
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 476, in _rollback_impl
    self._handle_dbapi_exception(e, None, None, None, None)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 991, in _handle_dbapi_exception
    exc_info
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 185, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 473, in _rollback_impl
    self.engine.dialect.do_rollback(self.connection)
  File "/path/to/syncserver/local/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 358, in do_rollback
    dbapi_connection.rollback()
ProgrammingError: (ProgrammingError) SQLite objects created in a thread can only be used in that same thread.The object was created in thread id 1078348544 and this is thread id 1089506112 None None

[rfk]

Ah, the syncstorage codebase has some special handling for this issue but it didn't make it into the syncserver repo. I've copied it across in https://github.com/mozilla-services/syncserver/commit/4b7cab5bd56a6dc5a1ada267cabe33e848682149, please try again.

(The right thing to do here is to factor this out into our shared utility library, but this quick fix should work in the meantime)

[alex-dot]

Yes, that worked smoothly. BTW, I had to manually install mysql-python through pip to make my setup work with MySQL. I don't think this is intended, is it?

Checking my other problem tomorrow.

Thanks!

[rfk]

BTW, I had to manually install mysql-python through pip to make my setup work with MySQL.

You should get PyMySQL by default, which you can use by specifying the sqluri like "pymysql://user:pass@host/db".