search

OpenDataNode / ckanext-odn-ic2pc-sync

CKAN Extenstion for synchronization of catalog records from internal catalog to public (external) catalog
GNU Affero General Public License v3.0
0 stars 0 forks source link

TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30. #14

Closed Jan-Marcek closed 8 years ago

Jan-Marcek commented 9 years ago

steps to reproduce: run https://github.com/OpenDataNode/ckanext-odn-ic2pc-sync/blob/develop/ckanext/tests/test.py (you has to set correct catalogs)

list of datasets:

["a-third-eco-data", "activity-areas", "agendas-of-institutions-of-public-power", "atc-groups", "bans-of-ctia", "businesses-medium-and-large-retailers-2013", "cadastre-offices", "code-lists-of-the-legal-relationships-and-documents", "comsode-datasets", "confiscations-of-ctia", "data-about-czech-business-entities-from-the-ares-system-trade-licensing-register", "divisions-of-statistical-classification-of-economic-activities-sk-nace-rev-2-2008", "elections-dataset", "employment-statistics", "focus-of-ctia-inspections", "infectious-and-contagious-diseases-by-months-year-2011", "inspections-of-ctia", "institutions-of-public-power", "integrated-pollution-registry", "international-roaming", "list-of-custody-procedures-for-procurement-of-works-services-and-supplies-2013", "maximum-prices-and-reimbursement-of-medicinal-products-marketed-in-czech-republic", "medicinal-product-active-ingredients-from-czech-sidc", "medicinal-products-marketed-in-czech-republic", "meteorological-data", "population-in-regions-of-cz", "prices-of-selected-retail-market-services-and-of-selected-wholesale-services", "projections-of-households-by-district-years-2013-2032-base-2012", "register-of-the-fixed-service-antennas", "register-of-the-fixed-service-equipment", "register-of-the-local-wireless-information-systems-access-points", "register-of-the-radio-relay-connectors", "sanctions-of-ctia", "schools-list", "sections-of-statistical-classification-of-economic-activities-sk-nace-rev-2-2008", "social-services-for-elder-people", "source-data-for-the-price-barometer", "state-inspection-and-penalties-statistics", "statistical-data-about-social-service-facilities-and-nursing-in-districts-of-czech-republic", "statistics-of-complaints-pursuant-to-the-act-no-5002004-coll", "test-3", "tourist-flow-in-2010"

less /var/log/apache2/odn-ckan-ic.error.log

[[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] Error - <class 'sqlalchemy.exc.TimeoutError'>: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] URL: https://test.comsode.eea.sk/internalcatalog/api/action/package_show
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/weberror/errormiddleware.py', line 162 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   app_iter = self.application(environ, sr_checker)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   resp = self.call_func(req, *args, **self.kwargs)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self.func(req, *args, **kwargs)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/fanstatic/publisher.py', line 234 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return request.get_response(self.app)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   application, catch_exc_info=False)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   app_iter = application(self.environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/dec.py', line 147 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   resp = self.call_func(req, *args, **self.kwargs)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/dec.py', line 208 in call_func
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self.func(req, *args, **kwargs)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/fanstatic/injector.py', line 54 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   response = request.get_response(self.app)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/request.py', line 1053 in get_response
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   application, catch_exc_info=False)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/webob/request.py', line 1022 in call_application
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   app_iter = application(self.environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/beaker/middleware.py', line 73 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self.app(environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/beaker/middleware.py', line 155 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self.wrap_app(environ, session_start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/routes/middleware.py', line 131 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   response = self.app(environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/pylons/wsgiapp.py', line 125 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   response = self.dispatch(controller, environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/pylons/wsgiapp.py', line 324 in dispatch
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return controller(environ, start_response)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/ckan/controllers/api.py', line 53 in __call__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   self._identify_user()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/ckan/lib/base.py', line 277 in _identify_user
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   self._identify_user_default()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/ckan/lib/base.py', line 327 in _identify_user_default
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   c.userobj = self._get_user_for_apikey()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/ckan/lib/base.py', line 419 in _get_user_for_apikey
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   user = query.filter_by(apikey=apikey).first()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2334 in first
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   ret = list(self[0:1])
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2201 in __getitem__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return list(res)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2405 in __iter__
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self._execute_and_instances(context)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2418 in _execute_and_instances
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   close_with_result=True)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/query.py', line 2409 in _connection_from_session
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   **kw)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/session.py', line 846 in connection
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   close_with_result=close_with_result)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/session.py', line 850 in _connection_for_bind
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return self.transaction._connection_for_bind(engine)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/orm/session.py', line 315 in _connection_for_bind
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   conn = bind.contextual_connect()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/engine/base.py', line 1737 in contextual_connect
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   self.pool.connect(),
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/pool.py', line 332 in connect
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   return _ConnectionFairy._checkout(self)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/pool.py', line 630 in _checkout
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   fairy = _ConnectionRecord.checkout(pool)
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/pool.py', line 433 in checkout
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   rec = pool._do_get()
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] File '/usr/share/python/odn-ckan-shared/lib/python2.7/site-packages/sqlalchemy/pool.py', line 945 in _do_get
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1]   (self.size(), self.overflow(), self._timeout))
[Wed May 20 10:39:05 2015] [error] [client 127.0.0.1] TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
Jan-Marcek commented 8 years ago

It appeared again on ODN A. When ODN A tried to synchronized ODN B. version: odn-ckan-shared: 2.3-odn1.1.1 ckanext-odn-ic2pc-sync: 0.6.1

TimeoutError: QueuePool limit of size 5 overflow 10 reached, connection timed out, timeout 30
[Thu Aug 20 08:54:20 2015] [error] 2015-08-20 08:54:20,999 ERROR [ckanext] error response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
[Thu Aug 20 08:54:20 2015] [error] <html><head>
[Thu Aug 20 08:54:20 2015] [error] <title>500 Internal Server Error</title>
[Thu Aug 20 08:54:20 2015] [error] </head><body>
[Thu Aug 20 08:54:20 2015] [error] <h1>Internal Server Error</h1>
[Thu Aug 20 08:54:20 2015] [error] <p>The server encountered an internal error or
[Thu Aug 20 08:54:20 2015] [error] misconfiguration and was unable to complete
[Thu Aug 20 08:54:20 2015] [error] your request.</p>
[Thu Aug 20 08:54:20 2015] [error] <p>Please contact the server administrator,
[Thu Aug 20 08:54:20 2015] [error]  [no address given] and inform them of the time the error occurred,
[Thu Aug 20 08:54:20 2015] [error] and anything you might have done that may have
[Thu Aug 20 08:54:20 2015] [error] caused the error.</p>
[Thu Aug 20 08:54:20 2015] [error] <p>More information about this error may be available
[Thu Aug 20 08:54:20 2015] [error] in the server error log.</p>
[Thu Aug 20 08:54:20 2015] [error] <hr>
[Thu Aug 20 08:54:20 2015] [error] <address>Apache/2.2.22 (Debian) Server at test.comsode.eea.sk Port 80</address>
[Thu Aug 20 08:54:20 2015] [error] </body></html>
[Thu Aug 20 08:54:20 2015] [error]
mvi-eea-sk commented 8 years ago

@Jan-Marcek i have problem replicating this problem

mvi-eea-sk commented 8 years ago

ready for testing