AttributeError: 'NoneType' object has no attribute 'query'

[msrb]

2017-06-08 18:23:53,618 - Scheduling #50408.
INFO:f8a_jobs.handlers.base:Scheduling #50408.
2017-06-08 18:23:53,618 - Scheduling io.mraa.upm:upm_ds1808lc/1.3.0
DEBUG:f8a_jobs.handlers.base:Scheduling io.mraa.upm:upm_ds1808lc/1.3.0
2017-06-08 18:23:53,618 - Scheduling Selinon flow 'bayesianFlow' with node_args: '{'name': 'io.mraa.upm:upm_ds1808lc', 'force_graph_sync': False, 'force': False, 'ecosystem': 'maven', 'recursive_limit': 0, 'version': '1.3.0'}'
DEBUG:f8a_jobs.handlers.base:Scheduling Selinon flow 'bayesianFlow' with node_args: '{'name': 'io.mraa.upm:upm_ds1808lc', 'force_graph_sync': False, 'force': False, 'ecosystem': 'maven', 'recursive_limit': 0, 'version': '1.3.0'}'
ERROR:root:Job 'None' failed, registering ErrorHandler: 'NoneType' object has no attribute 'query'
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/f8a_jobs/scheduler.py", line 190, in job_execute
    instance.execute(**handler_kwargs)
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/base.py", line 264, in execute
    return self.do_execute(popular)
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/maven_popular_analyses.py", line 174, in do_execute
    self._use_maven_index_checker()
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/maven_popular_analyses.py", line 136, in _use_maven_index_checker
    if package_postgres.get_analysis_count(self.ecosystem, name) > 0:
  File "/usr/lib/python3.4/site-packages/cucoslib/storages/package_postgres.py", line 59, in get_analysis_count
    count = PostgresBase.session.query(PackageAnalysis).\
AttributeError: 'NoneType' object has no attribute 'query'
[pid: 1|app: 0|req: 63/63] 10.130.2.1 () {30 vars in 382 bytes} [Thu Jun 8 18:24:41 2017] GET /api/v1/readiness => generated 3 bytes in 1 msecs (HTTP/1.1 200) 2 headers in 70 bytes (1 switches on core 0)

[msrb]

This happens in jobs service, but only from time to time.

[msrb]

worker: d6111a622f64864fcfbab7719a09cb43188708e2 jobs: c2f809105a85f08677a8392727cee525536be313

[msrb]

Different error, but I guess the culprit will be the same:

ERROR:root:Job 'None' failed, registering ErrorHandler: (sqlalchemy.exc.ResourceClosedError) This Connection is closed [SQL: 'SELECT count(*) AS count_1 \nFROM (SELECT package_analyses.id AS package_analyses_id, package_analyses.package_id AS package_analyses_package_id, package_analyses.started_at AS package_analyses_started_at, package_analyses.finished_at AS package_analyses_finished_at \nFROM package_analyses JOIN packages ON packages.id = package_analyses.package_id JOIN ecosystems ON ecosystems.id = packages.ecosystem_id \nWHERE ecosystems.name = %(name_1)s AND packages.name = %(name_2)s) AS anon_1'] [parameters: [{}]]
Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1112, in _execute_context
    conn = self.__connection
AttributeError: 'Connection' object has no attribute '_Connection__connection'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1114, in _execute_context
    conn = self._revalidate_connection()
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection
    raise exc.ResourceClosedError("This Connection is closed")
sqlalchemy.exc.ResourceClosedError: This Connection is closed

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/f8a_jobs/scheduler.py", line 190, in job_execute
    instance.execute(**handler_kwargs)
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/base.py", line 264, in execute
    return self.do_execute(popular)
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/maven_popular_analyses.py", line 174, in do_execute
    self._use_maven_index_checker()
  File "/usr/lib/python3.4/site-packages/f8a_jobs/handlers/maven_popular_analyses.py", line 136, in _use_maven_index_checker
    if package_postgres.get_analysis_count(self.ecosystem, name) > 0:
  File "/usr/lib/python3.4/site-packages/cucoslib/storages/package_postgres.py", line 62, in get_analysis_count
    filter(Package.name == package).\
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 3029, in count
    return self.from_self(col).scalar()
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2783, in scalar
    ret = self.one()
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2754, in one
    ret = self.one_or_none()
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2724, in one_or_none
    ret = list(self)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2795, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/orm/query.py", line 2818, in _execute_and_instances
    result = conn.execute(querycontext.statement, self._params)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 945, in execute
    return meth(self, multiparams, params)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/sql/elements.py", line 263, in _execute_on_connection
    return connection._execute_clauseelement(self, multiparams, params)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1053, in _execute_clauseelement
    compiled_sql, distilled_params
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1121, in _execute_context
    None, None)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1393, in _handle_dbapi_exception
    exc_info
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb, cause=cause)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/util/compat.py", line 185, in reraise
    raise value.with_traceback(tb)
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 1114, in _execute_context
    conn = self._revalidate_connection()
  File "/usr/lib64/python3.4/site-packages/sqlalchemy/engine/base.py", line 429, in _revalidate_connection
    raise exc.ResourceClosedError("This Connection is closed")
sqlalchemy.exc.StatementError: (sqlalchemy.exc.ResourceClosedError) This Connection is closed [SQL: 'SELECT count(*) AS count_1 \nFROM (SELECT package_analyses.id AS package_analyses_id, package_analyses.package_id AS package_analyses_package_id, package_analyses.started_at AS package_analyses_started_at, package_analyses.finished_at AS package_analyses_finished_at \nFROM package_analyses JOIN packages ON packages.id = package_analyses.package_id JOIN ecosystems ON ecosystems.id = packages.ecosystem_id \nWHERE ecosystems.name = %(name_1)s AND packages.name = %(name_2)s) AS anon_1'] [parameters: [{}]]

[fridex]

I think both issues are likely to be caused by concurrency - we run a single-threaded single process worker instance - that's why we don't see these in worker. Jobs on the other hand is multi-threaded single process application so guards/context management should be used when it comes to resources.

[pkajaba]

I created https://github.com/fabric8-analytics/fabric8-analytics-jobs/pull/56

I am not sure whether it fixes the problem completely but it might help. Let's see whether this problem occurs even after merging

[msrb]

Just for the record, workaround from https://github.com/fabric8-analytics/fabric8-analytics-jobs/pull/58 did not help. In fact, the error started occurring much more often.

[fridex]

Let's keep this open until we are sure about the resolution.

[fridex]

AttributeError: 'NoneType' object has no attribute 'query'

What I think might cause the issue - we initialize Selinon each time when a jobs handler gets invoked. This results in generating new Python code from YAML config files and thus we lose the original storage adapter instances that are held by Selinon (in the configuration file). After that Python invokes destructor which frees our session variable. Note that the session is a class variable - that means it is freed if BayesianPostgres or PackagePostgres are cleaned, also note that this destructor invokes disconnect() method.

We should initialize Selinon only once on jobs start up so we do not overwrite code every time with each request.

[fridex]

Closing this as it should be fixed by #57, #65. Feel free to reopen if this issue persists.