search

FirebirdSQL / python3-driver

Firebird driver for Python that uses new Firebird API
https://www.firebirdsql.org/en/devel-python-driver/
MIT License
26 stars 10 forks source link

SIGABRT on interface detach #33

Closed Fred-si closed 7 months ago

Fred-si commented 7 months ago

Hello. I work on project that use SQLAlchemy+firebird. We have recently migrate from fdb to firebird-driver python package and we experience some python crashes with SIGABRT in our test suite.

These crashes happen after introduce little modification on code/test that seem to have no relation with test that crash. Sometime, the modified code is not executed before crash. Crashes can happen at different places in tests suite, during a database operation or not, but always at the place will code not change. They can happen in Gitlab runner or locally in venv.

Fatal Python error: Aborted

Current thread 0x00007fd5d3a7d000 (most recent call first):
  Garbage-collecting
  File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/interfaces.py", line 1179 in detach
  File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1645 in __del__
  File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1145 in __init__
  File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1545 in __init__
  File "/project_root/venv/lib/python3.11/site-packages/firebird/driver/core.py", line 1935 in info
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy_firebird/firebird.py", line 147 in _get_server_version_info
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/default.py", line 514 in initialize
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy_firebird/base.py", line 428 in initialize
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 732 in first_connect
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/util/langhelpers.py", line 1910 in go
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 487 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/event/attr.py", line 473 in _exec_w_sync_on_first_run
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 916 in __connect
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 678 in __init__
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 393 in _create_connection
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/impl.py", line 167 in _do_get
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 716 in checkout
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 1269 in _checkout
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/pool/base.py", line 452 in connect
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3292 in raw_connection
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 145 in __init__
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/engine/base.py", line 3268 in connect
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 1143 in _connection_for_bind
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/state_changes.py", line 139 in _go
  File "<string>", line 2 in _connection_for_bind
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2045 in _connection_for_bind
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2178 in _execute_internal
  File "/project_root/venv/lib/python3.11/site-packages/sqlalchemy/orm/session.py", line 2306 in execute
  File "/project_root/tests/utils/abstract_test_base.py", line 63 in _clean_database
  File "/project_root/tests/utils/abstract_test_base.py", line 40 in setup_class
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 777 in _call_with_optional_argument
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 856 in xunit_setup_class_fixture
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 895 in call_fixture_func
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1123 in pytest_fixture_setup
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 1069 in execute
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 693 in _compute_fixture_value
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 607 in _get_active_fixturedef
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 585 in getfixturevalue
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/fixtures.py", line 566 in _fillfixtures
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/python.py", line 1795 in setup
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 494 in setup
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 157 in pytest_runtest_setup
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 127 in runtestprotocol
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/project_root/venv/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/project_root/venv/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/project_root/venv/bin/pytest", line 8 in <module>

Extension modules: _cffi_backend, charset_normalizer.md, simplejson._speedups, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, yaml._yaml, markupsafe._speedups, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, dependency_injector.providers, dependency_injector._cwiring, dependency_injector.containers, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, google._upb._message (total: 97)
zsh: IOT instruction (core dumped)  pytest


Fatal Python error: Aborted
Current thread 0x00007f509adb9b80 (most recent call first):
  Garbage-collecting
  File "/usr/local/lib/python3.11/site-packages/firebird/driver/interfaces.py", line 1179 in detach
  File "/usr/local/lib/python3.11/site-packages/firebird/driver/core.py", line 1645 in __del__
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2167 in lr0_goto
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2334 in reads_relation
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2454 in <lambda>
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2074 in traverse
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2065 in digraph
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2455 in compute_read_sets
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2518 in add_lalr_lookaheads
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2551 in lr_parse_table
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 2135 in __init__
  File "/usr/local/lib/python3.11/site-packages/ply/yacc.py", line 3438 in yacc
  File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 47 in parse_token_stream
  File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 32 in parse
  File "/usr/local/lib/python3.11/site-packages/jsonpath_rw/parser.py", line 14 in parse
  File "/builds/a6cmo/agde6_support/shared/services/bodacc/helpers.py", line 12 in nested_get
  File "/builds/a6cmo/agde6_support/shared/services/bodacc/pcl_service.py", line 160 in _create_evt_jur
  File "/builds/a6cmo/agde6_support/shared/services/bodacc/pcl_service.py", line 89 in _import_data
  File "/builds/a6cmo/agde6_support/shared/services/bodacc/bodacc_service.py", line 210 in import_files
  File "/builds/a6cmo/agde6_support/tests/services/bodacc/test_pcl_service.py", line 343 in test_import_creation_evt_existe_deja
  File "/usr/local/lib/python3.11/unittest/mock.py", line 1375 in patched
  File "/usr/local/lib/python3.11/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/usr/local/lib/python3.11/site-packages/_pytest/python.py", line 1792 in runtest
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 341 in from_call
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/usr/local/lib/python3.11/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 325 in _main
  File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/usr/local/lib/python3.11/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/usr/local/lib/python3.11/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/usr/local/lib/python3.11/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/usr/local/lib/python3.11/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/usr/local/lib/python3.11/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/usr/local/bin/pytest", line 8 in <module>
Extension modules: _cffi_backend, charset_normalizer.md, simplejson._speedups, multidict._multidict, yarl._quoting_c, aiohttp._helpers, aiohttp._http_writer, aiohttp._http_parser, aiohttp._websocket, frozenlist._frozenlist, yaml._yaml, markupsafe._speedups, pydantic.typing, pydantic.errors, pydantic.version, pydantic.utils, pydantic.class_validators, pydantic.config, pydantic.color, pydantic.datetime_parse, pydantic.validators, pydantic.networks, pydantic.types, pydantic.json, pydantic.error_wrappers, pydantic.fields, pydantic.parse, pydantic.schema, pydantic.main, pydantic.dataclasses, pydantic.annotated_types, pydantic.decorator, pydantic.env_settings, pydantic.tools, pydantic, dependency_injector.providers, dependency_injector._cwiring, dependency_injector.containers, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, google._upb._message (total: 97)
/usr/bin/bash: line 151:    18 Aborted                 (core dumped) pytest -p no:warnings --cov --cov-report term --cov-report html

The common points on all crashes is that happen on detach method of iAttachment_v3 on interface.py.

    def detach(self) -> None:
        "Replaces `isc_detach_database()`. On success releases interface."
        self.vtable.deprecatedDetach(self, self.status)
        self._check()
        self._refcnt -= 1

Adding if condition before vtable.deprecatedDetach seems to solve the problem, but I have no experience on database driver development, I don't know if it’s good idea.

    def detach(self) -> None:
        "Replaces `isc_detach_database()`. On success releases interface."
        if self._refcnt:
            self.vtable.deprecatedDetach(self, self.status)
            self._check()
            self._refcnt -= 1

Someone might be able to help us solve this problem properly?

pcisar commented 7 months ago

First, could you please provide information about: firebird-driver version, Python version and Firebird version?

Second, the error happens in __del__ destructor. This is always very tricky to debug as the condition when this code is executed is not deterministic (hence it's not easy if even possible to create reliable reproducible test case). The general recommendation is to always close resources such as Connection explicitly (or use it as context manager) to avoid situations when object is released by memory manager.

But in this case, it could be something more subtle, as the culprit seems to be the broken interface reference counting (which couldn't happen in normal circumstances).

Fred-si commented 7 months ago

Hello

project run on python 3.11, firebird-driver is in 1.10.0 and SQLAlchemy in version 2.0.22, the project use also sqlalchemy-firebird in version 2.0.2. Firebird server is jacobalberty/firebird:v4.0.2 docker image. Firebird client lib can be in 4.0.2 when project run in docker or 3.08 when used in venv (provided by ubuntu firebird package).

firebird-driver v1.8.0 does not seem to be affected, but I’m not sure. firebird-driver v1.9.0 and v1.8.0 are also affected

pcisar commented 7 months ago

I see. I think that problem could come from use of 3.0.8 Firebird client library to access the v4.0.2 server.

Sure, this should be still considered as bug (although probably not in firebird-driver, I don't see how driver could be a culprit here), as this shouldn't happen even if older client is used to access newer server. But these cross-version scenarios are not part of testing (neither Firebird QA or firebird-driver), so it's possible that some backward compatibility issues could get unnoticed.

Could you please verify whether this issue goes away with 4.0.2 client lib vs. 4.0.2 server?

Fred-si commented 7 months ago

No, issue still occurs when using 4.0.2 client lib with 4.0.2 server.

pcisar commented 7 months ago

Interesting. What really puzzles me is that it fails in deprecatedDetach i.e. detach in IAttachment version 3. This interface was changed to version 4 and then to 5 where detach and dropDatabase are redefined (hence the old detach become deprecatedDetach). This change to new interface version happened in 3.0.7 => 3.0.8, 4.0.0 => 4.0.1, so neither your version 4.0.2 or 3.0.8 should use this old interface version.

Anyway, the Connection.__del__ is a tricky part, so your proposed fix is good as any other, so I'll adopt it.

Fred-si commented 7 months ago

Ok, thanks a lot for your help :pray: