search

pymssql / pymssql

Official home for the pymssql source code.
https://pymssql.readthedocs.io/en/latest/
GNU Lesser General Public License v2.1
836 stars 310 forks source link

Symbol not found in flat namespace '_bcp_batch' causes an ImportError #822

Open monocongo opened 1 year ago

monocongo commented 1 year ago

Describe the bug I have code that imports pymssql. When I run this code I get the following import error indicating that a BCP symbol is missing:

% python -m poetry run pytest
========================================================= test session starts ==========================================================
platform darwin -- Python 3.11.4, pytest-7.4.0, pluggy-1.2.0
rootdir: /Users/jadams/git/relay-data-jobs
plugins: requests-mock-1.11.0
collected 41 items / 1 error

================================================================ ERRORS ================================================================
__________________________________________ ERROR collecting util_libs/tests/test_sql_utils.py __________________________________________
ImportError while importing test module '/Users/jadams/git/relay-data-jobs/util_libs/tests/test_sql_utils.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../miniconda3/envs/mpw680/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
tests/test_sql_utils.py:5: in <module>
    from util_libs.sql.sql_connector import SQLConnector
src/util_libs/sql/sql_connector.py:8: in <module>
    import pymssql
../../../miniconda3/envs/mpw680/lib/python3.11/site-packages/pymssql/__init__.py:3: in <module>
    from ._pymssql import *
src/pymssql/_pymssql.pyx:1: in init pymssql._pymssql
    ???
E   ImportError: dlopen(/Users/jadams/miniconda3/envs/mpw680/lib/python3.11/site-packages/pymssql/_mssql.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'

The project's pyproject.toml file includes the latest version of pymssql in its list of dependencies:

[tool.poetry.dependencies]
python = '>=3.8'
boto3 = '>=1.24.58'
dask = '>=2023.5.0'
pandas = '>=1.5.2'
paramiko = '>=2.12.0'
psycopg2_binary = '>=2.9.5'
pymssql = '>=2.2.7'
pyodbc = '>=4.0.39'
pytest = '>=7.2.0'
pytz = '>=2022.7'
requests = '>=2.22.0'
requests_mock = '>=1.10.0'
SQLAlchemy = '=1.4.46'
tenacity = '>=8.2.2'

To Reproduce Since the code I'm seeing this happen in is proprietary I can't post that here. But the issue happens at the time of import, at least that's what it looks like to me. Starting with a fresh clone of the repository I am doing these steps to get the above error:

conda create -n myvenv python poetry
conda activate myvenv
python -m poetry install
python -m poetry run pytest

Expected behavior No import error

Current Behavior Please see the above description of the error.

Context (Environment)

==> freetds: stable 1.3.18 (bottled), HEAD Libraries to talk to Microsoft SQL Server and Sybase databases https://www.freetds.org/ Not installed From: https://github.com/Homebrew/homebrew-core/blob/HEAD/Formula/freetds.rb License: GPL-2.0-or-later ==> Dependencies Build: pkg-config ✔ Required: openssl@3 ✔, unixodbc ✔ ==> Options --HEAD Install HEAD version ==> Analytics install: 3,673 (30 days), 3,673 (90 days), 3,673 (365 days) install-on-request: 0 (30 days), 0 (90 days), 0 (365 days) build-error: 0 (30 days)

 - please include the output of the following command:
    `/<path to your python>/python -c "import pymssql; print(pymssql.version_info())"`

% python -c "import pymssql; print(pymssql.version_info())" Traceback (most recent call last): File "", line 1, in File "/Users/jadams/miniconda3/envs/mpw680/lib/python3.11/site-packages/pymssql/init.py", line 3, in from ._pymssql import * File "src/pymssql/_pymssql.pyx", line 1, in init pymssql._pymssql ImportError: dlopen(/Users/jadams/miniconda3/envs/mpw680/lib/python3.11/site-packages/pymssql/_mssql.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'



**Additional context**
Is there a way to enforce the inclusion of BCP via dependencies in pyproject.toml? Or is BCP assumed to be present on any system? I will need this code to run via GitHub Action jobs and with other containers in Airflow, so maybe there is a requirement to include a BCP-related package in the Linux environment for the container when it's built via Dockerfile or part of the GitHub Action's workflow?

I appreciate any help on this issue and the effort to keep this package alive. Thank you!
monocongo commented 1 year ago

This shows that the package is included in the poetry packages:

% python -m poetry show pymssql
 name         : pymssql
 version      : 2.2.7
 description  : DB-API interface to Microsoft SQL Server for Python. (new Cython-based version)
edgarrmondragon commented 1 year ago

I can reproduce this with

$ docker run -d \
 -p "1433:1433" \
 -e "SA_PASSWORD=Meltan0admin" \
 -e "ACCEPT_EULA=Y" \
 --name mssql \
 --health-cmd "/opt/mssql-tools/bin/sqlcmd -U sa -P Meltan0admin -Q 'select 1' -b -o /dev/null" \
 --health-interval 10s \
 --health-timeout 5s \
 --health-retries 5 \
 mcr.microsoft.com/mssql/server:2019-latest

and

>>> import sqlalchemy as sa
>>> from sqlalchemy.engine import URL
>>> url = URL.create(
...     "mssql+pymssql",
...     username="sa",
...     password="Meltan0admin",
...     host="localhost",
...     port=1433,
...     database="wh",
...     query={"charset": "utf8"},
... )
>>> engine = sa.create_engine(url)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "<string>", line 2, in create_engine
  File "/path/to/.venv/lib/python3.11/site-packages/sqlalchemy/util/deprecations.py", line 281, in warned
    return fn(*args, **kwargs)  # type: ignore[no-any-return]
           ^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.11/site-packages/sqlalchemy/engine/create.py", line 601, in create_engine
    dbapi = dbapi_meth(**dbapi_args)
            ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.11/site-packages/sqlalchemy/dialects/mssql/pymssql.py", line 65, in import_dbapi
    module = __import__("pymssql")
             ^^^^^^^^^^^^^^^^^^^^^
  File "/path/to/.venv/lib/python3.11/site-packages/pymssql/__init__.py", line 3, in <module>
    from ._pymssql import *
  File "src/pymssql/_pymssql.pyx", line 1, in init pymssql._pymssql
ImportError: dlopen(/path/to/.venv/lib/python3.11/site-packages/pymssql/_mssql.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'
kiilivi commented 11 months ago

I get this same error using Python 3.11.6 in MacOS. I don't use Python directly, I use it with Robot Framework and I get this error when I try to connect to SQL Server database using Robot Framework DatabaseLibrary method Connect To Database with pymssql as a first parameter. https://marketsquare.github.io/Robotframework-Database-Library/#Connect%20To%20Database

Test The Database Connection                                          | FAIL |
ImportError: dlopen(/Users/vmkiili/.pyenv/versions/3.11.6/lib/python3.11/site-packages/pymssql/_mssql.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'

It works fine in Windows environment. I don't have any problems with pymssql in MacOS if I use Python 3.10.13.

er-eis commented 9 months ago

MacOS Sonoma 14.2 Apple M2 Max Python 3.11.6 pymssql 2.2.11, 2.2.10, 2.2.9, 2.2.8, 2.2.7, 2.2.6, 2.1.5

all same behavior: 

ImportError: dlopen(/venv/lib/python3.11/site-packages/pymssql/_mssql.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_bcp_batch'
eshwen commented 9 months ago

To add to the momentum, I've been having this issue ever since attempting to switch from Python 3.10 to 3.11. This is the only blocker preventing me from bumping Python above 3.10

System:

(edit: typo in pymssql version)

termim commented 9 months ago

I think that the problem is similar to what is described here. Try to compile pymssql from source using hint from #769

eshwen commented 9 months ago

I think that the problem is similar to what is described here. Try to compile pymssql from source using hint from #769

Thanks, this does seem to work for me. With the following setup

I added this to the poetry.toml file in my project:

[installer]
no-binary = ["pymssql"]

After rebuilding my Poetry environment, I can connect to SQL Server and run queries without issue

vinicinbgs commented 7 months ago

Another way to fix it is using the following command:

pip install --pre --no-binary :all: pymssql==2.2.11 --no-cache --force