Cannot import `pyodbc` with latest version on Apple Silicon

[nleroy917]

Environment

To diagnose, we usually need to know the following, including version numbers. On Windows, be sure to specify 32-bit Python or 64-bit:

• Python: 3.10.6
• pyodbc: 4.0.35
• OS: macOS Ventura (13.0)
• DB: N/A
• driver: N/A

Issue

overview: I upgraded my pyodbc version to the latest on PyPi (v4.0.35) and noticed I was getting a new error that I had never dealt with before. Specifically, when importing pyodbc I receive the following:

(0x0002): symbol not found in flat namespace '_SQLAllocHandle'

An issue for this was raised and closed a while ago; it references this exact problem: #885

expected behavior: pyodbc should import without issue

steps to reproduce:

1. Create new virtual environment and activate:

   python -m venv .venv
   source .venv/bin/activate

2. Install latest pyodbc:

   pip install pyodbc

3. Import pyodbc and observe error:

   python
   >>> import pyodbc
   >>> (0x0002): symbol not found in flat namespace '_SQLAllocHandle'

[alexandriapawlik]

@mrluthra What specifically did you do in the rosetta terminal?

[dsantiago]

@anibal2j do you have unixodbc installed (brew install unixodbc)? If yes, try ``` export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/your-version/lib" export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/your-version/include" pip install --no-binary :all: pyodbc

This 👆🏻 solved for me!

[amit-chetwood]

Having issues with --no-binary :all: pyodbc I get errors to do with setup.py and so I tried using --use-pep517 but still get symbol not found in flat namespace '_SQLAllocHandle'. Please help, this issue is destroying me

[v-chojas]

Do you have unixodbc installed?

[amit-chetwood]

yes I have UNIXODBC version 2.3.11

[v-chojas]

Make sure you have arm64 version of it, and did you specify its location as shown above?

[zaphodnothingth]

i ran into this issue on mac M1 & fixed with conda install ilo pip

[amit-chetwood]

Make sure you have arm64 version of it, and did you specify its location as shown above?

yep i did this

i ran into this issue on mac M1 & fixed with conda install ilo pip

what were your steps specifically please?

[wasiba100]

Ok this works for me, Mac M2, Ventura.

1. brew install unixodbc
2. pip uninstall pyodbc
3. pip install --no-binary :all: pyodbc
4. install drivers for 18 (copy and paste bash code), https://learn.microsoft.com/en-us/sql/connect/odbc/linux-mac/install-microsoft-odbc-driver-sql-server-macos?view=sql-server-ver15
5. run python code and get connected

  import pyodbc 
  server = 'xxxxx.database.windows.net' 
  database = 'xxx' 
  username = 'xxx@xx.x.com' 
  password = 'xxxxx' 
  cnxn = pyodbc.connect('DRIVER={ODBC Driver 18 for SQL Server};SERVER='+server+';DATABASE='+database+';ENCRYPT=yes;UID='+username+';PWD='+ password )
  cursor = cnxn.cursor()

THX !!!!!!!

[MuhammedBuyukkinaci]

I also encountered the same problem. I am using Python 3.11 and Mac M1 Pro. pyodbc version 4.0.35 works fine in a dockerized container. pyodbc version 4.0.34 was also working fine with Python 3.10.

The problem is related to OS, which is Mac.

To overcome the problem,

1- Activate your virtual environment

2- Remove non-working pyodbc installation via pip uninstall pyodbc

3- Install pyodbc with non-binary option via pip install --no-binary :all: pyodbc==4.0.35.

[mkleehammer]

I have added Apple Silicon wheels to 5.0.0b2. Can someone please try installing them with just pip install pyodbc==5.0.0b2? Thanks.

In the past Github actions, which we use to build the wheels, did not support macOS ARM very well. Github is not finished, but it looks like there might be enough support now.

[keitherskine]

Use only wheels for the installation by running python -m pip install --only-binary=:all: pyodbc==5.0.0b2

[airdrik]

I have added Apple Silicon wheels to 5.0.0b2. Can someone please try installing them with just pip install pyodbc==5.0.0b2? Thanks.

In the past Github actions, which we use to build the wheels, did not support macOS ARM very well. Github is not finished, but it looks like there might be enough support now.

I just tried this on my Apple M2 Pro, Ventura, python 3.9.16 (installed using asdf-vm):

$ python -m venv pyodbc_beta
$ source pyodbc_beta/bin/activate
$ pip install pyodbc==5.0.0b2
$ python
>>> import pyodbc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(pyodbc_beta/lib/python3.9/site-packages/pyodbc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'
$ echo $LDFLAGS
-L/opt/homebrew/Cellar/unixodbc/2.3.11/lib
$ echo $CPPFLAGS
-L/opt/homebrew/Cellar/unixodbc/2.3.11/include
$ ls /opt/homebrew/Cellar/unixodbc/2.3.11
AUTHORS         ChangeLog       NEWS            bin         lib
COPYING         INSTALL_RECEIPT.json    README          include         share

edit: installing using pip install --no-binary :all: pyodbc==5.0.0b2 seems to work just fine.

[falkben]

$ python --version                                          
Python 3.11.4
$ pip install pyodbc==5.0.0b2              
Collecting pyodbc==5.0.0b2
  Obtaining dependency information for pyodbc==5.0.0b2 from https://files.pythonhosted.org/packages/4f/bc/9eee7a7f79f1ebdd85a4de28e57b922e61664500e63a9fc4ce979a399c51/pyodbc-5.0.0b2-cp311-cp311-macosx_11_0_arm64.whl.metadata
  Downloading pyodbc-5.0.0b2-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.7 kB)
Downloading pyodbc-5.0.0b2-cp311-cp311-macosx_11_0_arm64.whl (71 kB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 71.8/71.8 kB 1.6 MB/s eta 0:00:00
Installing collected packages: pyodbc
Successfully installed pyodbc-5.0.0b2
$ python                     
Python 3.11.4 | packaged by conda-forge | (main, Jun 10 2023, 18:08:41) [Clang 15.0.7 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import pydobc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ModuleNotFoundError: No module named 'pydobc'
>>> import pyodbc
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: dlopen(/Users/bfalk/miniconda3/envs/test/lib/python3.11/site-packages/pyodbc.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'

[mkleehammer]

Thanks. Can you run the following to help me troubleshoot this:

import platform
print(platform.processor())

[mkleehammer]

UPDATE: Never mind - I see the previous post has the same error and does have unixodbc installed.

@falkben Could your error indicate it is not finding the ODBC driver manager?

ImportError: 
dlopen(/Users/bfalk/miniconda3/envs/test/lib/python3.11/site-packages/pyodbc.cpython-311-darwin.so, 0x0002): 
symbol not found in flat namespace '_SQLAllocHandle'

This function is not a pyodbc function but rather an ODBC function that should be implemented by the driver manager. Are you able to run pyodbc when builing from source? Do you have unixodbc installed?

[mkleehammer]

It is somehow related to either (1) not finding unixodbc or (2) an architecture mismatch. Running the Python snippet I supplied above will tell us if Python is running under Rosetta.

Can you also run file $(which isql) and paste the output here?

Does anyone have any other recommended ways to tell? I'm going by this: https://apple.stackexchange.com/a/416044

[falkben]

>>> import platform
>>> print(platform.processor())
arm

(.venv) $ file $(which isql)
/opt/homebrew/bin/isql: Mach-O 64-bit executable arm64

i've tried in a conda environment (with unixodbc installed into the conda environment) and a virtualenv virtual environment with this new wheel and neither work.

When installing with:

pip install --no-binary :all: --no-cache-dir --force-reinstall pyodbc

it is able to import pyodbc without error

[mkleehammer]

@falkben OK, thanks. I'm going to have to borrow my wife's M1 mac and work on this.

[ebauch]

@anibal2j do you have unixodbc installed (brew install unixodbc)? If yes, try ``` export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/your-version/lib" export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/your-version/include" pip install --no-binary :all: pyodbc

Worked on Apple M1 Max

[Nonary]

By the way if anyone else is also having problems even with the workaround, make sure you uninstall the x86 unixodbc and Microsoft odbc installations if you happen to have a Rosetta installation of brew on the side. That was what caused my problem, once I removed those it worked.

[wilsonatmaddisonclarke]

Hi @nleroy917! I've just had the same issue the other day. I managed to fix it with a --no-binary option which forces pip to compile the module from source instead of installing from precompiled wheel.

pip install --no-binary :all: pyodbc

that's very advanced problem solving, I salut you sir.

[mkleehammer]

I thought we were just waiting for some Apple Silicon GitHub action runners for building, but now I seriously don't know if we can even have a good solution given that the environment on Mac varies so much due to Rosetta.

Does anyone feel like they are sure what the solution really is and could spell it out in detail? Do we pick the "best", like all AS, and somehow detect and raise an error if the env doesn't match?

[jordantshaw]

Macbook Pro M1.

I am able to successfully install and import pyodbc using conda by running. conda install pyodbc

I am able able to install using pip, but fails upon import. pip install pyodbc

If I run pip install --no-binary :all: pyodbc then I am able to successfully install and import.

Any ideas why it works fine with conda but not pip?

[iloveitaly]

This is what worked for me:

poetry install

# make sure this runs successfully
odbcinst -j

export LDFLAGS="-L$(brew --prefix unixodbc)/lib"
export CPPFLAGS="-I$(brew --prefix unixodbc)/include"

pip uninstall pyodbc
pip cache remove pyodbc

pip install --no-binary :all: pyodbc

[stealthco-tony]

Any chance we can resolve this internally so M1 users don't need a special command/flag? We need our code to work across OSX, unix, Windows servers.

Would prefer not to write conditional logic based on OS. Thanks!

[softgandhi]

Uninstall pyodbc and reinstall with: pip3 install --no-cache-dir --no-binary :all: pyodbc worked for me on Mac m3. Thanks!!

[cagrimunyas]

Uninstall pyodbc and reinstall with: pip3 install --no-cache-dir --no-binary :all: pyodbc worked for me on Mac m3. Thanks!!

Worked for me as well on macOS Sonoma with M1 Max

[igorvgalas]

hey. i work on mac m3 try all recomendation. i had an Error loading pyodbc module: dlopen(/Users/cheef/.local/share/virtualenvs/FAM-LuwvcngT/lib/python3.11/site-packages/pyodbc.cpython-311-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'.

by the way for python 3.9 the same error. Error loading pyodbc module: dlopen(/Users/cheef/.local/share/virtualenvs/FAM-e8HsoArV/lib/python3.9/site-packages/pyodbc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'

unixODBC 2.3.12 DRIVERS............: /usr/local/etc/odbcinst.ini SYSTEM DATA SOURCES: /usr/local/etc/odbc.ini FILE DATA SOURCES..: /usr/local/etc/ODBCDataSources USER DATA SOURCES..: /Users/xxxx/.odbc.ini SQLULEN Size.......: 8 SQLLEN Size........: 8 SQLSETPOSIROW Size.: 8

oh, and one more i use a pipenv.

Can some one help me with that please?)))

[mkleehammer]

This was a very long road. I hope we're at the end.

Version 5.1.0 has been released with both Intel and ARM binaries, so hopefully pip will do the right thing.

We'd appreciate any feedback from Mac users. I'll keep this open for a while until we find out how well they will work.

[jordantshaw]

This was a very long road. I hope we're at the end.

Version 5.1.0 has been released with both Intel and ARM binaries, so hopefully pip will do the right thing.

We'd appreciate any feedback from Mac users. I'll keep this open for a while until we find out how well they will work.

Just tested on Macbook M1, works as intended! Thank you for all the hard work that went into this. So glad its fixed!

❯ pip install pyodbc==5.1.0
Looking in indexes: https://pypi.org/simple
Collecting pyodbc==5.1.0
  Obtaining dependency information for pyodbc==5.1.0 from https://files.pythonhosted.org/packages/a5/2f/32e8845205e7fc31f67d8b01c8906b964bfbf640aa6306aba8e696eeef79/pyodbc-5.1.0-cp311-cp311-macosx_11_0_arm64.whl.metadata
  Using cached pyodbc-5.1.0-cp311-cp311-macosx_11_0_arm64.whl.metadata (2.7 kB)
Using cached pyodbc-5.1.0-cp311-cp311-macosx_11_0_arm64.whl (71 kB)
Installing collected packages: pyodbc
Successfully installed pyodbc-5.1.0

[notice] A new release of pip is available: 23.2.1 -> 24.0
[notice] To update, run: python3.11 -m pip install --upgrade pip

❯ python -c 'import pyodbc; print(pyodbc)'
<module 'pyodbc' from '/Users/jshaw/.pyenv/versions/datapipeline/lib/python3.11/site-packages/pyodbc.cpython-311-darwin.so'>

[airdrik]

Tested on a Macbook M2 (pip install --upgrade pyodbc) with similar results as jordantshaw. Nice work, thank you!

[igorvgalas]

on Mac M3 ok too. Thanks a lot.

[13brindarao]

on Mac M3 ok too. Thanks a lot. @igorvgalas I have Mac M3 too and facing the same issue. Could you please write down the steps you followed. Many thanks.

[igorvgalas]

on Mac M3 ok too. Thanks a lot. @igorvgalas I have Mac M3 too and facing the same issue. Could you please write down the steps you followed. Many thanks.

Hey. Are you made a copy of the system to your Mac m3 from your pervious Mac? Why i asking it, because I do and it copied also a homebrew. My previous Mac was on intel processor. That’s a problem. You need to uninstall howebrew, then install all from scratch.

[13brindarao]

@igorvgalas No, I did not copy any settings from my old Mac. This is a new laptop setup. I freshly installed homebrew, yet i am facing the same issue.

[igorvgalas]

@igorvgalas No, I did not copy any settings from my old Mac. This is a new laptop setup. I freshly installed homebrew, yet i am facing the same issue.

What error message do you get?

[13brindarao]

@igorvgalas

The steps followed:

Step1: brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release brew update HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18 mssql-tools18

Step2: brew install unixodbc

Step3: export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/your-version/lib" export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/your-version/include"

Step4: python -m .venv venv pip install --no-binary :all: pyodbc pip install aioodbc

output of odbcinst -j unixODBC 2.3.12 DRIVERS............: /opt/homebrew/etc/odbcinst.ini SYSTEM DATA SOURCES: /opt/homebrew/etc/odbc.ini FILE DATA SOURCES..: /opt/homebrew/etc/ODBCDataSources USER DATA SOURCES..: /Users/brindarao/.odbc.ini SQLULEN Size.......: 8 SQLLEN Size........: 8 SQLSETPOSIROW Size.: 8

ImportError: dlopen(/Users/brindarao/.venv/lib/python3.9/site-packages/pyodbc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'

[igorvgalas]

@igorvgalas

The steps followed:

Step1: brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release brew update HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18 mssql-tools18

Step2: brew install unixodbc

Step3: export LDFLAGS="-L/opt/homebrew/Cellar/unixodbc/your-version/lib" export CPPFLAGS="-I/opt/homebrew/Cellar/unixodbc/your-version/include"

Step4: python -m .venv venv pip install --no-binary :all: pyodbc pip install aioodbc

output of odbcinst -j unixODBC 2.3.12 DRIVERS............: /opt/homebrew/etc/odbcinst.ini SYSTEM DATA SOURCES: /opt/homebrew/etc/odbc.ini FILE DATA SOURCES..: /opt/homebrew/etc/ODBCDataSources USER DATA SOURCES..: /Users/brindarao/.odbc.ini SQLULEN Size.......: 8 SQLLEN Size........: 8 SQLSETPOSIROW Size.: 8

ImportError: dlopen(/Users/brindarao/.venv/lib/python3.9/site-packages/pyodbc.cpython-39-darwin.so, 0x0002): symbol not found in flat namespace '_SQLAllocHandle'

Try to install windows driver 17.

[anguspmitchell]

I followed some variation of recommended steps 9 different ways. Couldn't figure out what was going on. I consistently got the 0x0002): symbol not found in flat namespace '_SQLAllocHandle' error

Steps I kept following

brew install unixodbc

# make sure this runs successfully
odbcinst -j

brew tap microsoft/mssql-release https://github.com/Microsoft/homebrew-mssql-release
brew update
HOMEBREW_ACCEPT_EULA=Y brew install msodbcsql18 mssql-tools18

# check out installed drivers
cat /opt/homebrew/etc/odbcinst.ini

export LDFLAGS="-L$(brew --prefix unixodbc)/lib"
export CPPFLAGS="-I$(brew --prefix unixodbc)/include"

pip uninstall pyodbc
pip cache remove pyodbc

pip install --no-binary :all: pyodbc

x86 Python vs ARM Python

Finally I noticed that wheel in my pip install output was pyodbc-5.1.0-cp38-cp38-macosx_10_9_x86_64.whl, emphasis on the x86_64 part. I thought this was odd, since I was working in an ARM terminal.

What I did not realize is that there are ARM versions of Python and x86 versions of python, and x86 versions can work from ARM terminals.

I confirmed this by running

python -c "import platform; print(platform.processor())"
> i386

So the problem was that my Homebrew was in ARM world, and my Python was in x86 world. I assume that I could have fixed the problem by installing an ARM version of Python. However, I simply installed an x86 version of unixodbc and the MSFT drivers through an x86 version of Homebrew.

[v-chojas]

Yes, you need to have either consistently arm64 or amd64 software. Python needs to match pyODBC which then needs to match unixODBC and the ODBC driver(s) you need to use.