psycopg / psycopg2

PostgreSQL database adapter for the Python programming language
https://www.psycopg.org/
Other
3.36k stars 505 forks source link

(Django)(MAC Big Sur 11.0.1) ImportError: dlopen #1216

Closed awildjosh closed 2 years ago

awildjosh commented 3 years ago

Apologies in advance -- this is probably doubling up on #1200 and #1208, but I'm curious as to whether there's any workaround whatsoever (as I really would like to use postgresql with my project, but I'm too much of a newb to comprehend most of the issues I run into). I've tried pretty much every suggestion to be found through googling to no avail. The issue is as below:

Installed pyscopg2 to the venv without any issues using: pip install psycopg2

Meanwhile, Postgresql was successfully installed and the db I'd like to connect to is active (using pgadmin4). Django settings are configured accordingly.

However, when trying to access psycopg2 through the terminal, whether it be through: python -c "import psycopg2" or python manage.py migrate, etc.

The below error occurs: (venv) ...-MacBook-Air project % python -c "import psycopg2" Traceback (most recent call last): File "<string>", line 1, in <module> File "/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module> from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): no suitable image found. Did find: /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture /Users/.../Documents/project/venv/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture

Any help/insight would be greatly appreciated. Or are all of us who jumped ship to M1 doomed until Apple stop breaking things? :)

bschollnick commented 3 years ago

Second this. Unable to use natively under M1. I am forced to use under Intel Translation / Rosetta 2.

Psycopg3 also does not compile natively...

dvarrazzo commented 3 years ago

Is there an M1 to test with?

bschollnick commented 3 years ago

I'm more than happy to volunteer my own time to test... I can either run whatever tests you need, or I can volunteer to allow you to remote in, etc.

dvarrazzo commented 3 years ago

I would appreciate that, thank you.

Most likely rebuilding packages using the newer pypa infrastructure would just solve the issue.

Could you please checkout https://github.com/psycopg/psycopg2-wheels/ and check if the scripts/build_macos.sh produces working packages?

Thank you very much.

Ref. https://github.com/pypa/pip/issues/9138

bschollnick commented 3 years ago

As far as I can tell, with Rosetta 2, everything was fine (which seems correct, since I can install via PIP under Rosetta 2).

README.rst  appveyor.yml    data        psycopg2    scripts
benjamin@Nerv psycopg2-wheels-master % cd scripts
benjamin@Nerv scripts % build_libpq_macos.sh
configure: error: library 'crypto' is required for OpenSSL
benjamin@Nerv scripts % pip install crypto
Defaulting to user installation because normal site-packages is not writeable
Collecting crypto
  Downloading crypto-1.4.1-py2.py3-none-any.whl (18 kB)
Collecting Naked
  Downloading Naked-0.1.31-py2.py3-none-any.whl (590 kB)
     |████████████████████████████████| 590 kB 2.9 MB/s 
Requirement already satisfied: requests in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from Naked->crypto) (2.25.1)
Collecting pyyaml
  Using cached PyYAML-5.3.1-cp38-cp38-macosx_10_14_x86_64.whl
Requirement already satisfied: idna<3,>=2.5 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2.10)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (1.26.2)
Requirement already satisfied: chardet<5,>=3.0.2 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (4.0.0)
Requirement already satisfied: certifi>=2017.4.17 in /Users/benjamin/Library/Python/3.8/lib/python/site-packages (from requests->Naked->crypto) (2020.12.5)
Collecting shellescape
  Downloading shellescape-3.8.1-py2.py3-none-any.whl (3.1 kB)
Installing collected packages: pyyaml, shellescape, Naked, crypto
  WARNING: The script naked is installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  WARNING: The scripts crypto and decrypto are installed in '/Users/benjamin/Library/Python/3.8/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed Naked-0.1.31 crypto-1.4.1 pyyaml-5.3.1 shellescape-3.8.1
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
Makefile:14: ../../../src/Makefile.global: No such file or directory
Makefile:94: /src/Makefile.shlib: No such file or directory
make: *** No rule to make target `/src/Makefile.shlib'.  Stop.
benjamin@Nerv scripts % build_libpq_macos.sh
benjamin@Nerv scripts % ls      
build-manylinux.sh      build_manylinux1_x86_64.sh
build_libpq.sh          build_manylinux2014_aarch64.sh
build_libpq_macos.sh        build_sdist.sh
build_macos.sh          configure_postgres.sh
build_manylinux1_i686.sh    upload_packages.sh
benjamin@Nerv scripts % build_macos.sh
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(binary-io.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(getprogname.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(lock.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-context.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(se-selinux.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(stat-time.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(threadlib.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(unistd.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(wctype-h.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: lib/libsed.a(localtime-buffer.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(fd-hook.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(localename-table.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sig-handler.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(sys_socket.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: file: libtests.a(thread.o) has no symbols
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
      ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
      for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
        /usr/bin/install -c -m 644 ./$file \
                /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
      done; \
      for file in Makevars; do \
        rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
      done; \
    else \
      : ; \
    fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2
benjamin@Nerv scripts % 

But at the end, it dies looking for Cellar? I don't know what that's about?

Trying next under Native M1.

bschollnick commented 3 years ago

First attempt:

(testing) benjamin@Nerv scripts % build_macos.sh      
+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(c-ctype.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(stripslash.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(threadlib.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: archive member: lib/libsed.a(strerror-override.o) cputype (16777223) does not match previous archive members cputype (16777228) (all members must match)
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(c-ctype.o) has no symbols
/Library/Developer/CommandLineTools/usr/bin/ranlib: for architecture: x86_64 file: lib/libsed.a(threadlib.o) has no symbols
make[2]: *** [lib/libsed.a] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

But that may be due to previously running the build under Rosetta 2 in that folder structure. I nuked it, and re-unarchived it, which resulted in:

+ PYVERSIONS='2.7.15 3.5.4 3.6.6 3.7.0 3.8.0 3.9.0'
+++ dirname build_macos.sh
++ cd .
++ pwd
+ dir='/Volumes/4TB_Drive/sorted_downloads/Zip Files/psycopg2-wheels-master/scripts'
+ curl -sL https://ftp.gnu.org/gnu/sed/sed-4.8.tar.xz
+ tar xf -
+ cd sed-4.8/
+ ./configure --prefix=/usr/local/Cellar/gnu-sed/4.8 --program-prefix=g gl_cv_func_ftello_works=yes
+ make
lib/obstack.c:351:31: warning: incompatible pointer types initializing 'void (*)(void) __attribute__((noreturn))' with an expression of type 'void (void)'
      [-Wincompatible-pointer-types]
__attribute_noreturn__ void (*obstack_alloc_failed_handler) (void)
                              ^
1 warning generated.
+ make install
/Library/Developer/CommandLineTools/usr/bin/make  install-recursive
Making install in po
if test "sed" = "gettext-tools"; then \
      ../build-aux/install-sh -c -d /usr/local/Cellar/gnu-sed/4.8/share/gettext/po; \
      for file in Makefile.in.in remove-potcdate.sin quot.sed boldquot.sed en@quot.header en@boldquot.header insert-header.sin Rules-quot   Makevars.template; do \
        /usr/bin/install -c -m 644 ./$file \
                /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
      done; \
      for file in Makevars; do \
        rm -f /usr/local/Cellar/gnu-sed/4.8/share/gettext/po/$file; \
      done; \
    else \
      : ; \
    fi
Making install in .
 build-aux/install-sh -c -d '/usr/local/Cellar/gnu-sed/4.8/bin'
mkdir: /usr/local/Cellar: Permission denied
mkdir: /usr/local/Cellar: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed: No such file or directory
mkdir: /usr/local/Cellar/gnu-sed/4.8: No such file or directory
make[3]: *** [install-binPROGRAMS] Error 1
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install] Error 2

But a significantly faster built (Rosetta took about 2-3 minutes?, M1 about 20-30 seconds).

And ended with the seller / sed messages above.

dvarrazzo commented 3 years ago

make install should run as sudo I think. This script is designed to run in Travis with a superuser account.

bschollnick commented 3 years ago

I'm trying to make sense out of the build_macos script?
And I'm failing hard. There are a lot of moving parts here, and I'm not making headway on why any of it is there...

And the more I look through the build script, it's heavily personalized, and is requiring virtual environments that I don't have configured, and home-brew which is not yet working reliably under the native M1.

I couldn't get the scripts as supplied to work under either Intel Mode, or M1 Native mode.

Instead I just grabbed the psycopg2 dist from GitHub, and ran:

psycopg2 % python3 setup.py sdist bdist_wheel

Which created the Intel and m1 native based wheels. I'm unclear on what all the futzing around in the other scripts is suppose to accomplish?

I haven't had a chance to test these yet...

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl.zip

psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_x86_64.whl.zip

dvarrazzo commented 3 years ago

The futzing is to create self-contained pacakges, containing the libpq. Without it, the wheels need some system libraries to run, and because on mac os there is no standard for installation, people complain that it's too difficult.

I'll try and re-run the packages build, maybe you can help me testing if the packages work alright.

dvarrazzo commented 3 years ago

@bschollnick Do packages available here work? https://upload.psycopg.org/psycopg2-2.8.7.dev0/ Thank you!

bschollnick commented 3 years ago

Works fine under Intel Translation, but fails under M1 Native.

(testing) Nerv ~ % pip install /Volumes/4TB_Drive/sorted_downloads/2021-01-11/psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl
ERROR: psycopg2_binary-2.8.7.dev0-cp38-cp38-macosx_10_9_x86_64.macosx_10_9_intel.macosx_10_10_intel.macosx_10_10_x86_64.whl is not a supported wheel on this platform.

python3 -c"from packaging import tags; print('\n'.join([str(t) for t in tags.sys_tags()]))" |head -5 Requirement already satisfied: packaging in ./venvs/testing/lib/python3.8/site-packages (20.8) Requirement already satisfied: pyparsing>=2.0.2 in ./venvs/testing/lib/python3.8/site-packages (from packaging) (2.4.7) cp38-cp38-macosx_11_0_arm64 cp38-cp38-macosx_11_0_universal2 cp38-abi3-macosx_11_0_arm64 cp38-abi3-macosx_11_0_universal2 cp38-none-macosx_11_0_arm64

bschollnick commented 3 years ago

Is there anything else that I can do to help?

bschollnick commented 3 years ago

FYI, I meant to post this here. But I ended up reposting it back to the same Pillow Thread (too many windows open at once).

Okay, obviously I'm missing something here.

_

(testing) dist % pip install psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl ERROR: psycopg2-2.9.dev0-cp38-cp38-macosx_10_14_arm64.whl is not a supported wheel on this platform. From a comment from Pillow:

I build Pillow and jpegsrc-6b from source before filing this issue and that didn't need any changes other than specifying "-arch arm64 -arch x86_64" when building the latter. That said, I haven't tested the intel part of the wheel yet (but I don't expect problems there, given past experience with the ppc to intel transition).

_

So I probably need to change the target build architecture, but I don't see where to do that... ?

dvarrazzo commented 3 years ago

Hello @bschollnick

I think you should hear from the people who contributed to https://github.com/pypa/pip/issues/9138

I have no details about the M1 architecture and the specific details to address to build packages there.

awildjosh commented 3 years ago

As a quick aside, thank you both very much for your efforts :^]

If anyone with more know-how would like to remote in on my machine to further troubleshoot, let me know

hishamkaram commented 3 years ago

version 2.8.1 works fine for me

Screen Shot 2021-01-15 at 10 13 16 PM
Andrey-Yakovtsev commented 3 years ago

version 2.8.1 works fine for me Good for you, but I tried that and failed as well ((( As far as I understood from this thread the issue isn't resolved yet?

kevbradwick commented 3 years ago

I can confirm 2.8.6 (including psycopg2-binary) works for me.

➜  psycopg2-test pip install psycopg2
Collecting psycopg2
  Downloading psycopg2-2.8.6.tar.gz (383 kB)
     |████████████████████████████████| 383 kB 3.8 MB/s
Building wheels for collected packages: psycopg2
  Building wheel for psycopg2 (setup.py) ... done
  Created wheel for psycopg2: filename=psycopg2-2.8.6-cp39-cp39-macosx_10_9_universal2.whl size=226356 sha256=bf4bdc2b8d73783423eec6a5f094d5d18f8b178f83a67017db821fb1e7353e6a
  Stored in directory: /Users/kevinbradwick/Library/Caches/pip/wheels/a2/07/10/a9a82e72d50feb8d646acde6a88000bbf2ca0f82e41aea438a
Successfully built psycopg2
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6

This is on a Macbook Air M1 running arm64.

dvarrazzo commented 3 years ago

@kevbradwick yes, it works for you, but you are a good soul who has a compiler and libraries on your machine. Which I appreciate, were everyone like you...

This problem here is with people who expect a binary package to work. Every few years, OSX people come out with a different machine with all the cheese moved around. or missing keys. Why people buy this stuff is beyond my belief.

So, this ticket is about making the binary package work. psycopg2 code works, but "Downloading psycopg2-2.8.6.tar.gz" says that you have installed the package from source, not from binary (although you indeed requested the psycopg2-binary package - that package has a source fallback for reason that now I can't remember but it was necessary).

Solving this problem requires the OSX build script to be fixed and the package to be generated by Travis. Happy to receive a MR for it.

bschollnick commented 3 years ago

Okay, it's now installing via PIP... Which is great, but when Django attempts to access the database..

I am seeing an exception for Symbol not found (Symbol not found: _PQbackendPID)

I have rebuilt the virtual environment, just in case there had been some left-overs from the trials... But no change.

- Benjamin
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
    import psycopg2 as Database
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 954, in _bootstrap_inner
    self.run()
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 892, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 790, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/opt/homebrew/Cellar/python@3.9/3.9.1_6/Frameworks/Python.framework/Versions/3.9/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
bschollnick commented 3 years ago

Wait a sec. Why is it calling something in python 3.9? I'm running stock MOSX 3.8. I'll have to check that home-brew hasn't messed my paths..

bschollnick commented 3 years ago

Okay, rebuild the virtual environment again, and now it's showing the right python version... But same issue.

Successfully built psycopg2-binary
Installing collected packages: psycopg2-binary
Successfully installed psycopg2-binary-2.8.6
(quickbbs) benjamin@Nerv quickbbs % server4
Running on nerv.local
Debug is  True
Running on nerv.local
Debug is  True
Watching for file changes with StatReloader
Exception in thread django-main-thread:
Traceback (most recent call last):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 25, in <module>
    import psycopg2 as Database
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/__init__.py", line 51, in <module>
    from psycopg2._psycopg import (                     # noqa
ImportError: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/core/management/commands/runserver.py", line 110, in inner_run
    autoreload.raise_last_exception()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 76, in raise_last_exception
    raise _exception[1]
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/core/management/__init__.py", line 357, in execute
    autoreload.check_errors(django.setup)()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/utils/autoreload.py", line 53, in wrapper
    fn(*args, **kwargs)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/apps/registry.py", line 114, in populate
    app_config.import_models()
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/apps/config.py", line 211, in import_models
    self.models_module = import_module(models_module_name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/contrib/auth/models.py", line 2, in <module>
    from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/contrib/auth/base_user.py", line 48, in <module>
    class AbstractBaseUser(models.Model):
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/base.py", line 122, in __new__
    new_class.add_to_class('_meta', Options(meta, app_label))
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/base.py", line 326, in add_to_class
    value.contribute_to_class(cls, name)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/models/options.py", line 206, in contribute_to_class
    self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/__init__.py", line 28, in __getattr__
    return getattr(connections[DEFAULT_DB_ALIAS], item)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/utils.py", line 214, in __getitem__
    backend = load_backend(db['ENGINE'])
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/utils.py", line 111, in load_backend
    return import_module('%s.base' % backend_name)
  File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 29, in <module>
    raise ImproperlyConfigured("Error loading psycopg2 module: %s" % e)
django.core.exceptions.ImproperlyConfigured: Error loading psycopg2 module: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID
  Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
  Expected in: flat namespace
 in /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so
kevbradwick commented 3 years ago

how did you install postgres? did you compile it manually?

dvarrazzo commented 3 years ago

Note that you don't need to compile postgres in order to build psycopg2, but you may need to compile the libpq, the postgres client-side library. psycopg2 does it in the scripts/build_libpq_macos.sh (in the psycopg2-wheels project).

bschollnick commented 3 years ago

As I mentioned, I can install it (via pip install psycopg2), but when used to connect to a postgresql server on the same machine, I get the symbol error.

➜ psycopg2-test pip install psycopg2

That’s not binary? That’s the source. Isn’t psycopg2-binary is the binary version? And our configurations are not the same… You are running Python39, whereas I am using stock 3.8.2.

I hate to ask, but have you used it in the test environment?

For example:

(quickbbs) benjamin@Nerv quickbbs % pip install psycopg2 Collecting psycopg2 Using cached psycopg2-2.8.6.tar.gz (383 kB) Building wheels for collected packages: psycopg2 Building wheel for psycopg2 (setup.py) ... done Created wheel for psycopg2: filename=psycopg2-2.8.6-cp38-cp38-macosx_10_14_arm64.whl size=225151 sha256=207fec8ca5550433e0d15746854c5910d5b3d017c1a4986e0f420446c7fae75d Stored in directory: /Users/benjamin/Library/Caches/pip/wheels/70/5e/69/8a020d78c09043156a7df0b64529e460fbd922ca065c4b795c Successfully built psycopg2 Installing collected packages: psycopg2 Successfully installed psycopg2-2.8.6

WTH? Why are you getting the universal wheel, and I’m getting the Arm64?

That’s could be the issue…. Successfully built psycopg2 Installing collected packages: psycopg2 Successfully installed psycopg2-2.8.6 (quickbbs) benjamin@Nerv quickbbs % server4 Running on nerv.local Debug is True Running on nerv.local Debug is True Watching for file changes with StatReloader Exception in thread django-main-thread: Traceback (most recent call last): File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/django/db/backends/postgresql/base.py", line 25, in import psycopg2 as Database File "/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/init.py", line 51, in from psycopg2._psycopg import ( # noqa ImportError: dlopen(/Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so, 2): Symbol not found: _PQbackendPID Referenced from: /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so Expected in: flat namespace in /Users/benjamin/venvs/quickbbs/lib/python3.8/site-packages/psycopg2/_psycopg.cpython-38-darwin.so

During handling of the above exception, another exception occurred:

But I don’t know.

- Benjamin

On Jan 15, 2021, at 1:06 PM, Kevin Bradwick notifications@github.com wrote:

I can confirm 2.8.6 (including psycopg2-binary) works for me.

➜ psycopg2-test pip install psycopg2 Collecting psycopg2 Downloading psycopg2-2.8.6.tar.gz (383 kB) |████████████████████████████████| 383 kB 3.8 MB/s Building wheels for collected packages: psycopg2 Building wheel for psycopg2 (setup.py) ... done Created wheel for psycopg2: filename=psycopg2-2.8.6-cp39-cp39-macosx_10_9_universal2.whl size=226356 sha256=bf4bdc2b8d73783423eec6a5f094d5d18f8b178f83a67017db821fb1e7353e6a Stored in directory: /Users/kevinbradwick/Library/Caches/pip/wheels/a2/07/10/a9a82e72d50feb8d646acde6a88000bbf2ca0f82e41aea438a Successfully built psycopg2 Installing collected packages: psycopg2 Successfully installed psycopg2-2.8.6 This is on a Macbook Air M1 running arm64.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/psycopg/psycopg2/issues/1216#issuecomment-761096793, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHPEZXHUTSG6LB5ZZLJRZLS2B735ANCNFSM4VS5GJ4Q.

bschollnick commented 3 years ago

Postgres is working fine. I’m using PostgresApp…

And Psycopg2 is working fine in Intel mode, as well.

- Benjamin

On Jan 15, 2021, at 5:33 PM, Kevin Bradwick notifications@github.com wrote:

how did you install postgres? did you compile it manually?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/psycopg/psycopg2/issues/1216#issuecomment-761232375, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHPEZUNGB23RBYFHKOLWHDS2C7DTANCNFSM4VS5GJ4Q.

djch commented 3 years ago

@bschollnick Exact same problem here per #1208

I'm using python@3.9 from homebrew, though

dvarrazzo commented 3 years ago

@bschollnick

That’s not binary? That’s the source. Isn’t psycopg2-binary is the binary version?

For example:

(quickbbs) benjamin@Nerv quickbbs % pip install psycopg2 Collecting psycopg2 Using cached psycopg2-2.8.6.tar.gz (383 kB)

This ^ says that the package installed is the source one. The packages are found from this list:

https://pypi.org/project/psycopg2-binary/#files

Your pip run identifies which package is the most suitable for your system. For some reasons, which might be incompatibility with your system but also an out-of-date pip, which doesn't know enough of it, the macos package is not chosen. Because there is a source package too in the list (the .tar.gz one), the latter is selected.

It sounds silly that there is a source choice too in the binary package... Initially there wasn't, but that created more problems. I can't remember exactly, but they were of the tone that if someone has psycopg2-binary in a requirements.txt file, and tries to install the system e.g. in a python-alpine docker (which is not compatible with wheel packages), it would have resulted impossible, whereas now it is possible as long as they have the build tools too.

However, if you pip install psycopg2-binary and you get the .tar.gz package, even if your installation works alrigh I would say that the binary package is not doing its job.

kevbradwick commented 3 years ago

Postgres is working fine. I’m using PostgresApp… And Psycopg2 is working fine in Intel mode, as well. - Benjamin

could it be that the binary needs some shared dynamic library (also arm64) but the one you have installed is via Postgress.app which is x86?

dvarrazzo commented 3 years ago

Yes: It needs the libraries built by scripts/build_libpq_macos.sh

kevbradwick commented 3 years ago

also, may not be necessary, but i installed OpenSSL 1.1.1i via source in place of the LibreSSL version found on Big Sur. This fixed a different error message I received relating to ssl library not found.

bschollnick commented 3 years ago

Postgres is working fine. I’m using PostgresApp… And Psycopg2 is working fine in Intel mode, as well. - Benjamin … <x-msg://8/#> On Jan 15, 2021, at 5:33 PM, Kevin Bradwick @.***> wrote: how did you install postgres? did you compile it manually? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1216 (comment) https://github.com/psycopg/psycopg2/issues/1216#issuecomment-761232375>, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHPEZUNGB23RBYFHKOLWHDS2C7DTANCNFSM4VS5GJ4Q https://github.com/notifications/unsubscribe-auth/AAHPEZUNGB23RBYFHKOLWHDS2C7DTANCNFSM4VS5GJ4Q.

could it be that the binary needs some shared dynamic library (also arm64) but the one you have installed is via Postgress.app which is x86?

Possibly?!?!

I’ve considering brewing up postgresql, but I can’t tell if brew will cook it up as M1 Native or Intel based. I stumbled across a page that had a break down on their site, but I can’t find it now.

I just simply want to run Django in M1 native, instead of via intel translation… >g<

- Benjamin
bschollnick commented 3 years ago

also, may not be necessary, but i installed OpenSSL 1.1.1i via source in place of the LibreSSL version found on Big Sur. This fixed a different error message I received relating to ssl library not found.

Yep, that was a large stumbling block, until Brew said that they were Big Sur ready a few days ago.

I used home-brew to cook up OpenSSL 1.1.1 which fixed the issues I was having installing Cryptopgraphy.

- Benjamin
kevbradwick commented 3 years ago

I just simply want to run Django in M1 native, instead of via intel translation… >g< - Benjamin

Right now, the only way that I am aware of is to compile and install Postgres manually, which is pretty straight forward, and then install psycopg2. This is what has worked for me. It'll be a while before brew and other tools catch up - silicon is still in early days. I find it part of the fun 🙂

tesh-raichandani commented 3 years ago

thank you all for the answers. I faced similar issue on m1 mac and uninstalling psycopg2 and installing it from binary fixed it for me. arch -x86_64 pip install psycopg2-binary

tesh-raichandani commented 3 years ago

It fails because it is looking for system libraries in /usr/lib directory instead of loading them from dynamic linker cache, (see https://developer.apple.com/forums/thread/659285) for a package that reads libraries from cache:

$ otool -L /Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so
/Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so:
    @loader_path/.dylibs/libpq.5.11.dylib (compatibility version 5.0.0, current version 5.11.0)
    /usr/lib/libssl.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libcrypto.0.9.8.dylib (compatibility version 0.9.8, current version 0.9.8)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.60.2)

And for the failing case:

$ otool -L /Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so
/Users/tesh/rescale/venv3.9/lib/python3.9/site-packages/xmlsec.cpython-39-darwin.so:
    /usr/lib/libxslt.1.dylib (compatibility version 3.0.0, current version 3.26.0)
    /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.11)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.60.1)
    /usr/lib/libicucore.A.dylib (compatibility version 1.0.0, current version 66.1.0)
grahamcracker commented 3 years ago

I'm having the exact same problem (Big Sur, M1) in lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so:

2): Symbol not found: _PQbackendPID

I've tried uninstalling/reinstalling psycopg2:

  Using cached psycopg2-2.8.6-cp39-cp39-macosx_11_0_arm64.whl
Installing collected packages: psycopg2
Successfully installed psycopg2-2.8.6

Ilooks like I'm installing an OSX 11 / ARM64 package, but the Postgres.app that I'm running is Intel. I'm going to try reinstalling Postgres via homebrew and see if that fixes it.

grahamcracker commented 3 years ago

^ I can confirm that the above solution worked for me (installing Homebrew Postgres, which is the Apple architecture instead of Intel from Postgres.app).

However, for anyone else that runs into this, I had to uninstall psycopg2 and re-install using pip with --no-cache-dirs since I needed the build process to run again.

bschollnick commented 3 years ago

I have considered installing postgres from home-brew, but I couldn’t track down any information on how to configure, etc?

Any pointers on where to get some instructions on command line configuring postgres from scratch?

- Benjamin

On Jan 23, 2021, at 1:10 PM, Graham Ramsey notifications@github.com wrote:

^ I can confirm that the above solution worked for me (installing Homebrew Postgres, which is the Apple architecture instead of Intel from Postgres.app).

However, for anyone else that runs into this, I had to uninstall psycopg2 and re-install using pip with --no-cache-dirs since I needed the build process to run again.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/psycopg/psycopg2/issues/1216#issuecomment-766154154, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHPEZW3GJW7TUCL4LOGSSTS3MGKDANCNFSM4VS5GJ4Q.

grahamcracker commented 3 years ago

@bschollnick Simply install homebrew and install Postgres (brew install postgresql). On my M1 pro, the install automatically used the arm64 build of Postgres and symlinked so my psycopg2 install could be rebuilt :)

bschollnick commented 3 years ago

Okay.. That was more simple than I thought… I wasn’t expecting Home-brew to give me the details on start and stop.

The only headache was getting the initial user password set, and a few things, that I guess are more automatic with the pre-build binaries? But finding the psql command eventually solved that for me.

One thing that surprised me at the start is the Intel/rosetta psycopg is working connecting the ARM server, I assume that’s because it’s still using the INTEL based postgres dll since I haven’t uninstalled those yet.

But, I can confirm that the ARM version of postgres is running fine, I’ll look at compiling psycopg in arm mode either tonight or tomorrow.

But everyone thanks for the assistance…

- Benjamin

On Jan 23, 2021, at 4:50 PM, Graham Ramsey notifications@github.com wrote:

@bschollnick https://github.com/bschollnick Simply install homebrew https://brew.sh/ and install Postgres https://wiki.postgresql.org/wiki/Homebrew (brew install postgresql). On my M1 pro, the install automatically used the arm64 build of Postgres and symlinked so my psycopg2 install could be rebuilt :)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/psycopg/psycopg2/issues/1216#issuecomment-766185150, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAHPEZURRRS3DNPEFUT6YDDS3NADVANCNFSM4VS5GJ4Q.

bschollnick commented 3 years ago

Everyone Thanks.

I just got Psycopg2 to pip install under M1 native. Thank you

However, for anyone else that runs into this, I had to uninstall psycopg2 and re-install using pip with --no-cache-dirs since I needed the build process to run again.

Thank you for mentioning this (BTW, singular not plural). You saved me a bit of a headache, since I couldn't get it to work, until I forced it to not use the cache.

Sadly, the PyMuPDF (aka fitz) library isn't updated yet, and that's a large dependency... It's not something I'm tackling tonight, so I'll just have to cope with yet another dependency to wait for.

RyanDurk commented 3 years ago

Enormous thanks to the psycopg team! I ran into this problem with a Rosetta-based install of Postgres.app and an Apple Silicon install of python3.9.

Installing libpq from source got things working again.

` $ brew install libpq --build-from-source

$ export LDFLAGS="-L/opt/homebrew/opt/libpq/lib"

$ pip install psycopg2 `

Cheers, Ryan

FabianBruengerPexon commented 3 years ago

Hi guys,

i am currently running into the same issues and so far I can not resolve them. My set up: Big Sur v11.2.1

Installed: postgres (running) python3.9 psycopg2

I created a data base and just wrote a test script to connect: import psycopg2

def main():

connect to the created database in postrgeSQL ""

conn = psycopg2.connect(
    host="localhost",
    database="suppliers",
    user="name",
    password="")
print("hello")

if name == "main":

execute only if run as a script

main()

This is the error I am getting:

name@MBP-von-name postgresSQL % python3 tutorial1.py
Traceback (most recent call last): File "/Users/name/Documents/VSCode/Python/postgresSQL/tutorial1.py", line 1, in import psycopg2 File "/opt/homebrew/lib/python3.9/site-packages/psycopg2/init.py", line 51, in from psycopg2._psycopg import ( # noqa ImportError: dlopen(/opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so, 2): no suitable image found. Did find: /opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture /opt/homebrew/lib/python3.9/site-packages/psycopg2/_psycopg.cpython-39-darwin.so: mach-o, but wrong architecture

Any chance of cure?

tesh-raichandani commented 3 years ago

@FabianBruengerPexon you need to use rosetta.

Either prefix your command with arch -x86_64 ($ arch -x86_64 python3 tutorial1.py) or you can open the terminal using rosetta (go to applications > right click on terminal > get info > check Open using rosetta`) and run python command that finds the correct architecture binary.

bschollnick commented 3 years ago

Please note, At least with Homebrew python, it doesn't seem to work reliably anymore. For homebrew to work correctly, I had to install both "Rosetta2" version of homebrew, and M1 native homebrew.

In addition, I actually had to do this, in my profile file.

# https://cutecoder.org/software/detecting-apple-silicon-shell-script/
arch_name="$(uname -m)"

if [ "${arch_name}" = "x86_64" ]; then
    if [ "$(sysctl -in sysctl.proc_translated)" = "1" ]; then
        echo "Running on Rosetta 2"
        ARM=0
    else
        echo "Running on native Intel"
        ARM=0
    fi 
elif [ "${arch_name}" = "arm64" ]; then
    echo "Running on ARM"
    ARM=1
else
    echo "Unknown architecture: ${arch_name}"
fi
echo "Arm Status: $ARM, $arch_name"

#Shellscript updates, imports from .bash_profile
#set PYTHONIOENCODING=UTF-8

export PATH=".:$PATH"

# brew path 
if [ $ARM -eq 0 ]; then
    # intel 
    echo "Rosetta2"
    # homebrew
    export PATH="/opt/local/bin:$PATH"
    # homebrew python
    export PATH="/usr/local/opt/python@3.9/libexec/bin:$PATH"
    export PATH="/usr/local/lib/python3.9/site-packages:$PATH"
    export PLAT="ROSETTA2"
    alias brew='arch -x86_64 /usr/local/bin/brew'
    alias virtualenv='arch -x86_64 python -m virtualenv ~/venvs/%*'
    export LDFLAGS="-L/opt/homebrew/opt/libpq/lib"
else
    echo "M1 Native!"
    # Homebrew
    export PATH="/opt/homebrew/bin:$PATH"
    # Homebrew - Python
    export PATH="/opt/homebrew/opt/python@3.9/libexec/bin:$PATH"
    export PATH="/opt/homebrew/lib/python3.9/site-packages:$PATH"
    # OpenSSH
    export LDFLAGS="-L/opt/homebrew/opt/openssl@1.1/lib"
    export CPPFLAGS="-I/opt/homebrew/opt/openssl@1.1/include"
    export PLAT="M1"
    alias brew='arch -arm64e /opt/homebrew/bin/brew'
    alias virtualenv='arch -arm64e python -m virtualenv ~/venvs/$*'
fi

You're ask why?

Because if I didn't forcibly set the python environment, it was possible in an Rosetta enabled terminal, to get an M1 native Python running, without realizing it.

My best guess, is that when running an application, it's now possible for the shell to create an sub shell, and run the application? I don't know if this change is due to 11.21 or due to a homebrew change?

https://github.com/Homebrew/discussions/discussions/784

FabianBruengerPexon commented 3 years ago

@hitesh-raichandani Thanks for your response. But.. :

arch: posix_spawnp: python3: Bad CPU type in executable

tesh-raichandani commented 3 years ago

@hitesh-raichandani Thanks for your response. But.. :

arch: posix_spawnp: python3: Bad CPU type in executable

I guess it's the issue that @bschollnick mentioned above. I had the same thing, what worked for me was uninstalling the brew version of python and installing 3.9 using the installer.

bschollnick commented 3 years ago

@hitesh-raichandani Thanks for your response. But.. :

arch: posix_spawnp: python3: Bad CPU type in executable

Which could be exactly what I ran into.

If you are running Rosetta2 python, which has pip installing the Intel version of the Psycopg2, and then when running Python it ends up running the M1 version of the Python executable, which then "barfs" on the Intel module you are trying to load. (Or vice Versa)

My suggestion is to open Activity monitor, and check to see what Architecture is listed, and ensure that matches with the library you are loading.

FabianBruengerPexon commented 3 years ago

@bschollnick Thanks for your input.

I guess this is regarding the supported architecture: Python 3.9.1 (default, Feb 2 2021, 22:54:59) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information.

import platform platform.architecture() ('64bit', '')

@hitesh-raichandani I think I will just completely delete python and try a fresh install