search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.83k stars 1.87k forks source link

pipenv cannot install psycopg2-binary, but pip can (py 3.8) #4073

Closed kristjanvalur closed 4 years ago

kristjanvalur commented 4 years ago

Issue description

I'm trying to bring a project using requirements.txt into using Pipfile. I am stumped by the fact that pipfile cannot install psycopg2-binary, but pip can. I'm using windows, and developers will not generally be simply able to build psycopg2 from sources.

Problem:

Create a pipenv using Python 3.8 calling pipfile install psycopg2-binary

Expected result

I'll expect it to be installed to the virtualenv.

Actual result

We get the error

 raise InstallationError(
pipenv.patched.notpip._internal.exceptions.InstallationError: Command "python setup.py egg_info" failed with error code 1 in C:\Users\kristjan\AppData\Local\Temp\tmp1wcw4j8zbuild\psycopg2-binary\

workaround:

calling pipenv run pip install psycopg2-binary results in the package being installed. But this is not adequate, because it means that Pipfile and Pipfile.lock can not be used to deploy the application any more.

Steps to replicate

On windows:

pipenv --python 3.8
pipenv install psycopg2-binary

Other

I am aware that psycopg2-binary is not considered optimal for production. However, for development on windows that is the only practical way to bring the driver in for regular users. I would be happy to have a separate requirement if running on linux with suitable prerequisites installed.

antnieszka commented 4 years ago

It may be something with python3.8 & pipenv - I had the same issue few days ago.

kristjanvalur commented 4 years ago

Interesting, did you find a resolution? Meanwhile, I can confirm that the problem disappears if I use 3.7

antnieszka commented 4 years ago

No. I just used pip install. But interestingly it worked without issue on Linux (tested Manjaro and openSUSE), so it may be windows-only issue 😞

joaomarccos commented 4 years ago

Same problem here using linux.

Only works when I use --sequential flag

EdinsonRequena commented 4 years ago

Same problem with python 3.7

yelled1 commented 4 years ago

"psycopg2-binary" = "*" This works for me in Windows 10:1909 w/ 3.7.x as well as Linux(ubuntu). Close time!

Levadi commented 4 years ago

I am new to programming and I don't quite understand what do you mean by:

"psycopg2-binary" = "*"

kristjanvalur commented 4 years ago

It's been established that it works for python 3.7. It is 3.8 that is the problem and I just confirmed that it is still a problem.

yelled1 commented 4 years ago

Oh, this does NOT work even inside the Pipfile? I first thought that @Levadi did not understand that this should be done inside Pipfile & installed with: $ pipenv install --dev (optional dev) Also, @EdinsonRequena mentioned that this was problem even on 3.7. May be everytone should post minor versions as well. 3.6.5, 3.7.5 & 3.7.6 worked. I also executed 

$ pipenv check 
$ cd ~/.pyenv
$ git pull 

:/tmp/py3_8$ pipenv install --dev
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✔ Success!
Updated Pipfile.lock (996803)!
Installing dependencies from Pipfile.lock (996803)…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:01
To activate this project's virtualenv, run pipenv shell.
Alternatively, run a command inside the virtualenv with pipenv run.

My python below

Python 3.8.2

Linux mememe 4.15.0-1063-aws #67-Ubuntu SMP Mon Mar 2 07:24:29 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

My Pipfile below

:/tmp/py3_8$ cat Pipfile
[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
"psycopg2-binary" = "*"

[requires]
python_version = "3.8"
antnieszka commented 4 years ago

@yelled1 - do you have postgres development packages installed?

yelled1 commented 4 years ago

@yelled1 - do you have postgres development packages installed? @antnieszka You mean in Pipfile? Yes, as belows:

[packages]
pandas = "*"
numpy = "*"
peewee = "*"
peewee_mssql = "*"
pyodbc = "*"
python-dotenv = "*"
argparse = "*"
tqdm = "*"
jinja2 = "*"
"psycopg2-binary" = "*"
tabulate = "*"

but nowadays I am slowly trying out Poetry, which more like this: psycopg2-binary = "^2.8.5"

If you meant by Qry & looking at the data? Dbeaver all the way, as I hate PGtools & I only access data -> drivers only. Also, using VScode with SqlTools.

antnieszka commented 4 years ago

@yelled1 I mean system-wide installation of postgres tools - like the postgresql-devel, postgresql-contrib. E.g.: https://www.postgresql.org/download/linux/redhat/

...
postgresql-contrib | additional supplied modules libraries and headers for C language development
postgresql-devel | libraries and headers for C language development
yelled1 commented 4 years ago

@antnieszka in Ubuntu this was not a piece of cake but still fairly easy. Sorry, don't know much about Redhat/CentOs. I don't like'em & avoid it as much as I can. Package issue can be murderous. Also, wrong place to ask. I know, the feeling of deadlines & pressure, but if I ping-pong this in a wrong forum then others may gang up on both of us.

BTW: If anyone hopes that I would test in 3.8 w/ Windows, then u will be disappointed. There is no real justification for using the latest/greatest, unless you are into hippos := Never touched any version with minor less than x.x.5, especially in troublesome Windows. I also cross develop in Win 10 & Ubuntu. So, this is from someone who had a lot more hair because of it. Also, Pyenv-Win despite best efforts, can't really let you go back Window Python versions.

antnieszka commented 4 years ago

@yelled1 nvm :)

@kristjanvalur I have a new snippet which fails:

pipenv --python 3.8
pipenv install "psycopg2-binary==2.7.7"

This one works fine without errors for me:

pipenv --python 3.8
pipenv install psycopg2-binary

So maybe when this issue was filled, there was no psycopg2-binary wheel for python 3.8? As there is no python3.8 wheel for psycopg2-biary==2.7.7? https://pypi.org/project/psycopg2-binary/2.7.7/#files (no 2.7.7 wheel for py3.8)

As for TODAY, if you have no cached psycopg2-binary, python 3.8 + psycopg2-binary>=2.8.4 should be fine: https://pypi.org/project/psycopg2-binary/2.8.4/#files

Edit: Note that windows wheel have later date (Nov 10, 2019) than many-linux ones.

antnieszka commented 4 years ago

You may ask - why pip worked when pipenv didn't. I think, and prove me wrong, pipenv has separate cache folder. So pip always got the newest version.

techalchemy commented 4 years ago

For anyone continuing to run into this issue, go ahead and try using the new prerelease of pipenv (i.e. pip install --upgrade --pre pipenv) & make sure you have postgres client binaries installed (i.e. via sudo apt install libpq-dev on ubuntu/debian for the minimal headers). I confirmed this on a fresh container and could not reproduce the issue with the headers installed. Without the headers, I saw a rather long error message, at the top of which I was instructed to install them.

Closing for now, please file a new issue if the pre-release + installing appropriate client binaries or headers does not resolve this. Thanks!

techalchemy commented 4 years ago

You may ask - why pip worked when pipenv didn't. I think, and prove me wrong, pipenv has separate cache folder. So pip always got the newest version.

For reference, this is correct -- pipenv lock --clear clears the cache as needed

xlanor commented 4 years ago

@techalchemy , thanks! installing the required headers on macOS was enough to get me past this problem.

For macOS, simply running brew install postgresql was sufficient to grab the required headers.

fallenreaper commented 3 years ago

I was running into this issue on my Mac, with regards to pipenv. Postgres works just fine normally, but using pipenv will throw an error.

[dev-packages]

[packages] django = "" pyyaml = "" djangorestframework = "" django-cors-headers = "" psycopg2 = "" pillow = "" whitenoise = "*"

[requires] python_version = "3.7"


* OS:  Catalina 10.15.6

I have been trying to figure out why it is acting the way it is.  I was also trying by adding psycopg2-binary but that seems to fail as well.  It seems that the psycopg2 is causing a lot of issues because when I attempt to use Django, it will error on the djangorestframework etc.

Not sure what the next steps are other than creating a docker container and running everything from within a CENTOS machine, but i am not sure that is the right way.
mchogan commented 3 years ago

pipenv install psycopg2-binary worked with Python 3.8 and Postgres 12.5 on Windows 10. I followed the GeoDjango Installation Instructions for Windows. The key was using Postgres 12.5 rather than the latest release.

Hope this is helpful. Just stopping by since I found this issue while looking for a solution to this problem. I posted a little more detail on another GitHub issue here.

fallenreaper commented 3 years ago

Note to self. Don't use latest in personal builds. Find 1, use it for awhile, then update is over time. Seems like between versions I was also building with different instances of postgresql.

Thanks

On Sat, Jan 9, 2021, 22:11 mchogan notifications@github.com wrote:

pipenv install psycopg2-binary worked with Python 3.8 and Postgres 12.5 on Windows 10. I followed the GeoDjango Installation Instructions for Windows https://docs.djangoproject.com/en/3.1/ref/contrib/gis/install/#windows. The key was using Postgres 12.5 rather than the latest release.

Hope this is helpful. Just stopping by since I found this issue while looking for a solution to this problem.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pypa/pipenv/issues/4073#issuecomment-757405646, or unsubscribe https://github.com/notifications/unsubscribe-auth/AACM7ZYGRTUS4ON65XJXWVDSZELHPANCNFSM4J4LSLAQ .

yayimadinosaur commented 3 years ago

python3.8 works with psycopg2="2.7.7"

my issue - install error installing psycopg2="*" with python 3.7 on Pipfile for pipenv

solution

running macOS catalina 10.15.7 resolved the pscopg2 issue without using psycopg2-binary found this link https://stackoverflow.com/questions/55581941/getting-error-when-installing-psycopg2-2-8-1

brew update openssl

.env

LDFLAGS="-L/usr/local/opt/openssl@1.1/lib"
CPPFLAGS="-I/usr/local/opt/openssl@1.1/include"

Pipfile

[packages]
psycopg2="2.7.7"

[requires]
python_version = "3.8"
ghost commented 3 years ago

Please don't avoid this issue by closing it! The fact that pip (an inferior tool) can do this out-of-the-box while PipEnv cannot is a big pain point, especially considering the fact that a lot of beginner tutorials teach using "deploy to heroku with postgres" where this bug breaks everything and is off-putting for new learners (and potentially new users of pipenv).

akshatd commented 3 years ago

Hey @utk-dev , just installing the development libs for postgres helped, so you can give it a try as well: sudo apt-get install libpq-dev

igorMIA commented 2 years ago

Python3.8, pipenv==2022.3.28, and Ubuntu, issue still exists

meeb commented 1 year ago

I stumbled over this on a Debian-based system that had Py3.8 installed earlier today. This was resolved by installing libpq-dev (apt install libpq-dev). I've not looked into this further as obviously the binary package should not require the dev libraries installed, however this seems to work around the error with pipenv at least on the couple of Debian and Ubuntu based systems I've tested it on.