Difference in binary builds between pip 19.3 and 19.3.1

[iamareebjamal]

Environment

• pip version: 19.3.1
• Python version: 3.7.4/3.7.5
• OS: Linux (alpine)

Description

We use multistage builds to make our docker image compact.

https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/Dockerfile

We are using python:3.7-alpine as base image which had pip 19.3 pre-installed. Installing dependencies with build tools like gcc and others in the initial stage and then copying them to the later stage. This is a common practice and recommended in the docker community.

We installed the build tools and requirements using apt and pip which installed correctly using this command:

pip install --install-option="--prefix=/install" setuptools && \
    LIBRARY_PATH=/lib:/usr/lib pip install --install-option="--prefix=/install" -r /requirements.txt

Then, we copy the installed libraries to the next stage using the command COPY --from=builder /install /usr/local.

Now, all the libraries installed in the previous stage should be present in the next stage. Which was the case until today.

When we built the release container, it did not have importlib_metadata and threw ModuleError.

After hours of debugging, and jumping through diffs of previously working diff environment, we found that it was not installed at all along with some other libraries. More time consuming debugging later, I found out a difference between environments. The broken environment had Python 3.7.5 and pip 19.3.1 and working one had Python 3.7.4 and pip 19.3.

A quick matrix test on Python 3.7.4 with pip 19.3.1 and Python 3.7.5 with pip 19.3 showed that indeed the problem was with pip and not Python version.

After analyzing the missing dependencies, we found out that the only ones missing were the ones that were built during the previous stage. One of which was the aforementioned importlib_metadata along with several other built libraries.

Expected behavior

pip 19.3.1 which is a patch version should not break backward compatibility and retain the behavior as 19.3.

How to Reproduce

1. Disable wheel packages like in alpine and build the libraries
2. Copy from one install to another
3. Find the library to be missing

OR

1. Use the above-linked dockerfile.
2. Build it using this command - docker build . -t eventyay/open-event-server
3. Run the shell using this command - docker run -it --entrypoint /bin/sh eventyay/open-event-server -s
4. Run pip freeze | grep import
5. Change the version from python:3.7-alpine to python:3.7.4-alpine or add pip install pip==19.3 in the first RUN step to lock pip, and repeat steps 2-4 and you'll find the library

Output

import importlib_metadata
ModuleNotFoundError: No module named 'importlib_metadata'

[xavfernandez]

Hello @iamareebjamal,

If I understand correctly your Dockerfile basically installs https://github.com/fossasia/open-event-server/blob/b7ba05cfaefdc257e9ae07d264044668dadffce5/requirements/common.txt and nothing in there depends on importlib_metadata so having it installed was a bug that was fixed in the patch version ?

[iamareebjamal]

No, the problem is not related to importlib_metadata. You change importlib_metadata to any library which is built during install

Also, importlib_metadata is a transitive dependency of several of our dependencies.

I can provide a pipdeptree if it helps. Many direct dependencies weren't installed as well

[xavfernandez]

I can provide a pipdeptree if it helps.

It would :+1:

[iamareebjamal]

https://www.diffchecker.com/ryCo05HG

19.3

info

``` APScheduler==3.6.1 - pytz [required: Any, installed: 2019.3] - setuptools [required: >=0.7, installed: 41.4.0] - six [required: >=1.4.0, installed: 1.12.0] - tzlocal [required: >=1.2, installed: 2.0.0] - pytz [required: Any, installed: 2019.3] arrow==0.15.2 - python-dateutil [required: Any, installed: 2.8.0] - six [required: >=1.5, installed: 1.12.0] bleach==3.1.0 - six [required: >=1.9.0, installed: 1.12.0] - webencodings [required: Any, installed: 0.5.1] blinker==1.4 diff-match-patch==20181111 elasticsearch-dsl==7.0.0 - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5] - urllib3 [required: >=1.21.1, installed: 1.25.6] - python-dateutil [required: Any, installed: 2.8.0] - six [required: >=1.5, installed: 1.12.0] - six [required: Any, installed: 1.12.0] envparse==0.2.0 eventlet==0.25.1 - dnspython [required: >=1.15.0, installed: 1.16.0] - greenlet [required: >=0.3, installed: 0.4.15] - monotonic [required: >=1.4, installed: 1.5] - six [required: >=1.10.0, installed: 1.12.0] factory-boy==2.12.0 - Faker [required: >=0.7.0, installed: 2.0.3] - python-dateutil [required: >=2.4, installed: 2.8.0] - six [required: >=1.5, installed: 1.12.0] - six [required: >=1.10, installed: 1.12.0] - text-unidecode [required: ==1.3, installed: 1.3] Flask-Admin==1.5.4 - Flask [required: >=0.7, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - wtforms [required: Any, installed: 2.2.1] Flask-Caching==1.7.2 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-CeleryExt==0.3.2 - celery [required: >=3.1, installed: 4.3.0] - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0] - kombu [required: >=4.4.0,<5.0, installed: 4.6.5] - amqp [required: ==2.5.1, installed: 2.5.1] - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0] - importlib-metadata [required: >=0.18, installed: 0.23] - zipp [required: >=0.5, installed: 0.6.0] - more-itertools [required: Any, installed: 7.2.0] - pytz [required: >dev, installed: 2019.3] - vine [required: >=1.3.0, installed: 1.3.0] - celery [required: >=4.3, installed: 4.3.0] - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0] - kombu [required: >=4.4.0,<5.0, installed: 4.6.5] - amqp [required: ==2.5.1, installed: 2.5.1] - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0] - importlib-metadata [required: >=0.18, installed: 0.23] - zipp [required: >=0.5, installed: 0.6.0] - more-itertools [required: Any, installed: 7.2.0] - pytz [required: >dev, installed: 2019.3] - vine [required: >=1.3.0, installed: 1.3.0] - Flask [required: >=0.10, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Cors==3.0.8 - Flask [required: >=0.9, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - Six [required: Any, installed: 1.12.0] Flask-Elasticsearch==0.2.5 - Elasticsearch [required: Any, installed: 7.0.5] - urllib3 [required: >=1.21.1, installed: 1.25.6] - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-JWT-Extended==3.23.0 - Flask [required: >=1.0, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - PyJWT [required: >=1.6.4, installed: 1.7.1] - six [required: Any, installed: 1.12.0] - Werkzeug [required: >=0.14, installed: 0.16.0] Flask-Limiter==1.1.0 - Flask [required: >=0.8, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - limits [required: Any, installed: 1.3] - six [required: >=1.4.1, installed: 1.12.0] - six [required: >=1.4.1, installed: 1.12.0] Flask-Login==0.4.1 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Migrate==2.5.2 - alembic [required: >=0.7, installed: 1.2.1] - Mako [required: Any, installed: 1.1.0] - MarkupSafe [required: >=0.9.2, installed: 1.1.1] - python-dateutil [required: Any, installed: 2.8.0] - six [required: >=1.5, installed: 1.12.0] - python-editor [required: >=0.3, installed: 1.0.4] - SQLAlchemy [required: >=1.1.0, installed: 1.3.10] - Flask [required: >=0.9, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1] - Flask [required: >=0.10, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - SQLAlchemy [required: >=0.8.0, installed: 1.3.10] flask-redis==0.4.0 - Flask [required: >=0.8, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - redis [required: >=2.7.6, installed: 3.3.11] Flask-REST-JSONAPI==0.12.6 - Flask [required: >=0.11, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - marshmallow [required: >=2.15.1,<3, installed: 2.15.6] - marshmallow-jsonapi [required: Any, installed: 0.22.0] - marshmallow [required: >=2.15.2, installed: 2.15.6] - six [required: Any, installed: 1.12.0] - sqlalchemy [required: Any, installed: 1.3.10] Flask-Script==2.0.6 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Scrypt==0.1.3.6 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - scrypt [required: Any, installed: 0.8.13] forex-python==1.5 - requests [required: Any, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] - simplejson [required: Any, installed: 3.16.0] geoip2==2.9.0 - maxminddb [required: >=1.4.0, installed: 1.5.1] - requests [required: >=2.9, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] google-compute-engine==2.8.13 - boto [required: Any, installed: 2.49.0] - distro [required: Any, installed: 1.4.0] - setuptools [required: Any, installed: 41.4.0] gunicorn==19.9.0 healthcheck==1.3.3 humanize==0.5.1 icalendar==4.0.3 - python-dateutil [required: Any, installed: 2.8.0] - six [required: >=1.5, installed: 1.12.0] - pytz [required: Any, installed: 2019.3] marrow.mailer==4.0.3 - marrow.util [required: <2.0, installed: 1.2.3] oauth2==1.9.0.post1 - httplib2 [required: Any, installed: 0.14.0] omise==0.8.1 - requests [required: >=2.12.1, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] paypalrestsdk==1.13.1 - pyopenssl [required: >=0.15, installed: 19.0.0] - cryptography [required: >=2.3, installed: 2.8] - cffi [required: >=1.8,!=1.11.3, installed: 1.13.0] - pycparser [required: Any, installed: 2.19] - six [required: >=1.4.1, installed: 1.12.0] - six [required: >=1.5.2, installed: 1.12.0] - requests [required: >=1.0.0, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] - six [required: >=1.0.0, installed: 1.12.0] pipdeptree==0.13.2 - pip [required: >=6.0.0, installed: 19.3] psycopg2-binary==2.8.3 pycountry==19.8.18 pycryptodome==3.9.0 python-dotenv==0.10.3 python-geoip-geolite2==2015.303 - python-geoip [required: Any, installed: 1.2] python-magic==0.4.15 python-pentabarf-xml==0.19 PyYAML==5.1.2 qrcode==6.1 - six [required: Any, installed: 1.12.0] requests-oauthlib==1.2.0 - oauthlib [required: >=3.0.0, installed: 3.1.0] - requests [required: >=2.0.0, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] sendgrid==6.1.0 - python-http-client [required: >=3.2.1, installed: 3.2.1] sentry-sdk==0.13.0 - certifi [required: Any, installed: 2019.9.11] - urllib3 [required: >=1.9, installed: 1.25.6] SQLAlchemy-Continuum==1.3.9 - SQLAlchemy [required: >=1.0.8, installed: 1.3.10] - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2] - six [required: Any, installed: 1.12.0] - SQLAlchemy [required: >=1.0, installed: 1.3.10] stripe==2.37.2 - requests [required: >=2.20, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] unicode-slugify==0.1.3 - six [required: Any, installed: 1.12.0] - unidecode [required: Any, installed: 1.1.1] wheel==0.33.6 xhtml2pdf==0.2.3 - html5lib [required: >=1.0, installed: 1.0.1] - six [required: >=1.9, installed: 1.12.0] - webencodings [required: Any, installed: 0.5.1] - Pillow [required: Any, installed: 6.2.0] - pyPdf2 [required: Any, installed: 1.26.0] - reportlab [required: >=3.0, installed: 3.5.31] - pillow [required: >=4.0.0, installed: 6.2.0] - six [required: Any, installed: 1.12.0] ``` 19.3.1 ``` Warning!!! Possibly conflicting dependencies found: * pyOpenSSL==19.0.0 - cryptography [required: >=2.3, installed: ?] * kombu==4.6.5 - importlib-metadata [required: >=0.18, installed: ?] * icalendar==4.0.3 - python-dateutil [required: Any, installed: ?] * Faker==2.0.3 - python-dateutil [required: >=2.4, installed: ?] * elasticsearch-dsl==7.0.0 - python-dateutil [required: Any, installed: ?] * arrow==0.15.2 - python-dateutil [required: Any, installed: ?] * alembic==1.2.1 - python-dateutil [required: Any, installed: ?] ------------------------------------------------------------------------ APScheduler==3.6.1 - pytz [required: Any, installed: 2019.3] - setuptools [required: >=0.7, installed: 41.4.0] - six [required: >=1.4.0, installed: 1.12.0] - tzlocal [required: >=1.2, installed: 2.0.0] - pytz [required: Any, installed: 2019.3] arrow==0.15.2 - python-dateutil [required: Any, installed: ?] bleach==3.1.0 - six [required: >=1.9.0, installed: 1.12.0] - webencodings [required: Any, installed: 0.5.1] blinker==1.4 cffi==1.13.0 - pycparser [required: Any, installed: 2.19] diff-match-patch==20181111 elasticsearch-dsl==7.0.0 - elasticsearch [required: >=7.0.0,<8.0.0, installed: 7.0.5] - urllib3 [required: >=1.21.1, installed: 1.25.6] - python-dateutil [required: Any, installed: ?] - six [required: Any, installed: 1.12.0] envparse==0.2.0 eventlet==0.25.1 - dnspython [required: >=1.15.0, installed: 1.16.0] - greenlet [required: >=0.3, installed: 0.4.15] - monotonic [required: >=1.4, installed: 1.5] - six [required: >=1.10.0, installed: 1.12.0] factory-boy==2.12.0 - Faker [required: >=0.7.0, installed: 2.0.3] - python-dateutil [required: >=2.4, installed: ?] - six [required: >=1.10, installed: 1.12.0] - text-unidecode [required: ==1.3, installed: 1.3] Flask-Admin==1.5.4 - Flask [required: >=0.7, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - wtforms [required: Any, installed: 2.2.1] Flask-Caching==1.7.2 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-CeleryExt==0.3.2 - celery [required: >=3.1, installed: 4.3.0] - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0] - kombu [required: >=4.4.0,<5.0, installed: 4.6.5] - amqp [required: ==2.5.1, installed: 2.5.1] - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0] - importlib-metadata [required: >=0.18, installed: ?] - pytz [required: >dev, installed: 2019.3] - vine [required: >=1.3.0, installed: 1.3.0] - celery [required: >=4.3, installed: 4.3.0] - billiard [required: >=3.6.0,<4.0, installed: 3.6.1.0] - kombu [required: >=4.4.0,<5.0, installed: 4.6.5] - amqp [required: ==2.5.1, installed: 2.5.1] - vine [required: >=1.1.3,<5.0.0a1, installed: 1.3.0] - importlib-metadata [required: >=0.18, installed: ?] - pytz [required: >dev, installed: 2019.3] - vine [required: >=1.3.0, installed: 1.3.0] - Flask [required: >=0.10, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Cors==3.0.8 - Flask [required: >=0.9, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - Six [required: Any, installed: 1.12.0] Flask-Elasticsearch==0.2.5 - Elasticsearch [required: Any, installed: 7.0.5] - urllib3 [required: >=1.21.1, installed: 1.25.6] - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-JWT-Extended==3.23.0 - Flask [required: >=1.0, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - PyJWT [required: >=1.6.4, installed: 1.7.1] - six [required: Any, installed: 1.12.0] - Werkzeug [required: >=0.14, installed: 0.16.0] Flask-Limiter==1.1.0 - Flask [required: >=0.8, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - limits [required: Any, installed: 1.3] - six [required: >=1.4.1, installed: 1.12.0] - six [required: >=1.4.1, installed: 1.12.0] Flask-Login==0.4.1 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Migrate==2.5.2 - alembic [required: >=0.7, installed: 1.2.1] - Mako [required: Any, installed: 1.1.0] - MarkupSafe [required: >=0.9.2, installed: 1.1.1] - python-dateutil [required: Any, installed: ?] - python-editor [required: >=0.3, installed: 1.0.4] - SQLAlchemy [required: >=1.1.0, installed: 1.3.10] - Flask [required: >=0.9, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - Flask-SQLAlchemy [required: >=1.0, installed: 2.4.1] - Flask [required: >=0.10, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - SQLAlchemy [required: >=0.8.0, installed: 1.3.10] Flask-REST-JSONAPI==0.12.6 - Flask [required: >=0.11, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - marshmallow [required: >=2.15.1,<3, installed: 2.15.6] - marshmallow-jsonapi [required: Any, installed: 0.22.0] - marshmallow [required: >=2.15.2, installed: 2.15.6] - six [required: Any, installed: 1.12.0] - sqlalchemy [required: Any, installed: 1.3.10] Flask-Script==2.0.6 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] Flask-Scrypt==0.1.3.6 - Flask [required: Any, installed: 1.1.1] - click [required: >=5.1, installed: 7.0] - itsdangerous [required: >=0.24, installed: 1.1.0] - Jinja2 [required: >=2.10.1, installed: 2.10.3] - MarkupSafe [required: >=0.23, installed: 1.1.1] - Werkzeug [required: >=0.15, installed: 0.16.0] - scrypt [required: Any, installed: 0.8.13] forex-python==1.5 - requests [required: Any, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] - simplejson [required: Any, installed: 3.16.0] geoip2==2.9.0 - maxminddb [required: >=1.4.0, installed: 1.5.1] - requests [required: >=2.9, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] google-compute-engine==2.8.13 - boto [required: Any, installed: 2.49.0] - distro [required: Any, installed: 1.4.0] - setuptools [required: Any, installed: 41.4.0] gunicorn==19.9.0 healthcheck==1.3.3 humanize==0.5.1 icalendar==4.0.3 - python-dateutil [required: Any, installed: ?] - pytz [required: Any, installed: 2019.3] marrow.mailer==4.0.3 - marrow.util [required: <2.0, installed: 1.2.3] more-itertools==7.2.0 oauth2==1.9.0.post1 - httplib2 [required: Any, installed: 0.14.0] omise==0.8.1 - requests [required: >=2.12.1, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] paypalrestsdk==1.13.1 - pyopenssl [required: >=0.15, installed: 19.0.0] - cryptography [required: >=2.3, installed: ?] - six [required: >=1.5.2, installed: 1.12.0] - requests [required: >=1.0.0, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] - six [required: >=1.0.0, installed: 1.12.0] pipdeptree==0.13.2 - pip [required: >=6.0.0, installed: 19.3.1] psycopg2-binary==2.8.3 pycountry==19.8.18 pycryptodome==3.9.0 python-dotenv==0.10.3 python-geoip-geolite2==2015.303 - python-geoip [required: Any, installed: 1.2] python-magic==0.4.15 python-pentabarf-xml==0.19 PyYAML==5.1.2 qrcode==6.1 - six [required: Any, installed: 1.12.0] redis==3.3.11 requests-oauthlib==1.2.0 - oauthlib [required: >=3.0.0, installed: 3.1.0] - requests [required: >=2.0.0, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] sendgrid==6.1.0 - python-http-client [required: >=3.2.1, installed: 3.2.1] sentry-sdk==0.13.0 - certifi [required: Any, installed: 2019.9.11] - urllib3 [required: >=1.9, installed: 1.25.6] SQLAlchemy-Continuum==1.3.9 - SQLAlchemy [required: >=1.0.8, installed: 1.3.10] - SQLAlchemy-Utils [required: >=0.30.12, installed: 0.34.2] - six [required: Any, installed: 1.12.0] - SQLAlchemy [required: >=1.0, installed: 1.3.10] stripe==2.37.2 - requests [required: >=2.20, installed: 2.22.0] - certifi [required: >=2017.4.17, installed: 2019.9.11] - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4] - idna [required: >=2.5,<2.9, installed: 2.8] - urllib3 [required: >=1.21.1,<1.26,!=1.25.1,!=1.25.0, installed: 1.25.6] unicode-slugify==0.1.3 - six [required: Any, installed: 1.12.0] - unidecode [required: Any, installed: 1.1.1] wheel==0.33.6 xhtml2pdf==0.2.3 - html5lib [required: >=1.0, installed: 1.0.1] - six [required: >=1.9, installed: 1.12.0] - webencodings [required: Any, installed: 0.5.1] - Pillow [required: Any, installed: 6.2.0] - pyPdf2 [required: Any, installed: 1.26.0] - reportlab [required: >=3.0, installed: 3.5.31] - pillow [required: >=4.0.0, installed: 6.2.0] - six [required: Any, installed: 1.12.0] ```

[pradyunsg]

@iamareebjamal Hi! I've edited your bug report to read more easily (shorter sentences, imperative tone) and to tone down some of the wording used.

Using all caps (equivalent to shouting via text) isn't a good move when you're asking for help from volunteers who do this stuff because they find it fun. :)

If you have any concerns with that, please file a new issue on pip or drop me an email on {username}@gmail.com.

[pradyunsg]

Looking at the diff between the release tags: https://github.com/pypa/pip/compare/19.3...19.3.1

I think what broke your usecase was the fact that we fixed a major bug in PEP 517 based builds. I'd have liked to have made that fix in 19.3 but I didn't have the bandwidth to and AFAICT, that fix was fairly innocuous. I'm not sure there's anything actionable now for pip, other than acknowledging that cached docker containers that were built earlier, should be invalidated since a bug in pip's build process was fixed which affects how we can use the cache.

Sorry if that sounds a little bleak but well, pip's used in so many ways that at this point, every non-refactoring patch breaks something, usually in ways that no one thought of when working on this stuff. 🤷🏻‍♂️

[chrahunt]

@iamareebjamal, in your Dockerfile please pass an explicit --prefix=/install argument to pip itself instead of --install-options='--prefix=/install'. Pip will internally update --install-options as well as take other necessary measures to ensure installed packages respect the --prefix argument. I tested this locally and the resulting image (containing dependencies installed with pip 19.3.1) had the same dependencies as the image built with dependencies installed with 19.3. You'll probably also want to pass --no-warn-script-location to reduce some of the noise which is irrelevant to your use case.

There was a bug fixed in #6606 (released in 19.3.1) as mentioned by @pradyunsg that now correctly builds projects that use PEP 517 without implicitly going through setuptools. This fix was causing an issue in your case because the packages that were now using PEP 517 were not picking up the prefix directory passed via --install-options.

[iamareebjamal]

Sorry if it felt like I was using caps to emphasize or shout. I was simply referring to MAJOR.MINOR.PATCH in semver context, not for shouting

[iamareebjamal]

@chrahunt If pip indeed follows semantic versioning, then its behaviour should not change between patch updates. I can add the fix from my end for sure, but this is considered a breaking change, and hence should be communicated and moved to a minor release, rather than a patch one. I understand that a major release will not be practical, but even minor releases warrant backward compatibility with feature additions. And patch updates should only fix bugs without compromising any previous behaviour

About action that pip can take, I don't know if this would be considered rude or impossible as well, but most other tool authors create a new patch version with behaviour restored to surprise the least amount of users, and then introduce the fix with a minor update

Reference: https://github.com/stripe/stripe-python/issues/582

[pfmoore]

@iamareebjamal pip does not follow semantic versioning, we use calendar based versioning. See the documentation for details. Specifically, version 19.3.1 is the first bugfix release of the 3rd release in 2019.

[iamareebjamal]

Well, that's sad :( Documentation update of 19.3.1 won't notify people of the issue. And they may spend days debugging the issue and may not even be able to source it to pip. This will cause a waste of time and stress. I know something might break in any patch version of pip because of its vast use case surface area, but the next section in the documentation you linked talks about Deprecation Policy

It says that pip will start issuing warnings for 6 months before changing public visible behaviour. https://pip.pypa.io/en/latest/development/release-process/#deprecation-policy

That is awesome! Because it is stricter than semantic versioning and would align with pip's own guidelines as well. And also follows the principle of least surprises. Now, it may happen that this behaviour was not documented, but it is widely used. If the policy is followed and respected, it's great for both pip and its users. If not, then it is a dependency hell :( https://semver.org/#introduction

[chrahunt]

Hi @iamareebjamal. One thing I think would be completely reasonable (and that we've done in the past) is to make (and pin, if common enough) a canonical issue to communicate the best information we have on how to identify and remedy the problem. This should get picked up by search engines too, reducing any burden.

To help in creating that, can you provide information on the origin of --install-option in your Dockerfile? Another issue #7253 seems similar enough that maybe they are from the same place (blog post, SO post, etc) and maybe we could take some wording from that or change it at the source.

[iamareebjamal]

Sounds good enough. The maintainer who added the --install-option flag is no longer available but I'll try to contact him and search on my own as well for articles mentioning this behavior.

Thanks for the help

[hugovk]

Is this issue still valid or can it be closed?

[pradyunsg]

This can be closed!