search

pypa / pipenv

Python Development Workflow for Humans.
https://pipenv.pypa.io
MIT License
24.87k stars 1.87k forks source link

using different source achieve different result when execute pipenv lock #1701

Closed threegirl2014 closed 6 years ago

threegirl2014 commented 6 years ago

ENV: Pipenv version: '11.6.0'

Pipenv location: '/Library/Python/2.7/site-packages/pipenv'

PEP 508 Information:

{'implementation_name': 'cpython',
 'implementation_version': '0',
 'os_name': 'posix',
 'platform_machine': 'x86_64',
 'platform_python_implementation': 'CPython',
 'platform_release': '15.0.0',
 'platform_system': 'Darwin',
 'platform_version': 'Darwin Kernel Version 15.0.0: Sat Sep 19 15:53:46 PDT 2015; root:xnu-3247.10.11~1/RELEASE_X86_64',
 'python_full_version': '2.7.10',
 'python_version': '2.7',
 'sys_platform': 'darwin'}

the requirements.txt is this:

elasticsearch-dsl
requests

when pipfile is this:

[[source]]
url = "http://mirrors.aliyun.com/pypi/simple/"
name = "aliyun"

[dev-packages]

[packages]

elasticsearch-dsl = "*"
requests = "*"

[requires]

python_version = "2.7"

excute pipenv lock --verbose, the result is wrong:

(test_pipenv2-i1pi3f19) ☁  test_pipenv2  pipenv lock --verbose
Locking [dev-packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  elasticsearch-dsl
  requests

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  elasticsearch-dsl
  requests

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches elasticsearch-dsl
Tried: (no version found at all)
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches elasticsearch-dsl
Tried: (no version found at all)

however, if I change the Pipfile like this(change the source):

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

elasticsearch-dsl = "*"
requests = "*"

[requires]

python_version = "2.7"

or just add one extra source:

[[source]]
url = "http://mirrors.aliyun.com/pypi/simple/"
name = "aliyun"

[[source]]
url = "https://pypi.python.org/simple"
verify_ssl = true
name = "pypi"

[dev-packages]

[packages]

elasticsearch-dsl = "*"
requests = "*"

[requires]

python_version = "2.7"

the result is right:

(test_pipenv2-i1pi3f19) ☁  test_pipenv2  pipenv lock --verbose
Locking [dev-packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i https://pypi.python.org/simple

                          ROUND 1
Current constraints:
  elasticsearch-dsl
  requests

Finding the best candidates:
  found candidate elasticsearch-dsl==6.1.0 (constraint was <any>)
  found candidate requests==2.18.4 (constraint was <any>)

Finding secondary dependencies:
  elasticsearch-dsl==6.1.0  requires elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six
  requests==2.18.4          requires certifi>=2017.4.17, chardet<3.1.0,>=3.0.2, idna<2.7,>=2.5, urllib3<1.23,>=1.21.1

New dependencies found in this round:
  adding [u'certifi', '>=2017.4.17', '[]']
  adding [u'chardet', '<3.1.0,>=3.0.2', '[]']
  adding [u'elasticsearch', '<7.0.0,>=6.0.0', '[]']
  adding [u'idna', '<2.7,>=2.5', '[]']
  adding [u'ipaddress', '', '[]']
  adding [u'python-dateutil', '', '[]']
  adding [u'six', '', '[]']
  adding [u'urllib3', '<1.23,>=1.21.1', '[]']
Removed dependencies in this round:
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 1: not stable

                          ROUND 2
Current constraints:
  certifi>=2017.4.17
  chardet<3.1.0,>=3.0.2
  elasticsearch<7.0.0,>=6.0.0
  elasticsearch-dsl
  idna<2.7,>=2.5
  ipaddress
  python-dateutil
  requests
  six
  urllib3<1.23,>=1.21.1

Finding the best candidates:
  found candidate certifi==2018.1.18 (constraint was >=2017.4.17)
  found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0)
  found candidate elasticsearch==6.1.1 (constraint was >=6.0.0,<7.0.0)
  found candidate elasticsearch-dsl==6.1.0 (constraint was <any>)
  found candidate idna==2.6 (constraint was >=2.5,<2.7)
  found candidate ipaddress==1.0.19 (constraint was <any>)
  found candidate python-dateutil==2.7.0 (constraint was <any>)
  found candidate requests==2.18.4 (constraint was <any>)
  found candidate six==1.11.0 (constraint was <any>)
  found candidate urllib3==1.22 (constraint was >=1.21.1,<1.23)

Finding secondary dependencies:
  ipaddress==1.0.19         requires -
  elasticsearch==6.1.1      requires urllib3<1.23,>=1.21.1
  elasticsearch-dsl==6.1.0  requires elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six
  certifi==2018.1.18        requires -
  six==1.11.0               requires -
  chardet==3.0.4            requires -
  idna==2.6                 requires -
  urllib3==1.22             requires -
  requests==2.18.4          requires certifi>=2017.4.17, chardet<3.1.0,>=3.0.2, idna<2.7,>=2.5, urllib3<1.23,>=1.21.1
  python-dateutil==2.7.0    requires six>=1.5, six>=1.5

New dependencies found in this round:
  adding [u'six', '>=1.5', '[]']
Removed dependencies in this round:
  removing [u'six', '', '[]']
Unsafe dependencies in this round:
------------------------------------------------------------
Result of round 2: not stable

                          ROUND 3
Current constraints:
  certifi>=2017.4.17
  chardet<3.1.0,>=3.0.2
  elasticsearch<7.0.0,>=6.0.0
  elasticsearch-dsl
  idna<2.7,>=2.5
  ipaddress
  python-dateutil
  requests
  six>=1.5
  urllib3<1.23,>=1.21.1

Finding the best candidates:
  found candidate certifi==2018.1.18 (constraint was >=2017.4.17)
  found candidate chardet==3.0.4 (constraint was >=3.0.2,<3.1.0)
  found candidate elasticsearch==6.1.1 (constraint was >=6.0.0,<7.0.0)
  found candidate elasticsearch-dsl==6.1.0 (constraint was <any>)
  found candidate idna==2.6 (constraint was >=2.5,<2.7)
  found candidate ipaddress==1.0.19 (constraint was <any>)
  found candidate python-dateutil==2.7.0 (constraint was <any>)
  found candidate requests==2.18.4 (constraint was <any>)
  found candidate six==1.11.0 (constraint was >=1.5)
  found candidate urllib3==1.22 (constraint was >=1.21.1,<1.23)

Finding secondary dependencies:
  requests==2.18.4          requires certifi>=2017.4.17, chardet<3.1.0,>=3.0.2, idna<2.7,>=2.5, urllib3<1.23,>=1.21.1
  elasticsearch==6.1.1      requires urllib3<1.23,>=1.21.1
  certifi==2018.1.18        requires -
  idna==2.6                 requires -
  urllib3==1.22             requires -
  elasticsearch-dsl==6.1.0  requires elasticsearch<7.0.0,>=6.0.0, ipaddress, python-dateutil, six
  six==1.11.0               requires -
  chardet==3.0.4            requires -
  ipaddress==1.0.19         requires -
  python-dateutil==2.7.0    requires six>=1.5, six>=1.5
------------------------------------------------------------
Result of round 3: stable, done

Updated Pipfile.lock (97f2b9)!

and the Pipfile.lock is:

(test_pipenv2-i1pi3f19) ☁  test_pipenv2  cat Pipfile.lock
{
    "_meta": {
        "hash": {
            "sha256": "25a37ce64cbbb9b441cdc232b67eb7e9d4475803610111abfcf7582b8297f2b9"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "2.7"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.python.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "certifi": {
            "hashes": [
                "sha256:14131608ad2fd56836d33a71ee60fa1c82bc9d2c8d98b7bdbc631fe1b3cd1296",
                "sha256:edbc3f203427eef571f79a7692bb160a2b0f7ccaa31953e99bd17e307cf63f7d"
            ],
            "version": "==2018.1.18"
        },
        "chardet": {
            "hashes": [
                "sha256:84ab92ed1c4d4f16916e05906b6b75a6c0fb5db821cc65e70cbd64a3e2a5eaae",
                "sha256:fc323ffcaeaed0e0a02bf4d117757b98aed530d9ed4531e3e15460124c106691"
            ],
            "version": "==3.0.4"
        },
        "elasticsearch": {
            "hashes": [
                "sha256:307055861d0290b830bd1ec4b82d41ce0f19f6a4899635956bd16bc61e3e90b1",
                "sha256:8d91a3fce12123a187b673f18c23bcffa6e7b49ba057555d59eeeded0ba15dce"
            ],
            "version": "==6.1.1"
        },
        "elasticsearch-dsl": {
            "hashes": [
                "sha256:5114a38a88e93a4663782eae07a1e8084ba333c49887335c83de8b8043bc72b2",
                "sha256:d6d974cd2289543a3350690494a43fe9996485b8dc6f1d8758cb56bee01244bd"
            ],
            "version": "==6.1.0"
        },
        "idna": {
            "hashes": [
                "sha256:2c6a5de3089009e3da7c5dde64a141dbc8551d5b7f6cf4ed7c2568d0cc520a8f",
                "sha256:8c7309c718f94b3a625cb648ace320157ad16ff131ae0af362c9f21b80ef6ec4"
            ],
            "version": "==2.6"
        },
        "ipaddress": {
            "hashes": [
                "sha256:200d8686011d470b5e4de207d803445deee427455cd0cb7c982b68cf82524f81"
            ],
            "version": "==1.0.19"
        },
        "python-dateutil": {
            "hashes": [
                "sha256:07009062406cffd554a9b4135cd2ff167c9bf6b7aac61fe946c93e69fad1bbd8",
                "sha256:8f95bb7e6edbb2456a51a1fb58c8dca942024b4f5844cae62c90aa88afe6e300"
            ],
            "version": "==2.7.0"
        },
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "version": "==2.18.4"
        },
        "six": {
            "hashes": [
                "sha256:70e8a77beed4562e7f14fe23a786b54f6296e34344c23bc42f07b15018ff98e9",
                "sha256:832dc0e10feb1aa2c68dcc57dbb658f1c7e65b9b61af69048abc87a2db00a0eb"
            ],
            "version": "==1.11.0"
        },
        "urllib3": {
            "hashes": [
                "sha256:06330f386d6e4b195fbfc736b297f58c5a892e4440e54d294d7004e3a9bbea1b",
                "sha256:cc44da8e1145637334317feebd728bd869a35285b93cbb4cca2577da7e62db4f"
            ],
            "version": "==1.22"
        }
    },
    "develop": {}
}

while, the pipenv graph execute always correctly:

(test_pipenv2-i1pi3f19) ☁  test_pipenv2  pipenv graph
elasticsearch-dsl==6.1.0
  - elasticsearch [required: <7.0.0,>=6.0.0, installed: 6.1.1]
    - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]
  - ipaddress [required: Any, installed: 1.0.19]
  - python-dateutil [required: Any, installed: 2.7.0]
    - six [required: >=1.5, installed: 1.11.0]
  - six [required: Any, installed: 1.11.0]
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: 2018.1.18]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.7, installed: 2.6]
  - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

so, what is the problem or what should I do?

kennethreitz commented 6 years ago

is elasticsearch-dsl on your pypi server?

techalchemy commented 6 years ago

Wait is this a bug? It looks like one package is hosted locally and it’s dependencies are on pypi. Both servers need to be in your pipfile.

threegirl2014 commented 6 years ago

yes, of course. @kennethreitz this is the only requests situation:

(test_pipenv3-0zRcCNVK) zhangrujia@zhangrujias-MacBook-Pro:~/script/test_pipenv3 » pipenv lock --verbose                   1 ↵
Locking [dev-packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  requests

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  requests

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches requests
Tried: (no version found at all)
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches requests
Tried: (no version found at all)

well, when I use pip install xxx, this pypi source works correctly.

kennethreitz commented 6 years ago

is requests on your PyPI server?

threegirl2014 commented 6 years ago 
└[~/script/test_pipenv4]> pipenv install
Pipfile.lock not found, creating…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches requests
Tried: (no version found at all)

/Library/Python/2.7/site-packages/pipenv/utils.py:1157: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/var/folders/x4/fpjn14k11gdfw96th7bg7z4c0000gn/T/pipenv-WBSDme-requirements'>
  warnings.warn(warn_message, ResourceWarning)
└[~/script/test_pipenv4]> pipenv install --skip-lock
Installing dependencies from Pipfile…
  🐍   ▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉▉ 1/1 — 00:00:03

only the lock function be affected, the installing always give me result. @kennethreitz

threegirl2014 commented 6 years ago 
(test_pipenv4-fmshV4Om) script/test_pipenv4 » pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
certifi (2018.1.18)
chardet (3.0.4)
idna (2.6)
pip (9.0.1)
requests (2.18.4)
setuptools (38.5.2)
urllib3 (1.22)
wheel (0.30.0)
(test_pipenv4-fmshV4Om) script/test_pipenv4 » pipenv graph
requests==2.18.4
  - certifi [required: >=2017.4.17, installed: 2018.1.18]
  - chardet [required: >=3.0.2,<3.1.0, installed: 3.0.4]
  - idna [required: >=2.5,<2.7, installed: 2.6]
  - urllib3 [required: <1.23,>=1.21.1, installed: 1.22]

they are all in my enviroment. @techalchemy

threegirl2014 commented 6 years ago

when the package is only a urllib3 which does not have sub-dependences, the lock function also gives me this message:

>> pipenv lock --verbose                                                                                                         21:06.23 二  3 13 2018 >>>
Locking [dev-packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:

Finding the best candidates:

Finding secondary dependencies:
------------------------------------------------------------
Result of round 1: stable, done

Locking [packages] dependencies…
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  urllib3

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.
Using pip: -i http://mirrors.aliyun.com/pypi/simple/

                          ROUND 1
Current constraints:
  urllib3

Finding the best candidates:
Please check your version specifier and version number. See PEP440 for more information.

Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches urllib3
Tried: (no version found at all)
Warning: Your dependencies could not be resolved. You likely have a mismatch in your sub-dependencies.
  You can use $ pipenv install --skip-lock to bypass this mechanism, then run $ pipenv graph to inspect the situation.
Could not find a version that matches urllib3
Tried: (no version found at all)

@kennethreitz @techalchemy

kennethreitz commented 6 years ago

this appears to be fixed in master (using bandersnatch as a mirror)

threegirl2014 commented 6 years ago

I upgrade pipenv to the latest version 11.7.1. but this problem doesn't be fixed.

so I try to find something in the raw code:

def prepare_pip_source_args(sources, pip_args=None):
    if pip_args is None:
        pip_args = []

    if sources:
        # Add the source to pip9.
        pip_args.extend(['-i', sources[0]['url']])

        # Trust the host if it's not verified.
        if not sources[0].get('verify_ssl', True):
            pip_args.extend(['--trusted-host', urlparse(sources[0]['url']).netloc.split(':')[0]])

        # Add additional sources as extra indexes.
        if len(sources) > 1:
            for source in sources[1:]:
                pip_args.extend(['--extra-index-url', source['url']])

                # Trust the host if it's not verified.
                if not source.get('verify_ssl', True):
                    pip_args.extend(['--trusted-host', urlparse(source['url']).netloc.split(':')[0]])

    return pip_args

I set the verfiy_ssl=false in the Pipfile when the source url is http://mirrors.aliyun.com/pypi/simple/. and it works!

maybe this message can help you.

kennethreitz commented 6 years ago

Yes, that is a requirement :)