search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.46k stars 3k forks source link

`pip install --no-binary :all:` phones home to Fastly/pypi when pip.conf index-url set #6503

Closed cooperlees closed 1 year ago

cooperlees commented 5 years ago

Environment

Can try in Python 3.7 if required, but I'd expect that same behavior here.

Description With a pip.conf in the root of a venv to point at a PEP381 mirror it seems some sdist builds won't work if they can't phone home to https://pypi.org/. I'd like to see if this can be optional / data can be pulled from a PEP381 mirror.

Expected behavior Fetch the data from the local mirror.

How to Reproduce

  1. Build a PEP381 mirror - Potentially using bandersnatch
  2. Create a venv
  3. Add in a pip.conf like above
  4. pip install --no-binary ':all:' coverage black mypy flake8 pylint

Output

Stdout:

(test_gcc_pars) [cooper:~/meta-terragraph:( (master))]$ pip install --no-binary ':all:' coverage black mypy flake8 pylint
Looking in indexes: https://pypi.facebook.com/simple, https://pypi.facebook.com/internal
Collecting coverage
  Using cached https://pypi.facebook.com/packages/82/70/2280b5b29a0352519bb95ab0ef1ea942d40466ca71c53a2085bdeff7b0eb/coverage-4.5.3.tar.gz
Collecting black
  Using cached https://pypi.facebook.com/packages/89/07/aebb10fb8f2ffbac672dfbebffa724643bc84cf012a57737a622d1dabddb/black-19.3b0.tar.gz
Collecting mypy
  Downloading https://pypi.facebook.com/packages/ba/57/302f578708db7e059e793888238971f78b9219cd55f44148b435d71831c0/mypy-0.701.tar.gz (1.9MB)
     |████████████████████████████████| 1.9MB 56.1MB/s
Collecting flake8
  Downloading https://pypi.facebook.com/packages/23/e7/80626da76ff2b4c94ac9bcd92898a1011d1c891e0ba1343f24109923462d/flake8-3.7.7.tar.gz (148kB)
     |████████████████████████████████| 153kB 57.6MB/s
Collecting pylint
  Downloading https://pypi.facebook.com/packages/01/8b/538911c0ebc2529f15004f4cb07e3ca562bb9aacea5df89cc25b62e01891/pylint-2.3.1.tar.gz (577kB)
     |████████████████████████████████| 583kB 54.2MB/s
Collecting click>=6.5 (from black)
  Using cached https://pypi.facebook.com/packages/f8/5c/f60e9d8a1e77005f664b76ff8aeaee5bc05d0a91798afd7f53fc998dbc47/Click-7.0.tar.gz
Collecting attrs>=18.1.0 (from black)
  Using cached https://pypi.facebook.com/packages/cc/d9/931a24cc5394f19383fbbe3e1147a0291276afa43a0dc3ed0d6cd9fda813/attrs-19.1.0.tar.gz
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
    Preparing wheel metadata ... done
Collecting appdirs (from black)
  Using cached https://pypi.facebook.com/packages/48/69/d87c60746b393309ca30761f8e2b49473d43450b150cb08f3c6df5c11be5/appdirs-1.4.3.tar.gz
Collecting toml>=0.9.4 (from black)
  Using cached https://pypi.facebook.com/packages/b9/19/5cbd78eac8b1783671c40e34bb0fa83133a06d340a38b55c645076d40094/toml-0.10.0.tar.gz
Collecting typed-ast<1.4.0,>=1.3.1 (from mypy)
  Using cached https://pypi.facebook.com/packages/d3/b1/959c3ed4a9cc100feba7ad1a7d6336d8888937ee89f4a577f7698e09decd/typed-ast-1.3.5.tar.gz
Collecting mypy_extensions<0.5.0,>=0.4.0 (from mypy)
  Using cached https://pypi.facebook.com/packages/c2/92/3cc05d1206237d54db7b2565a58080a909445330b4f90a6436302a49f0f8/mypy_extensions-0.4.1.tar.gz
Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Downloading https://pypi.facebook.com/packages/b4/ef/063484f1f9ba3081e920ec9972c96664e2edb9fdc3d8669b0e3b8fc0ad7c/entrypoints-0.3.tar.gz
  Installing build dependencies ... /

Error:

Collecting entrypoints<0.4.0,>=0.3.0 (from flake8)
  Downloading https://pypi.facebook.com/packages/b4/ef/063484f1f9ba3081e920ec9972c96664e2edb9fdc3d8669b0e3b8fc0ad7c/entrypoints-0.3.tar.gz
  Installing build dependencies ... error
  ERROR: Complete output from command /home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 /home/cooper/virtualenvs/test_gcc_pars/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5a8v3xw9/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.facebook.com/simple --extra-index-url https://pypi.facebook.com/internal -- flit:
  ERROR: Looking in indexes: https://pypi.facebook.com/simple, https://pypi.facebook.com/internal, https://pypi.facebook.com/internal
  Collecting flit
    Using cached https://pypi.facebook.com/packages/1f/87/9ea76ab4cdf1fd36710d9688ec36a0053067c47e753b32272f952ff206c5/flit-1.3.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: still running...
    Getting requirements to build wheel: finished with status 'done'
    Installing backend dependencies: started
    Installing backend dependencies: finished with status 'done'
      Preparing wheel metadata: started
      Preparing wheel metadata: still running...
      Preparing wheel metadata: finished with status 'done'
  Collecting docutils (from flit)
    Using cached https://pypi.facebook.com/packages/84/f4/5771e41fdf52aabebbadecc9381d11dea0fa34e4759b4071244fa094804c/docutils-0.14.tar.gz
  Collecting requests (from flit)
    Using cached https://pypi.facebook.com/packages/52/2c/514e4ac25da2b08ca5a464c50463682126385c4272c18193876e91f4bc38/requests-2.21.0.tar.gz
  Collecting pytoml (from flit)
    Using cached https://pypi.facebook.com/packages/35/35/da1123673c54b6d701453fcd20f751d6a1fae43339b3993ae458875576e4/pytoml-0.1.20.tar.gz
  Collecting chardet<3.1.0,>=3.0.2 (from requests->flit)
    Using cached https://pypi.facebook.com/packages/fc/bb/a5768c230f9ddb03acc9ef3f0d4a3cf93462473795d18e9535498c8f929d/chardet-3.0.4.tar.gz
  Collecting idna<2.9,>=2.5 (from requests->flit)
    Using cached https://pypi.facebook.com/packages/ad/13/eb56951b6f7950cadb579ca166e448ba77f9d24efc03edd7e55fa57d04b7/idna-2.8.tar.gz
  Collecting urllib3<1.25,>=1.21.1 (from requests->flit)
    Using cached https://pypi.facebook.com/packages/8a/3c/1bb7ef6c435dea026f06ed9f3ba16aa93f9f4f5d3857a51a35dfa00882f1/urllib3-1.24.3.tar.gz
  Collecting certifi>=2017.4.17 (from requests->flit)
    Using cached https://pypi.facebook.com/packages/06/b8/d1ea38513c22e8c906275d135818fee16ad8495985956a9b7e2bb21942a1/certifi-2019.3.9.tar.gz
  Skipping bdist_wheel for flit, due to binaries being disabled for it.
  Installing collected packages: docutils, chardet, idna, urllib3, certifi, requests, pytoml, flit
    Running setup.py install for docutils: started
      Running setup.py install for docutils: finished with status 'done'
    Running setup.py install for chardet: started
      Running setup.py install for chardet: finished with status 'done'
    Running setup.py install for idna: started
      Running setup.py install for idna: finished with status 'done'
    Running setup.py install for urllib3: started
      Running setup.py install for urllib3: finished with status 'done'
    Running setup.py install for certifi: started
      Running setup.py install for certifi: finished with status 'done'
    Running setup.py install for requests: started
      Running setup.py install for requests: finished with status 'done'
    Running setup.py install for pytoml: started
      Running setup.py install for pytoml: finished with status 'done'
    Running setup.py install for flit: started
      Running setup.py install for flit: finished with status 'error'
      ERROR: Complete output from command /home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-r_lakb_r/flit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rqmvxt2r/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-5a8v3xw9/overlay --compile --install-headers /home/cooper/virtualenvs/test_gcc_pars/include/site/python3.6/flit:
      ERROR: Traceback (most recent call last):
        File "<string>", line 1, in <module>
      ModuleNotFoundError: No module named 'setuptools'
      ----------------------------------------
  ERROR: Command "/home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 -u -c 'import setuptools, tokenize;__file__='"'"'/tmp/pip-install-r_lakb_r/flit/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-rqmvxt2r/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-5a8v3xw9/overlay --compile --install-headers /home/cooper/virtualenvs/test_gcc_pars/include/site/python3.6/flit" failed with error code 1 in /tmp/pip-install-r_lakb_r/flit/
  ----------------------------------------
ERROR: Command "/home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 /home/cooper/virtualenvs/test_gcc_pars/lib/python3.6/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-5a8v3xw9/overlay --no-warn-script-location --no-binary :all: --only-binary :none: -i https://pypi.facebook.com/simple --extra-index-url https://pypi.facebook.com/internal -- flit" failed with error code 1 in None

/tmp file I see that gets created

/home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 /home/cooper/virtualenvs/test_gcc_pars/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmps9q1n60s

[cooper:~:]$ cat /tmp/tmps9q1n60s/input.json
{
  "kwargs": {
    "metadata_directory": "/tmp/pip-install-r_lakb_r/flit/pip-wheel-metadata",
    "config_settings": null
  }
}

strace + tcpdump

09:13:58.138005 IP6 2401:db00:21:6048:face:0:b:0.37130 > 2a04:4e42::223.443: Flags [S], seq 4050040434, win 28800, options [mss 1440,sackOK,TS val 1093309550 ecr 0,nop,wscale 10], length 0

[cooper:~:]$ strace -p 86617
strace: Process 86617 attached
connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2a04:4e42:200::223", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28) = -1 ETIMEDOUT (Connection timed out)
close(3)                                = 0
socket(AF_INET6, SOCK_STREAM|SOCK_CLOEXEC, IPPROTO_TCP) = 3
setsockopt(3, SOL_TCP, TCP_NODELAY, [1], 4) = 0
ioctl(3, FIONBIO, [0])                  = 0
connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2a04:4e42:600::223", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28
pfmoore commented 5 years ago

Can you reduce the example to the install of the single package that's failing? I can't work out from your output what's happening. I have a suspicion that this might be related to something using setup_requires, but your example is installing at least 8 different packages, and I'm not going to check each one in turn :-(

cooperlees commented 5 years ago

Rightio. Have been playing more. Seems just flake8 getting to a block on trying to call home.

Seems to get stuck Cleaning up:

[cooper:~:]$ strace -p 313601
strace: Process 313601 attached
connect(3, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "2a04:4e42::223", &sin6_addr), sin6_flowinfo=0, sin6_scope_id=0}, 28^Cstrace: Process 313601

...

    Successfully installed certifi-2019.3.9 chardet-3.0.4 docutils-0.14 idna-2.8 intreehooks-1.0 pytoml-0.1.20 requests-2.21.0 urllib3-1.24.3
    Cleaning up...
    Cleaned build tracker '/tmp/pip-req-tracker-lcudynom'
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Running command /home/cooper/virtualenvs/test_gcc_pars/bin/python3.6 /home/cooper/virtualenvs/test_gcc_pars/lib/python3.6/site-packages/pip/_vendor/pep517/_in_process.py get_requires_for_build_wheel /tmp/tmp02kkwgy7
    The file description seems not to be valid rst for PyPI; it will be interpreted as plain text
    <string>:25: (WARNING/2) Cannot analyze code. Pygments package not found.
cooperlees commented 5 years ago

Seems to be entrypoints - It's using a toml + depends on flit: https://github.com/takluyver/entrypoints

flit is also all toml'd.

pradyunsg commented 1 year ago

it seems some sdist builds won't work if they can't phone home to https://pypi.org/.

So, it seems like this is happening because those builds are reaching out to the https://pypi.org.

There's basically nothing that pip can do here -- we don't control the build process of package sdists, and the best move here is to try and identify which package is trying to reach out to PyPI (for checking the description, based on the output from the above) and requesting them to remove that bit since it's problematic to your workflow.

Closing this out, since I don't think there's anything we can do for this on pip's end basically.