search

pypa / pip

The Python package installer
https://pip.pypa.io/
MIT License
9.45k stars 3k forks source link

`pip list -o` doesn't signal a very old pre-existing lock file blocking itself #6238

Closed stonebig closed 5 years ago

stonebig commented 5 years ago

Environment

Description pip list -o doesn't work, seems waiting something forever

Output

pip list -o

is it normal ? has anyone else the problem ? If not, where could it come from ?

stonebig commented 5 years ago

update:

uranusjr commented 5 years ago

Are you using Command Prompt (cmd.exe)? Could you try whether Powershell is able to output? I have no evidence from the top of my head and a casual Google search, but I suspect there is a system restriction on print length.

uranusjr commented 5 years ago

Note: I’m having a lot of trouble trying to reproduce the above setup (many difficult-to-install and ill-bahaving packages). It would be extremely helpful if there is a more straightforward way to experiment on this…

stonebig commented 5 years ago

Indeed some package are compiled from master branch

Reproducing today with previous pip-18.1, I have the same issue.

How a package could make pip list -o "stall" ? maybe I have a package not yet in pypi, in this list

stonebig commented 5 years ago

removing per alphabetical order till it works

pip uninstall -y  absl-py adodbapi alabaster algopy altair   altair-widgets appdirs asciitree asteval astor astroid astroML astropy   atomicwrites attrs

pip uninstall -y                 Babel backcall backports-abc   baresql bcolz beautifulsoup4 bleach blosc bloscpack bokeh Bottleneck bqplot   branca brewer2mpl Brotli

pip uninstall -y   Cartopy certifi cffi cftime chardet Click   cloudpickle clrmagic colorama colorcet comtypes cvxopt cvxpy cx-Freeze cycler   cyordereddict Cython cytoolz dask dask-searchcv datashader datashape

pip uninstall -y   db.py decorator defusedxml descartes dill   distributed docopt docrepr docutils ecos edward emcee entrypoints   fast-histogram fastcache fasteners fastparquet feather-format Flask   formlayout future fuzzywuzzy

pip uninstall -y   gast GDAL geographiclib geopy geoviews   gmpy2 greenlet grpcio guidata guiqwt h5py HeapDict holoviews html5lib husl   hvplot idlex idna imageio imagesize imbalanced-learn intake ipydatawidgets   ipykernel ipyleaflet ipympl ipyparallel ipython ipython-genutils ipython-sql   ipywidgets isort itsdangerous jedi Jinja2 joblib jsonschema julia jupyter   jupyter-client jupyter-console jupyter-core jupyter-sphinx jupyterlab   jupyterlab-launcher jupyterlab-server

pip uninstall -y   Keras Keras-Applications   Keras-Preprocessing keras-vis keyring kiwisolver lazy-object-proxy llvmlite   lmfit locket loky lxml Markdown MarkupSafe matplotlib mccabe metakernel   mistune mizani mkl-service mlxtend monotonic more-itertools moviepy   mpl-scatter-density mpld3 mpldatacursor mpmath msgpack msgpack-numpy   msgpack-python multipledispatch multiprocess mypy mypy-extensions   mysql-connector-python nbconvert nbconvert-reportlab nbformat netCDF4 networkx   nltk notebook numba numcodecs numdifftools numexpr numpy numpydoc oct2py   octave-kernel osqp packaging palettable pandas pandas-datareader   pandocfilters panel param parambokeh paramnb parso partd patsy pdfrw pdvega   pep8 pexpect pg8000 pickleshare Pillow

pip uninstall -y   plotnine pluggy ppci prettytable   prometheus-client prompt-toolkit protobuf psutil ptpython PuLP py pyarrow   PyAudio pybars3 pybind11 pycodestyle pycparser pyct pyflakes pyflux pygame   pygbm Pygments pylint pymc PyMeta3 pymongo pyodbc PyOpenGL pypandoc pyparsing   PyQt5 pyqtgraph pyserial pyshp pystache pytest pytest-runner python-dateutil   python-hdf4 python-Levenshtein python-snappy pythonnet PythonQwt pythreejs   pytz pyviz-comms PyWavelets pywin32 pywin32-ctypes pywinpty pywinusb PyYAML   pyzmq pyzo

pip uninstall -y   QtAwesome qtconsole QtPy readme-renderer   redis regex reportlab requests requests-toolbelt rise rope rpy2 ruamel.yaml   Rx scikit-fuzzy scikit-image scikit-learn scikit-optimize scilab2py scipy scs   seaborn Send2Trash

pip uninstall -y    Shapely simplegeneric simplejson simpy   singledispatch sip six snakeviz snowballstemmer sortedcontainers sounddevice   soupsieve Sphinx sphinx-rtd-theme sphinxcontrib-websupport spyder   spyder-kernels SQLAlchemy sqlite-bro sqlparse statsmodels streamz   supersmoother sympy tables tblib tensorboard tensorflow   tensorflow-probability termcolor terminado

pip uninstall -y   thrift toolz torch torchvision tornado   tqdm traitlets traittypes twine twitter typed-ast tzlocal uncertainties   urllib3 vega vega-datasets ViTables wcwidth webencodings Werkzeug

pip uninstall -y   widgetsnbextension  wordcloud wrapt xarray xlrd XlsxWriter   xlwings zarr zict
--
and at last removal round, only,  it works...

pip list -o
Package    Version Latest Type
---------- ------- ------ -----
pip        18.1    19.0.1 wheel
setuptools 40.7.2  40.8.0 wheel
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
stonebig commented 5 years ago

so taking it in reverse, the problem seem to go away when I did this:

pip uninstall -y   widgetsnbextension
Uninstalling widgetsnbextension-3.4.2:
  Successfully uninstalled widgetsnbextension-3.4.2

pip list -o
Operation cancelled by user
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

pip uninstall -y     wordcloud
Uninstalling wordcloud-1.5.0:
  Successfully uninstalled wordcloud-1.5.0

pip list -o
Operation cancelled by user
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

pip uninstall -y      wrapt
Uninstalling wrapt-1.11.1:
  Successfully uninstalled wrapt-1.11.1

pip list -o
Operation cancelled by user
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

pip uninstall -y       xarray xlrd
Uninstalling xarray-0.11.3:
  Successfully uninstalled xarray-0.11.3
Uninstalling xlrd-1.2.0:
  Successfully uninstalled xlrd-1.2.0

pip list -o
Operation cancelled by user
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

pip uninstall -y         XlsxWriter xlwings zarr zict
Uninstalling XlsxWriter-1.1.2:
  Successfully uninstalled XlsxWriter-1.1.2
Uninstalling xlwings-0.15.1:
  Successfully uninstalled xlwings-0.15.1
Uninstalling zarr-2.2.0:
  Successfully uninstalled zarr-2.2.0
Uninstalling zict-0.1.3:
  Successfully uninstalled zict-0.1.3

pip list -o
Package                Version     Latest      Type
---------------------- ----------- ----------- -----
atomicwrites           1.2.1       1.3.0       wheel
bloscpack              0.13.0      0.16.0      sdist
cvxopt                 1.2.2       1.2.3       sdist
dask                   1.1.0       1.1.1       wheel
distributed            1.25.2      1.25.3      wheel
future                 0.17.0      0.17.1      sdist
GDAL                   2.3.3       2.4.0       sdist
intake                 0.2.3       0.4.1       sdist
mizani                 0.5.2       0.5.3       wheel
mlxtend                0.14.0      0.15.0.0    wheel
msgpack-python         0.5.4+dummy 0.5.6       sdist
mysql-connector-python 8.0.11      8.0.15      wheel
numcodecs              0.6.2       0.6.3       sdist
numpy                  1.15.4+mkl  1.16.1      wheel
pg8000                 1.12.4      1.13.0      wheel
pip                    18.1        19.0.1      wheel
pycodestyle            2.4.0       2.5.0       wheel
pypandoc               1.3.2       1.4         sdist
PyQt5                  5.9.2       5.11.3      wheel
pyshp                  2.0.0       2.0.1       sdist
pytest                 4.1.1       4.2.0       wheel
python-dateutil        2.7.5       2.8.0       wheel
regex                  2019.2.3    2019.2.5    wheel
requests-toolbelt      0.8.0       0.9.1       wheel
ruamel.yaml            0.15.85     0.15.87     wheel
scikit-fuzzy           0.3.1       0.4.0       sdist
setuptools             40.7.2      40.8.0      wheel
Shapely                1.6.4.post1 1.6.4.post2 sdist
Sphinx                 1.8.2       1.8.4       wheel
spyder-kernels         0.3.0       1.3.1       wheel
twitter                1.17.1      1.18.0      wheel
typed-ast              1.2.0       1.3.0       wheel
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.
stonebig commented 5 years ago

There is something wrong with one of these : XlsxWriter xlwings zarr zict.

stonebig commented 5 years ago

zict seems the problematic package .... what a strange thing. could there be something wrong on pypi server side ?

stonebig commented 5 years ago

zict RECORD is:

zict/__init__.py,sha256=pt9OP4ezD1XdH2gF1hZ4HoxN0CyteYebxAm07xyZiS4,186
zict/buffer.py,sha256=H-DRzVZB4hrHfWvgksazD1YtEllvGTzRVLGj4UVGtzo,3749
zict/common.py,sha256=peUA2daRk6OSnzPQhNlK3w_YyRPdGWKFdiT34xgJBcU,1426
zict/file.py,sha256=n3IfnKrXygUD8NI7nSyFUW1cBVwIsnu6QIVJCvseNcU,2629
zict/func.py,sha256=_q-mwCbllA93aNLJFiOXWaelZPldeCVxmQcS4_iz7pI,1951
zict/lmdb.py,sha256=wtdndt1Bz-iWDBmzlJhBU9a8xVL3KjzoLxlw9BsRghs,2922
zict/lru.py,sha256=jGbll_kWIho_BykwSoWQETh8B2lZF35IaKOgenDtodI,3047
zict/sieve.py,sha256=nqyeqSv2czxHAee3xjiLzodSD8PybK934C0I_o5Jdts,3201
zict/zip.py,sha256=yetz5EwpOGGfvhHCNrUdpkhpzh0gLP2kX2Z_Id8x_6g,2067
zict-0.1.3.dist-info/DESCRIPTION.rst,sha256=XzTuU2iST68wy2KOget-H6Fsj1qmAy1cu7sBl-ryqok,256
zict-0.1.3.dist-info/METADATA,sha256=TlyAOn_qhwsyVzCE8gkyj2dKU-iH6wcFFBFyckThcmU,567
zict-0.1.3.dist-info/RECORD,,
zict-0.1.3.dist-info/WHEEL,sha256=BTHHj6QhsHpWhRUrxioIvCAgjx81u-a8gkPTO6bi_A0,116
zict-0.1.3.dist-info/metadata.json,sha256=uXcmonEVzQb9NadRrhuz0SFfpWmbATZUnu_cmUH9Qs0,554
zict-0.1.3.dist-info/top_level.txt,sha256=ru4yqvaDGmbFDXeEif4vJBPJve86WggecSuZ07bdYIg,5
uranusjr commented 5 years ago

I can’t seem to reproduce this with a minimal setup:

λ pip install zict==0.1.3
Collecting zict==0.1.3
  Using cached https://files.pythonhosted.org/packages/5d/c9/eddd6c9a7ebd65fc799f9b87e56b45599a4e35d66e3da2722d7fc2a89f1f/zict-0.1.3-py2.py3-none-any.whl
Collecting heapdict (from zict==0.1.3)
Installing collected packages: heapdict, zict
Successfully installed heapdict-1.0.0 zict-0.1.3
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

λ pip list -o
Package    Version Latest Type
---------- ------- ------ -----
pip        18.1    19.0.1 wheel
setuptools 39.0.1  40.8.0 wheel
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

λ pip list
Package    Version
---------- -------
HeapDict   1.0.0
pip        18.1
setuptools 39.0.1
zict       0.1.3
You are using pip version 18.1, however version 19.0.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

There’s probably more in this :(

stonebig commented 5 years ago

retrying today with latest pip-19.0.2 and setuptools-40.8.0 ... same wired behavior.

C:\WinP\bd37\bu\WPy32-3720b5\scripts>pip list -o
Operation cancelled by user

C:\WinP\bd37\bu\WPy32-3720b5\scripts>pip uninstall zict -y
Uninstalling zict-0.1.3:
  Successfully uninstalled zict-0.1.3

C:\WinP\bd37\bu\WPy32-3720b5\scripts>pip list -o
Package                Version     Latest      Type
---------------------- ----------- ----------- -----
bloscpack              0.13.0      0.16.0      sdist
future                 0.17.0      0.17.1      sdist
GDAL                   2.3.3       2.4.0       sdist
intake                 0.2.3       0.4.1       sdist
mizani                 0.5.2       0.5.3       wheel
mlxtend                0.14.0      0.15.0.0    wheel
msgpack-python         0.5.4+dummy 0.5.6       sdist
mypy                   0.660       0.670       wheel
mysql-connector-python 8.0.11      8.0.15      wheel
numcodecs              0.6.2       0.6.3       sdist
numpy                  1.15.4+mkl  1.16.1      wheel
parso                  0.3.2       0.3.3       wheel
pg8000                 1.12.4      1.13.1      wheel
pypandoc               1.3.2       1.4         sdist
PyQt5                  5.9.2       5.12        wheel
pyshp                  2.0.0       2.0.1       sdist
ruamel.yaml            0.15.85     0.15.87     wheel
scikit-fuzzy           0.3.1       0.4.0       sdist
Shapely                1.6.4.post1 1.6.4.post2 sdist
spyder-kernels         0.4.2       1.3.2       wheel
tqdm                   4.30.0      4.31.1      wheel
twitter                1.17.1      1.18.0      wheel
typed-ast              1.2.0       1.3.1       wheel
XlsxWriter             1.1.2       1.1.4       wheel
xlwings                0.15.1      0.15.2      sdist

C:\WinP\bd37\bu\WPy32-3720b5\scripts>
stonebig commented 5 years ago

still the problem, even when using zict from github master

stonebig commented 5 years ago

installing zstd, so that zict is not the last package, doesn't resolve the problem either.

stonebig commented 5 years ago

ok, found th "-v" option. does this trace help ?


C:\WinP\bd37\buZero\WPy32-3720b5\scripts>pip list*pip list -o -v
ERROR: unknown command "list*pip" - maybe you meant "list"

C:\WinP\bd37\buZero\WPy32-3720b5\scripts>pip list
Package        Version
-------------- -------------
HeapDict       1.0.0
idlex          1.18
pip            19.0.2
prompt-toolkit 2.0.8
ptpython       2.0.4
Pygments       2.3.1
setuptools     40.8.0
six            1.12.0
wheel          0.33.0
winpython      1.11.20181222
zict           0.1.3

C:\WinP\bd37\buZero\WPy32-3720b5\scripts>pip list -o -v
Config variable 'Py_DEBUG' is unset, Python ABI tag may be incorrect
Config variable 'WITH_PYMALLOC' is unset, Python ABI tag may be incorrect
1 location(s) to search for versions of zict:
* https://pypi.org/simple/zict/
Getting page https://pypi.org/simple/zict/
Looking up "https://pypi.org/simple/zict/" in the cache
Request header has "max_age" as 0, cache bypassed
Starting new HTTPS connection (1): pypi.org:443
https://pypi.org:443 "GET /simple/zict/ HTTP/1.1" 304 0
Operation cancelled by user
Exception information:
Traceback (most recent call last):
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 40, in acquire
    os.mkdir(self.lock_file)
FileExistsError: [WinError 183] Impossible de créer un fichier déjà existant: 'c:\\users\\helene\\appdata\\local\\pip\\cache\\http\\f\\b\\3\\4\\2\\fb3424f809dd6dfe49e1a2574d3bc50f4fbdbce23f012bdf65e8a6d3.lock'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\cli\base_command.py", line 179, in main
    status = self.run(options, args)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\commands\list.py", line 144, in run
    packages = self.get_outdated(packages, options)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\commands\list.py", line 152, in get_outdated
    dist for dist in self.iter_packages_latest_infos(packages, options)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\commands\list.py", line 152, in <listcomp>
    dist for dist in self.iter_packages_latest_infos(packages, options)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\commands\list.py", line 179, in iter_packages_latest_infos
    all_candidates = finder.find_all_candidates(dist.key)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\index.py", line 610, in find_all_candidates
    for page in self._get_pages(url_locations, project_name):
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\index.py", line 743, in _get_pages
    page = _get_html_page(location, session=self.session)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\index.py", line 229, in _get_html_page
    resp = _get_html_response(url, session=session)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\index.py", line 177, in _get_html_response
    "Cache-Control": "max-age=0",
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\requests\sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\download.py", line 403, in request
    return super(PipSession, self).request(method, url, *args, **kwargs)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\requests\sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\requests\sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 53, in send
    resp = super(CacheControlAdapter, self).send(request, **kw)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\requests\adapters.py", line 533, in send
    return self.build_response(request, resp)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\cachecontrol\adapter.py", line 80, in build_response
    request, response
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\cachecontrol\controller.py", line 365, in update_cached_response
    self.cache.set(cache_url, self.serializer.dumps(request, cached_response))
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_internal\download.py", line 304, in set
    return super(SafeFileCache, self).set(*args, **kwargs)
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\cachecontrol\caches\file_cache.py", line 126, in set
    with self.lock_class(name) as lock:
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\lockfile\__init__.py", line 197, in __enter__
    self.acquire()
  File "C:\WinP\bd37\buZero\WPy32-3720b5\python-3.7.2\lib\site-packages\pip\_vendor\lockfile\mkdirlockfile.py", line 57, in acquire
    time.sleep(wait)
KeyboardInterrupt

C:\WinP\bd37\buZero\WPy32-3720b5\scripts>
stonebig commented 5 years ago

okay: prefixing the file and the directory in "C:\Users\helene\AppData\Local\pip\cache\http\f\b\3\4\2" with "zz_" solves the problem.

==> maybe one day someone will explain me this strange behavior of lock files. maybe pip would be friendly to suggest something is odd with a 12 days old lock file

cjerdonek commented 5 years ago

@stonebig Thanks for digging into this and diagnosing. This looks like a duplicate of #3532, then. Do you agree?

stonebig commented 5 years ago

Yes. Hope it will be solved one day

lock[bot] commented 5 years ago

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.