Race condition in several helpers

[amirouche]

While trying to build a dataset of pypi packages (I know it's not safe etc...), I have hit the exception CorruptCacheError. This happens because I pipenv install things in parallel.

The worst in this story, is that I use the following environment variables:

export PIP_NO_CACHE_DIR=false
export PIPENV_VENV_IN_PROJECT=1

The PIP_NO_CACHE_DIR is not respected AFAIU since the above exception is raised. It's prolly not the exact semantics of that env variable but it not expected at all.

Also, the other env var PIPENV_VENV_IN_PROJECT imply to me that everything happens in the .venv which is clearly not the case.

To workaround it, I lookup where that cache is used in pipenv:

~/.local/lib/python3.6/site-packages/pipenv$ ag read_cache_file
vendor/requirementslib/models/cache.py
42:def read_cache_file(cache_file_path):
111:            self._cache = read_cache_file(self._cache_file)
291:            self._cache = read_cache_file(self._cache_file)

vendor/passa/models/caches.py
90:def _read_cache_file(cache_file_path):
160:            self._cache = _read_cache_file(self._cache_file)

patched/piptools/cache.py
29:def read_cache_file(cache_file_path):
94:            self._cache = read_cache_file(self._cache_file)

And I replaced the read_cache-fu with an empty dict everything seems to work (I also silenced the write_cache method from the same files).

I used this program in parallel to raise the issue.

pipenv --version
pipenv, version 2018.10.13

[techalchemy]

Hmmmm interesting. Pipenv already installs things in parallel so can you clarify whether you are performing additional parallellization? We have a lot of reasons for not respecting pip’s cache options, and as a a side note you may be better served using an actual PyPI mirroring tool like devpi or bandersnatch.

[techalchemy]

Ooh I just read your script. Interesting!

[techalchemy]

Hm I should point out we don't actually use pip's cache, so simply telling pip not to cache things isn't actually going to do anything here. In any case, caches don't live in your virtualenv... and it doesn't seem like this is something that depends on anything we can have any substantive impact on. Thanks for the report and good luck!