Use cache when running pip-compile with non-standard pip-index

[fralik]

What's the problem this feature will solve?

I saw that if I run pip-compile -i <url> --resolver=backtracking --verbose requirements.in, then dependencies are always downloaded from the provided index url. In contrast, if I run the same command without -i flag, then cached version of packages may be used, which results in much fast compiling time.

Pip itself is able to cache packages from such indexes. If I run pip install -i <url> package==1.2.3, then the distributed binary will end up in cache. Next time I invoke the same command, there will be no download.

Describe the solution you'd like

I would like pip-compile to cache dependency information from third party indices similar to how it does it for PyPI.

Probably the main benefit will be for people who develop non-public packages, that are uploaded to private indices.

Alternative Solutions

I tried to specify cache directory directly in pip-compile, but this didn't have any effect.

Additional context

N/A

[fralik]

I looked a bit more into this. One observation: if I run the same requirements.in file with and without an index, then pip-tools itself never picks up any cache and tries to download files from the index. Then, there is a difference in pip behaviour:

• pip without an index => may try to download a file, but then uses a cache.
• pip with an index => always downloads the file, even after I perform pip install -i <url> package.

[hmc-cs-mdrissi]

I'm also running into this and it leads to very slow pip compile times. May look into it more later this week.

[hmc-cs-mdrissi]

After more debugging, I've found this happens with some index urls and wheel caching works fine with others. pytorch index url and jfrog internal index url both support wheel caching properly. A different internal index url, doesn't cache wheels at all. I think this is more likely not a bug in pip/pip-tools but instead a bug with the index server doing something non-standard. Yo (and me) will probably need to talk with whoever behind internal index server and have them debug.

edit: In case it helps the issue for internal index I use looks to be missing ETag. The main logic that controls pip http caching is here and breakpoint debugging that is how I eventually found issue. Ensuring the max age/expires header being set also would do trick. Some index servers have strange default of 0 max age.