Does PIP need to verify simple index HTML ?

jku / pip

pip fork to experiment with PEP-458 implementation https://www.python.org/dev/peps/pep-0458/: See branch tuf-v2 (and tuf-mvp and tuf-mvp-vendored for earlier work)

https://pip.pypa.io/

MIT License

1 stars 0 forks source link

Does PIP need to verify simple index HTML ? #7

Closed jku closed 4 years ago

jku commented 4 years ago

PEP-458:

When updating bin-n metadata for a consistent snapshot, the snapshot process SHOULD also include any new or updated hashes of simple index pages in the relevant bin-n metadata.

That's a "SHOULD" but dstufft is correct in saying that not verifying simple indexes means multiple attack types are not protected from... The current WIP Warehouse implementation does not do this and my pip implementation plan does not take this into account.

jku commented 4 years ago

Some notes:

without verified indexes we'll lose "Rollback", "Indefinite freeze" and "Mix-and-match" protection.
Current status is that Williams Warehouse work does not include index page verification at the moment, and I missed that in the PEP as well so my client plans do not cover that either
The "SHOULD" in the PEP seems dangerous: how does a client know if index pages are verified or not? Think of an attacker adding a new index page...
index pages are not consistent (in the way "consistent snapshot" means): is this a problem?

jku commented 4 years ago

The pep-458 implementation meeting result is that:

Warehouse should include index files in bin-n metadata
This may mean the index files actually get stored in DB (instead of generated on request)
pip should verify index files using tuf: this probably doubles the work but that does seem unavoidable

jku commented 4 years ago

This is now implemented (in this repo: Warehouse branch does not seem to have it yet):

pip looks for target <PROJECT>/index.html and downloads https://pypi.org/simple/<PROJECT>/<HASH>.index.html when it needs the project index file