search

google / gae-secure-scaffold-python3

Secure scaffold for Google App Engine static and dynamic Python websites
Apache License 2.0
31 stars 16 forks source link

Make installation work when using `pip install --require-hashes` #9

Open davidwtbuxton opened 2 years ago

davidwtbuxton commented 2 years ago

Find out what the pain points are for projects which want to use reproducible builds with pip, and fix them.

https://pip.pypa.io/en/stable/topics/secure-installs/

davidwtbuxton commented 2 years ago

I have a requriements.txt with a valid hash:

$ cat requirements.txt 
https://github.com/google/gae-secure-scaffold-python3/archive/2d34759b73491148063501533be8fb80d05cf8bb.zip --hash=sha256:de5b87a0a8177b57bfb0ce03f1ebffcc3d56ab69591fb326c68e89feee467ce3

Then installing with pip fails because I haven't specified all the dependencies:

$ pip install --requirement requirements.txt 
Collecting https://github.com/google/gae-secure-scaffold-python3/archive/2d34759b73491148063501533be8fb80d05cf8bb.zip (from -r requirements.txt (line 1))
  Using cached https://github.com/google/gae-secure-scaffold-python3/archive/2d34759b73491148063501533be8fb80d05cf8bb.zip
  Preparing metadata (setup.py) ... done
Collecting Flask
ERROR: In --require-hashes mode, all requirements must have their versions pinned with ==. These do not:
    Flask from https://files.pythonhosted.org/packages/0f/43/15f4f9ab225b0b25352412e8daa3d0e3d135fcf5e127070c74c3632c8b4c/Flask-2.2.2-py3-none-any.whl (from Secure-Scaffold==1.0.0->-r requirements.txt (line 1))

So we can use pip-tools to generate a complete set of requirements from a spec:

$ cat requirements.in 
https://github.com/google/gae-secure-scaffold-python3/archive/2d34759b73491148063501533be8fb80d05cf8bb.zip

pip-compile fails to resolve a set of compatible versions:

$ pip-compile requirements.in 
Could not find a version that matches protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<4.0.0dev,<5.0.0dev,>=3.15.0,>=3.19.5,>=4.21.3 (from google-cloud-datastore==1.15.5->google-cloud-ndb==1.11.1->Secure-Scaffold==1.0.0->-r requirements.in (line 1))
Tried: 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0, 3.0.0, 3.1.0, 3.1.0.post1, 3.1.0.post1, 3.2.0, 3.2.0, 3.3.0, 3.4.0, 3.4.0, 3.5.0.post1, 3.5.0.post1, 3.5.1, 3.5.1, 3.5.2, 3.5.2, 3.5.2.post1, 3.5.2.post1, 3.6.0, 3.6.0, 3.6.1, 3.6.1, 3.7.0, 3.7.0, 3.7.1, 3.7.1, 3.8.0, 3.8.0, 3.9.0, 3.9.0, 3.9.1, 3.9.1, 3.9.2, 3.9.2, 3.10.0, 3.10.0, 3.11.0, 3.11.0, 3.11.1, 3.11.1, 3.11.2, 3.11.2, 3.11.3, 3.11.3, 3.12.2, 3.12.2, 3.12.4, 3.12.4, 3.13.0, 3.13.0, 3.14.0, 3.14.0, 3.15.0, 3.15.0, 3.15.1, 3.15.1, 3.15.2, 3.15.2, 3.15.3, 3.15.3, 3.15.4, 3.15.4, 3.15.5, 3.15.5, 3.15.6, 3.15.6, 3.15.7, 3.15.7, 3.15.8, 3.15.8, 3.16.0, 3.16.0, 3.17.0, 3.17.0, 3.17.1, 3.17.1, 3.17.2, 3.17.2, 3.17.3, 3.17.3, 3.18.0, 3.18.0, 3.18.1, 3.18.1, 3.18.3, 3.18.3, 3.19.0, 3.19.0, 3.19.1, 3.19.1, 3.19.2, 3.19.2, 3.19.3, 3.19.3, 3.19.4, 3.19.4, 3.19.5, 3.19.5, 3.19.6, 3.19.6, 3.20.0, 3.20.0, 3.20.1, 3.20.1, 3.20.2, 3.20.2, 3.20.3, 3.20.3, 4.21.0, 4.21.0, 4.21.0, 4.21.0, 4.21.1, 4.21.1, 4.21.1, 4.21.1, 4.21.2, 4.21.2, 4.21.2, 4.21.2, 4.21.3, 4.21.3, 4.21.3, 4.21.3, 4.21.4, 4.21.4, 4.21.4, 4.21.4, 4.21.5, 4.21.5, 4.21.5, 4.21.5, 4.21.6, 4.21.6, 4.21.6, 4.21.6, 4.21.7, 4.21.7, 4.21.7, 4.21.7
Skipped pre-versions: 2.0.0b0, 3.0.0a2, 3.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2, 3.0.0b2.post1, 3.0.0b2.post1, 3.0.0b2.post2, 3.0.0b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0b4, 3.2.0rc1, 3.2.0rc1, 3.2.0rc1.post1, 3.2.0rc1.post1, 3.2.0rc2, 3.2.0rc2, 3.7.0rc2, 3.7.0rc2, 3.7.0rc3, 3.7.0rc3, 3.8.0rc1, 3.8.0rc1, 3.9.0rc1, 3.9.0rc1, 3.10.0rc1, 3.10.0rc1, 3.11.0rc1, 3.11.0rc1, 3.11.0rc2, 3.11.0rc2, 3.13.0rc3, 3.13.0rc3, 3.14.0rc1, 3.14.0rc1, 3.14.0rc2, 3.14.0rc2, 3.14.0rc3, 3.14.0rc3, 3.15.0rc1, 3.15.0rc1, 3.15.0rc2, 3.15.0rc2, 3.16.0rc1, 3.16.0rc1, 3.16.0rc2, 3.16.0rc2, 3.17.0rc1, 3.17.0rc1, 3.17.0rc2, 3.17.0rc2, 3.18.0rc1, 3.18.0rc1, 3.18.0rc2, 3.18.0rc2, 3.19.0rc1, 3.19.0rc1, 3.19.0rc2, 3.19.0rc2, 3.20.0rc1, 3.20.0rc1, 3.20.0rc2, 3.20.0rc2, 3.20.1rc1, 3.20.1rc1, 4.0.0rc1, 4.0.0rc1, 4.0.0rc2, 4.0.0rc2, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc1, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2, 4.21.0rc2
There are incompatible versions in the resolved dependencies:
  protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5 (from google-api-core[grpc]==2.10.2->google-cloud-datastore==1.15.5->google-cloud-ndb==1.11.1->Secure-Scaffold==1.0.0->-r requirements.in (line 1))
  protobuf>=4.21.3 (from grpcio-status==1.49.1->google-api-core[grpc]==2.10.2->google-cloud-datastore==1.15.5->google-cloud-ndb==1.11.1->Secure-Scaffold==1.0.0->-r requirements.in (line 1))
  protobuf<4.0.0dev (from google-cloud-datastore==1.15.5->google-cloud-ndb==1.11.1->Secure-Scaffold==1.0.0->-r requirements.in (line 1))
  protobuf<5.0.0dev,>=3.15.0 (from googleapis-common-protos==1.56.4->google-api-core[grpc]==2.10.2->google-cloud-datastore==1.15.5->google-cloud-ndb==1.11.1->Secure-Scaffold==1.0.0->-r requirements.in (line 1))

A very quick look suggests that google-cloud-ndb may have a dependency on an older version of google-cloud-datastore, and this ends up being incompatible.

davidwtbuxton commented 2 years ago

There is some recent effort to update dependencies for the NDB library https://github.com/googleapis/python-ndb/pull/803 , but I'm not sure that specific change fixes the dependency error reported by pip-compile.

Another approach would be to remove the direct dependency on google-cloud-ndb. We use it for fetching/creating a random secret at application start, and it should be easy to reimplement that logic with google-cloud-datastore. I think that would make pip-compile happy, but maybe there would still be issues for any project that was trying to use this package along with google-cloud-ndb?