search

google / cloud-forensics-utils

Python library to carry out DFIR analysis on the Cloud
Apache License 2.0
452 stars 89 forks source link

boto3/botocore dependencies pull multiple versions during installation #473

Closed jleaniz closed 1 year ago

jleaniz commented 1 year ago

Attempting to install the library in a Python 3.10 environment via pip results in pip trying to install dozens of versions of boto3 and botocore, probably due to libcloudforensics requirements.txt not specifying any version lower/upper bounds.

Collecting s3transfer<0.7.0,>=0.6.0 (from boto3->libcloudforensics->-r requirements.txt (line 19))
  Downloading s3transfer-0.6.0-py3-none-any.whl (79 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 79.6/79.6 kB 13.7 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of botocore to determine which version is compatible with other requirements. This could take a while.
Collecting boto3 (from libcloudforensics->-r requirements.txt (line 19))
  Downloading boto3-1.26.125-py3-none-any.whl (135 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 135.6/135.6 kB 20.0 MB/s eta 0:00:00
Collecting botocore (from libcloudforensics->-r requirements.txt (line 19))
  Downloading botocore-1.29.125-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 51.2 MB/s eta 0:00:00
  Downloading botocore-1.29.124-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 62.0 MB/s eta 0:00:00
  Downloading botocore-1.29.123-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 62.9 MB/s eta 0:00:00
  Downloading botocore-1.29.122-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 63.8 MB/s eta 0:00:00
  Downloading botocore-1.29.121-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 62.4 MB/s eta 0:00:00
  Downloading botocore-1.29.120-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 60.4 MB/s eta 0:00:00
INFO: pip is looking at multiple versions of botocore to determine which version is compatible with other requirements. This could take a while.
  Downloading botocore-1.29.119-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 61.7 MB/s eta 0:00:00
  Downloading botocore-1.29.118-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 62.4 MB/s eta 0:00:00
  Downloading botocore-1.29.117-py3-none-any.whl (10.7 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.7/10.7 MB 62.3 MB/s eta 0:00:00
  Downloading botocore-1.29.116-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 62.8 MB/s eta 0:00:00
  Downloading botocore-1.29.115-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 61.7 MB/s eta 0:00:00
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. See https://pip.pypa.io/warnings/backtracking for guidance. If you want to abort this run, press Ctrl + C.
  Downloading botocore-1.29.114-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 59.3 MB/s eta 0:00:00
  Downloading botocore-1.29.113-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 64.4 MB/s eta 0:00:00
  Downloading botocore-1.29.112-py3-none-any.whl (10.6 MB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.6/10.6 MB 63.9 MB/s eta 0:00:00
  Downloading botocore-1.29.111-py3-none-any.whl (10.6 MB)
       ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 15.5 MB/s eta 0:00:00
  Downloading boto3-1.16.29-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 17.0 MB/s eta 0:00:00
  Downloading boto3-1.16.28-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 17.5 MB/s eta 0:00:00
  Downloading boto3-1.16.27-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 16.5 MB/s eta 0:00:00
  Downloading boto3-1.16.26-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 17.6 MB/s eta 0:00:00
  Downloading boto3-1.16.25-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 16.7 MB/s eta 0:00:00
  Downloading boto3-1.16.24-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 15.3 MB/s eta 0:00:00
  Downloading boto3-1.16.23-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 17.5 MB/s eta 0:00:00
  Downloading boto3-1.16.22-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 16.8 MB/s eta 0:00:00
  Downloading boto3-1.16.21-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 16.1 MB/s eta 0:00:00
  Downloading boto3-1.16.20-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 16.1 MB/s eta 0:00:00
  Downloading boto3-1.16.19-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 20.3 MB/s eta 0:00:00
  Downloading boto3-1.16.18-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 19.8 MB/s eta 0:00:00
  Downloading boto3-1.16.17-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 17.5 MB/s eta 0:00:00
  Downloading boto3-1.16.16-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 20.1 MB/s eta 0:00:00
  Downloading boto3-1.16.15-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 18.1 MB/s eta 0:00:00
  Downloading boto3-1.16.14-py2.py3-none-any.whl (129 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 129.7/129.7 kB 20.6 MB/s eta 0:00:00
  Downloading boto3-1.16.13-py2.py3-none-any.whl (129 kB). 
...
jleaniz commented 1 year ago

For reference, this is currently causing Turbinia docker builds to balloon to over 8GB in size from 1GB.

jleaniz commented 1 year ago

Should be fixed in #474