nasa / cumulus-process-py

Python utilities library for Cumulus
Apache License 2.0
7 stars 10 forks source link

Can the boto dependency be made more explicit #136

Open fgreg opened 3 years ago

fgreg commented 3 years ago

With the new pip dependency resolver introducing the idea of 'backtracking' in order to resolve dependencies in an environment, we are noticing very long installation times for this python package because of the boto3~=1.4 requirement.

To reproduce what I'm talking about, create a new virtual environment (I'm using anaconda below), make sure pip is up to date, install the latest boto3 package, then try installing this package.

conda create -n temp-py 'python>=3.8'
conda activate temp-py
python -m pip install --upgrade pip
pip install boto3
pip install cumulus-process

The behavior I see is pip will tell me:

INFO: pip is looking at multiple versions of botocore to determine which version is compatible with other requirements. This could take a while.

And it ends up 'backtracking' all the way to version 1.4 of boto3 before resolving. I've seen a few examples where it never resolves and just gives an error.

I think this could be resolved by being more specific about which version of boto3 is required; I think ideally the project should be converted to use something like a pipfile and pipfile.lock to specify exact versions of libraries it is intended to be built against.

Jkovarik commented 3 years ago

Thanks for putting in the issue!

This is also demonstrated in CI builds (e.g. https://app.circleci.com/pipelines/github/nasa/cumulus-process-py/14/workflows/6aa01e9c-0de9-4c7b-87f9-eb1753af4431/jobs/949) .

I've added this to our internal tracker as CUMULUS-2151