[FEATURE] Add support for urllib3 > v2

Bialogs commented 9 months ago

What is the bug?

When using opensearch-py v2.3.2 and greater with boto3 provided by AWS Lambda , the following error occurs:

cannot import name 'DEFAULT_CIPHERS' from 'urllib3.util.ssl_'

How can one reproduce the bug?

Dockerfile:

FROM public.ecr.aws/lambda/python:3.9
RUN pip3 install opensearch-py==2.3.1 --target ${LAMBDA_TASK_ROOT}
COPY /app/ ${LAMBDA_TASK_ROOT}/
CMD ["app.function"]

app.py:

import boto3
import opensearchpy

def function(event, ctx)
    print("hello")

What is the expected behavior?

Imports are successful and code runs.

What is your host/environment?

public.ecr.aws/lambda/python:3.9

Do you have any screenshots?

N/A

Do you have any additional context?

Downgrade to 2.3.1 successfully executes.

dblock commented 9 months ago

Did you mean 2.4.2? If not, and you meant 2.3.2, would you be so kind to confirm that this still doesn't work in 2.4.2?

I found a number of hits on Google for this, e.g. https://github.com/psf/requests/issues/6443, so it sounds like this may be about versions of urllib3 or requests. Have you tried upgrading those?

Bialogs commented 9 months ago

Yes from my test it failed with v2.3.2 through v2.4.2.

In terms of the runtime I'm not installing any other packages other than opensearch-py at the specified versions. I'm noticing that the commits for v2.3.2 include changes to the urllib3 version. Perhaps that introduced this?

Bialogs commented 9 months ago

Examining the base image, public.ecr.aws/lambda/python:3.9 the following urllib3 is bundled. I assume because it is integrated with the other software packages that the base image contains.

From Docker Scout (docker scout sbom public.ecr.aws/lambda/python:3.9)

    {
      "type": "pypi",
      "name": "urllib3",
      "version": "1.26.18",
      "purl": "pkg:pypi/urllib3@1.26.18",
      "author": "Andrey Petrov",
      "licenses": [
        "MIT"
      ],
      "locations": [
        {
          "path": "/var/runtime/urllib3-1.26.18.dist-info/METADATA",
          "ordinal": 7,
          "digest": "sha256:4dd1a48d6392cbd0f03fdefc227609f129f9648bfc83a06f8435f8536eb1efbe",
          "diff_id": "sha256:8a810533d2d18a464c851d93a9aabf4ddbcdf6c171bef202acaeec45d29e9779"
        },
        {
          "path": "/var/runtime/urllib3-1.26.18.dist-info/RECORD",
          "ordinal": 7,
          "digest": "sha256:4dd1a48d6392cbd0f03fdefc227609f129f9648bfc83a06f8435f8536eb1efbe",
          "diff_id": "sha256:8a810533d2d18a464c851d93a9aabf4ddbcdf6c171bef202acaeec45d29e9779"
        },
        {
          "path": "/var/runtime/urllib3-1.26.18.dist-info/top_level.txt",
          "ordinal": 7,
          "digest": "sha256:4dd1a48d6392cbd0f03fdefc227609f129f9648bfc83a06f8435f8536eb1efbe",
          "diff_id": "sha256:8a810533d2d18a464c851d93a9aabf4ddbcdf6c171bef202acaeec45d29e9779"
        }
      ]
    }

And this PR pushed this package urllib3 version above this to "urllib3>=1.26.9" https://github.com/opensearch-project/opensearch-py/pull/518

dblock commented 9 months ago

Can you compare the versions of urllib and requests you get with 1.3.1 and 1.3.2? I want to know which one causes the break, and whether it's something in our code. Also does https://stackoverflow.com/questions/76414514/cannot-import-name-default-ciphers-from-urllib3-util-ssl-on-aws-lambda-us help?

Bialogs commented 9 months ago

Can you compare the versions of urllib and requests you get with 1.3.1 and 1.3.2?

Do you mean 2.3.1 and 2.3.2 of opensearch-py?

dblock commented 9 months ago

Can you compare the versions of urllib and requests you get with 1.3.1 and 1.3.2?

Do you mean 2.3.1 and 2.3.2 of opensearch-py?

Yes, trying to narrow down where the actual problem is.

Bialogs commented 9 months ago

opensearch-py v2.3.1:

urllib3 1.26.18
requests 2.31.0

opensearch-py v2.3.2:

urllib3 1.26.18
urllib3 2.1.0
requests 2.31.0

Also does https://stackoverflow.com/questions/76414514/cannot-import-name-default-ciphers-from-urllib3-util-ssl-on-aws-lambda-us help?

When adding the additional constraint RUN pip3 install opensearch-py==2.3.2 "urllib3<2" --target ${LAMBDA_TASK_ROOT}:

urllib3 1.26.18
requests 2.31.0

Seems like that should be the way forward for me.

dblock commented 9 months ago

So this is happening because urllib3 2.x is being installed and used, but it's not compatible. I tracked it down to https://github.com/opensearch-project/opensearch-py/pull/466 (cc @saimedhi) which was fixing https://github.com/opensearch-project/opensearch-py/issues/492. I think we need to make sure the library works with urllib3 >= 2, or re-add the <2 constraint.

Want to help @Bialogs?

Bialogs commented 9 months ago

To be clear, I don't mean to make any claims that this library is incompatible with urllib3 2.x.

What I have found is that the Lambda base image contains at least boto3 (which comes pre-installed) which does not seem compatible with urllib3 > 1.26.18.

If opensearch-py wants to support the Lambda base image containers (i.e. this workflow using these images) I think the way forward would be re-add <2 constraint on urllib3.

Alternatively, users can reference this issue and add the <2 constraint in their own code.

I defer to the maintainers as you know your installation base better and constraining urllib3 version may be something you don't want to do.

dblock commented 9 months ago

To be clear, I don't mean to make any claims that this library is incompatible with urllib3 2.x.

But I am. I believe this is why we have this error. Because 2.x is installed, it gets picked (it satisfies >= 1.26....). However I think it also does work with 2.x for some scenarios. Either way we don't use that version in tests, so I would want to know for sure by having a test run with 2.x and seeing what breaks to begin with.

I might get to it soon, but if you have time, don't hold back on helping ;)

dblock commented 9 months ago

Reopening to add support for urllib3 >= 2.

Ousret commented 8 months ago

Hello there,

Sorry to barge in. A bit off topic, but I wanted to recommend Niquests as a possible replacement for Requests+aiohttp. I am willing to help if this is something of potential interest.

dblock commented 8 months ago

@Ousret yes would love some help. Generally we want all these libraries to be switchable.

SamStephens commented 6 months ago

So the issue here is that the botocore packaged with the AWS Lambda Python 3.9 image (at least at the time this issue was created) requires urlllib3 < 2. The relevant versions of botocore actually specify this dependency in their requirements.txt.

However because this is the version of botocore packaged with the image, when users install their own dependencies using pip, the urlllib3 < 2 constraint botocore has is not taken into account.

The thing is, AWS Lambda best practices are to not use prepackaged libraries, to explicitly install all your dependencies:

Control the dependencies in your function's deployment package. The AWS Lambda execution environment contains a number of libraries such as the AWS SDK for the Node.js and Python runtimes (a full list can be found here: Lambda runtimes). To enable the latest set of features and security updates, Lambda will periodically update these libraries. These updates may introduce subtle changes to the behavior of your Lambda function. To have full control of the dependencies your function uses, package all of your dependencies with your deployment package.

From https://docs.aws.amazon.com/lambda/latest/dg/best-practices.html#function-code. Although the best practice here doesn't make it explicit, another reason to package all of your dependencies with your deployment package is that if you don't, the dependency graph your package manager is using is incomplete (missing packages included with your runtime), and you see issues like this.

As such, I would argue opensearch-py should not be pinning on urllib3 < 2.

Apart from anything else, if you're going to pin to urllib3 < 2 as long as the AWS Lambda Python 3.9 image does not support urllib3 >= 2, you will be keeping that pin for the lifetime of the AWS Lambda Python 3.9 runtime as the OpenSSL provided in that runtime is < 1.1.1 and will never be upgraded to 1.1.1 to the best of my knowledge.

iherasymenko commented 5 months ago

Perhaps a similar constraint can be applied to opensearch-py 🤔

   urllib3>=1.25.4,<1.27; python_version<"3.10"
   urllib3>=1.25.4,!=2.2.0,<3; python_version>="3.10"

https://github.com/boto/botocore/pull/3141

james-certn commented 5 months ago

Subjectively speaking, this feels like something relatively urgent. The longer the delay, the bigger the versioning dependency problem will get. I'm currently seeing:

The conflict is caused by:
    auth0-python 4.7.1 depends on urllib3<3.0.0 and >=2.0.7
    opensearch-py 2.5.0 depends on urllib3<2 and >=1.26.18

dblock commented 5 months ago

@james-certn There's an open PR in https://github.com/opensearch-project/opensearch-py/pull/719 that needs some tests. Want to help?

gabfelp commented 4 months ago

Subjectively speaking, this feels like something relatively urgent. The longer the delay, the bigger the versioning dependency problem will get. I'm currently seeing:
The conflict is caused by:
    auth0-python 4.7.1 depends on urllib3<3.0.0 and >=2.0.7
    opensearch-py 2.5.0 depends on urllib3<2 and >=1.26.18

I faced this same problem today.

Great that the PR supporting urllib3 > v2 has already been merged!

As I'm new to the repository, I'm unsure of how the release process works. Is there any estimate of when the next release (with this code) will come out?

dblock commented 4 months ago

As I'm new to the repository, I'm unsure of how the release process works. Is there any estimate of when the next release (with this code) will come out?

Open an issue to "release v. next" and we can do it quickly.

SamStephens commented 3 months ago

Should this issue be resolved, now version 2.6.0 has been released?

opensearch-project / opensearch-py