No module named 'uuid_utils._uuid_utils'"

DeboBurro commented 10 months ago

environments

uuid-utils version: 0.6.1
AWS Lambda python runtime: 3.11
OS: Ubuntu: 20.04.4
way to reproduce:
1. I am using python3.8 environment locally
2. I pip install -r $dir/requirements.txt --target $lambda_function_dir/third_party ( where I put the uuid_utils==0.6.1 in the requirements.txt )
3. I zipped all the dependencies needed to run in AWS Lambda, then I uploaded this zipped package to Lambda.

Screenshot from 2023-11-08 01-08-34

Issue Statement:

but when I create an test event and invoke the lambda function, it throws an error No module named 'uuid_utils._uuid_utils'

Test Event Name
update-sites

Response
{
  "errorMessage": "Unable to import module 'lambda_function': No module named 'uuid_utils._uuid_utils'",
  "errorType": "Runtime.ImportModuleError",
  "requestId": "7d706df9-dd9f-49d4-80ae-ea0dee96a5cc",
  "stackTrace": []
}

Function Logs
START RequestId: 7d706df9-dd9f-49d4-80ae-ea0dee96a5cc Version: $LATEST
[ERROR] Runtime.ImportModuleError: Unable to import module 'lambda_function': No module named 'uuid_utils._uuid_utils'
Traceback (most recent call last):END RequestId: 7d706df9-dd9f-49d4-80ae-ea0dee96a5cc
REPORT RequestId: 7d706df9-dd9f-49d4-80ae-ea0dee96a5cc  Duration: 8.30 ms   Billed Duration: 9 ms   Memory Size: 128 MB Max Memory Used: 59 MB  Init Duration: 261.54 ms

Request ID
7d706df9-dd9f-49d4-80ae-ea0dee96a5cc

additional context

locally run this lambda function is fine. I think there might be something wrong when creating this pip package.

aminalaee commented 10 months ago

This is interesting, and what OS are you running locally? Because I think the wheels are downloaded for your OS but the Lambda has a different wheel. One way is to ship Docker images to Lambda which is not your case.

DeboBurro commented 10 months ago

Using ubuntu 20.04.4.

burro@debo-burro:~$ uname -a
Linux debo-burro 5.11.0-46-generic #51~20.04.1-Ubuntu SMP Fri Jan 7 06:51:40 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

burro@debo-burro:~$ lsb_release -a
LSB Version:    core-11.1.0ubuntu2-noarch:security-11.1.0ubuntu2-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 20.04.4 LTS
Release:    20.04
Codename:   focal

Indeed, shipping a Docker images might resolve it. thanks for this suggestion This is the first time I see dependencies issue using this package on our AWS lambda CICD. ( since AWS allows us to zip all python dependencies in the same zip archive and upload, I believe this is probably one of the standard ways to do so. ) I don't have any issue with other python dependencies. So just wondering maybe we can help improve this package.

aminalaee commented 10 months ago

I haven’t really had time to check but my guess it has something to do with OS or Architecture of the wheel you ship to AWS. You can check your zip to see which wheel of uuid_utils is packaged, and probably that doesn’t work on AWS Linux 2 https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html so you might have to specify —platform in pip https://stackoverflow.com/a/74652148

aerickson1337 commented 10 months ago

This is an AWS Lambda python quirk, when you build and bundle packages that have binaries, that is to say they aren't pure python only packages, if the wheel you installed doesn't match the upstream there will usually be issues. In the Op's case the version mismatch is probably enough to cause them issues. Their 3.8 local environment installs the 3.8 wheel which installs the 3.8 binary which is subsequently (likely) broken in the 3.11 runtime they're trying to use. The version difference is often enough to cause issues but the platform the wheel's installed on can also cause issues. E.g. on a M1/M2 Mac those wheel binaries generally are not going to work deployed to a x86 lambda.

There are some other ephemeral issues that also cause problems, like if the environment you're building on is exposing other libraries via a linking process to your python install you may not be aware is setup. Wheels generally solve for this issue if the versions match though, so this quirk generally won't be a problem.

The safest fix for this if you can't deploy a container or need to be able to live edit your lambda function via the AWS console is to build your dependencies using the container image for the python version (and platform) you plan to deploy to. Those images are here: https://hub.docker.com/r/amazon/aws-lambda-python

As far as I'm aware the simplest approach if you need to deploy the zips as layers or deployment packages with the least amount of fiddling with your local python environments is to create a Dockerfile using the AWS lambda python image version you intend to target. Install your dependencies in there and apply your zipping/packaging process, then dump the output onto your host machine. This can be done pretty cleanly with BuildKit and custom build outputs. There's some minor fiddling likely required but that should get you most of the way there.

aminalaee commented 6 months ago

I guess we can close this for now.

aminalaee / uuid-utils