tchanxx commented 1 year ago

Problem

I searched through the issues and docs and I couldn't find anything mentioning support for running this on serverless functions.

I am trying to use the Serverless Framework to deploy this on AWS Lambda, but I cannot figure out how to package the generated client. I am using the serverless-python-requirements plugin and I am using the dockerizePip: true option to compile the Python requirements for Arm64 because my local is Macbook.

When I naively deploy this to AWS Lambda using sls deploy, I get ModuleNotFoundError: No module named 'prisma.models' which I think indicates that the client wasn't generated.

Is there a trick or any suggestions to get this to generate correctly to be used with the Serverless Framework? I was also looking for ways to run commands after the pip installation so that I could put python3 -m prisma generate there but I couldn't find any.

Suggested solution

Documentation on how to do this.

I am also cross-posting this in the serverless-python-requirements repo to ask there as well.

sarwaan001 commented 1 year ago

Hello,

I actually got this to work on lambda without using a lambda layer

Here is my python build function to build the package in pulumi but it can be inspiration for someone to create a guide


const buildPython = async (folderPath: string, prismaSchemaDirectory: string, requirementsPath: string): Promise<pulumi.asset.FileArchive> => {
        const tempFolder = await mkdtemp(path.join(tmpdir(), "temp-"));

        const commands: string[] = [
            `cd ${tempFolder}`, 
            `mkdir -p ${tempFolder}/out/`, 
            `mkdir -p ${tempFolder}/dist/`,
            `cd ${folderPath}`, 
            `pip install -U -r ${requirementsPath} --target ${tempFolder}/out/`,
            // `sed -i '27,55d' ${tempFolder}/out/generated_python/__init__.py`, // Removes some init code for debugging
            `rsync -avu -r ${folderPath}/ ${tempFolder}/out/ --ignore-existing --exclude=out/ --exclude=dist/`,
            `cd ${prismaSchemaDirectory}`, 
            "prisma generate",
            `rsync -av -r ${prismaSchemaDirectory}/.generated_python/node_modules/prisma/ ${tempFolder}/out/ --exclude=out/ --exclude=dist/`,
            `cd ${tempFolder}/out/`, 
            'for f in query-engine* ; do mv -- "$f" "prisma-$f" ; done',
            `zip -r ${tempFolder}/dist/out.zip .`, 
            `cd ${tempFolder}/`
        ];

        const {stdout, stderr} = await execAsync(commands.join('; '), {
            shell: "/bin/bash"
        });

        const zipFile = `${tempFolder}/dist/out.zip`;

        console.log("Built Prisma Package: ", zipFile);

        return new pulumi.asset.FileArchive(zipFile);
    }

sarwaan001 commented 1 year ago

also added the following lambda layer for ssl

arn:aws:lambda:us-east-1:034541671702:layer:openssl-lambda:1

following environment variables:

variables: {
                DATABASE_URL_SECRET_ARN: props.connectionString.arn,
                PRISMA_CLI_BINARY_TARGETS: "native,rhel-openssl-1.0.x", 
                PRISMA_HOME_DIR: "/var/task"
            }

following pyproject.toml

[tool.prisma]
# cache engine binaries in a directory relative to your project
binary_cache_dir = '.generated_python'

and client/generator in schema.prisma

generator client {
  provider = "prisma-client-js"
  previewFeatures = ["postgresqlExtensions"]  // Enable the postgresqlExtensions. Currently in preview
  output = "./generated"
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

// generator
generator pythonClient {
  provider = "prisma-client-py"
  output = "./PrismaPy"
  recursive_type_depth = 5
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

m-roberts commented 1 year ago

When I naively deploy this to AWS Lambda using sls deploy, I get ModuleNotFoundError: No module named 'prisma.models' which I think indicates that the client wasn't generated. Is there a trick or any suggestions to get this to generate correctly to be used with the Serverless Framework? I was also looking for ways to run commands after the pip installation so that I could put python3 -m prisma generate there but I couldn't find any.

You can use serverless-scriptable-plugin. Here is roughly how I have managed to work with Prisma Python client on AWS Lambda with Serverless Framework.

Some notes:

This implementation uses pyproject.toml and client/generator in schema.prisma as per sarwaan001's previous comment
Paths will probably need updating for your system. e.g. Path to generated models is hard-coded for a GitHub Actions runner, and schema.prisma location is for my project

`serverless.yml`

plugins:
  - serverless-scriptable-plugin

custom:
  scriptable:
    hooks:
      before:package:createDeploymentArtifacts: ./package-with-prisma-client.sh

package:
  artifact: .serverless/package.zip

`./package-with-prisma-client.sh`

#!/bin/bash

PACKAGE_FILE=.serverless/package.zip
rm -f $PACKAGE_FILE && rm -rf output && mkdir -p output

# Get non-Prisma dependencies
pip install -r requirements.txt --target output/libs

# Install Prisma and generate client
pip install prisma
prisma generate --schema ../prisma/schema.prisma --generator pythonClient
# Copy generated models
cp -R /opt/hostedtoolcache/Python/3.11.6/x64/lib/python3.11/site-packages/prisma* output/libs

# Add Prisma engine binaries to zip with `prisma-` prefix
mkdir output/prisma
cp -R .generated_python/node_modules/prisma/query-engine* output/prisma/
(
  cd output/prisma/
  for f in query-engine*; do
    mv -- "$f" "prisma-$f"
  done
  zip -r ../../$PACKAGE_FILE .
)

# Add dependencies to zip
(cd output/libs && zip -r ../../$PACKAGE_FILE . -x '*__pycache__*')

# Add app to zip
zip -r $PACKAGE_FILE your-src-folder -x '*__pycache__*'

thalesvon commented 3 months ago

Thanks @m-roberts for posting about serverless-scriptable-plugin. I came with a similar solution using a script to package two separate zip files:

layer.zip: zip layer with all python requirements
.serverless/package.zip: zip with source code and prisma binaries.

serverless.yml

custom:
  scriptable:
    hooks:
      before:package:createDeploymentArtifacts: ./package-with-prisma-client.sh

layers:
  pythonRequirements:
    compatibleRuntimes:
      - python3.10
    compatibleArchitectures:
      - x86_64
    package:
      artifact: output/layer.zip

package:
  artifact: .serverless/package.zip

`package-with-prisma-client.sh`

# #!/bin/bash

set -eE -o functrace

failure() {
  local lineno=$1
  local msg=$2
  echo "Failed at $lineno: $msg"
}
trap 'failure ${LINENO} "$BASH_COMMAND"' ERR

LAYER_FILE=layer.zip
PACKAGE_FILE=.serverless/package.zip

rm -f $LAYER_FILE && rm -rf output && mkdir -p output

# Get non-Prisma dependencies
pip install -r requirements.txt --target output/libs --platform manylinux2014_x86_64 --implementation cp --python-version 3.10 --only-binary=:all: --upgrade
prisma generate

# Get packages path, in my setup I am using venv 
PYTHON_PACKAGES_PATH=$(pip -V | cut -d ' ' -f4 | sed 's![^/]*$!!')
cp -R ${PYTHON_PACKAGES_PATH}prisma* output/libs

mkdir -p output/python
cp -r output/libs/* output/python/

cd output && zip -r $LAYER_FILE python
cd ..

# Add Prisma engine binaries to zip with `prisma-` prefix
mkdir -p output/prisma
cp -R .generated_python/node_modules/prisma/query-engine* output/prisma/
(
  cd output/prisma/
  for f in query-engine*; do
    mv -- "$f" "prisma-$f"
  done
zip -r ../../$PACKAGE_FILE .
)

# Add app to zip with prisma client binaries
zip -r $PACKAGE_FILE src handlers.py -x '*__pycache__*'

`schema.prisma`

generator pythonClient {
  provider                    = "prisma-client-py"
  interface                   = "asyncio"
  recursive_type_depth        = 5
  enable_experimental_decimal = true
  binaryTargets = ["native", "rhel-openssl-1.0.x"]
}

naveedgol commented 2 months ago

Future readers should know that this works for python3.10 lambdas as new python versions use OpenSSL 3

RobertCraigie / prisma-client-py

Serverless Framework / AWS Lambda Support? #800