Silent failure with dockerizePip on CI

timnoinc commented 1 year ago

Are you certain it's a bug?

[X] Yes, it looks like a bug

Are you using the latest plugin release?

[X] Yes, I'm using the latest plugin release

Is there an existing issue for this?

[X] I have searched existing issues, it hasn't been reported yet

Issue description

When I do serverless package --verbose -c serverless.yml -p dist --stage qa, it works locally, but in Github Actions and in Jenkins the docker command errors out but doesn't seem to include any error message.

It's clear that something has gone wrong, but I haven't been able to find any log files anywhere to find out what. I've removed a good chunk of the functions and configuration in my serverless.yml, but it still happens consistently with this serverless.yml.

I need to package individually for size reasons. I need pip to run in docker because it's compiling science packages.

Service configuration (serverless.yml) content

service: projectName 
frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.10
  environment:
      S3_BUCKET: projectBucket
      DEBUG: true

plugins:
  - serverless-python-requirements
  # - serverless-step-functions

custom:
  pythonRequirements:
    dockerizePip: true

functions:
  Postback:
    module: postback
    handler: postback.lambda_handler

package:
  individually: true
  exclude:
    - .git/**
    - node_modules/**

Command name and used flags

package --verbose -c serverless.yml -p dist --stage qa

Command output

Run serverless/github-action@v3.2
/usr/bin/docker run --name e519d1f229f1572e3f48b3a1f7801d30a01c33_ab7ec9 --label e519d1 --workdir /github/workspace --rm -e "SLACK_WEBHOOK_URL" -e "DEPLOY_ENVIRONMENT" -e "DEPLOY_DESCRIPTION" -e "DEPLOYMENT_ID" -e "AWS_DEFAULT_REGION" -e "AWS_REGION" -e "AWS_ACCESS_KEY_ID" -e "AWS_SECRET_ACCESS_KEY" -e "AWS_SESSION_TOKEN" -e "NPM_CONFIG_USERCONFIG" -e "NODE_AUTH_TOKEN" -e "pythonLocation" -e "PKG_CONFIG_PATH" -e "Python_ROOT_DIR" -e "Python2_ROOT_DIR" -e "Python3_ROOT_DIR" -e "LD_LIBRARY_PATH" -e "INPUT_ARGS" -e "HOME" -e "GITHUB_JOB" -e "GITHUB_REF" -e "GITHUB_SHA" -e "GITHUB_REPOSITORY" -e "GITHUB_REPOSITORY_OWNER" -e "GITHUB_REPOSITORY_OWNER_ID" -e "GITHUB_RUN_ID" -e "GITHUB_RUN_NUMBER" -e "GITHUB_RETENTION_DAYS" -e "GITHUB_RUN_ATTEMPT" -e "GITHUB_REPOSITORY_ID" -e "GITHUB_ACTOR_ID" -e "GITHUB_ACTOR" -e "GITHUB_TRIGGERING_ACTOR" -e "GITHUB_WORKFLOW" -e "GITHUB_HEAD_REF" -e "GITHUB_BASE_REF" -e "GITHUB_EVENT_NAME" -e "GITHUB_SERVER_URL" -e "GITHUB_API_URL" -e "GITHUB_GRAPHQL_URL" -e "GITHUB_REF_NAME" -e "GITHUB_REF_PROTECTED" -e "GITHUB_REF_TYPE" -e "GITHUB_WORKFLOW_REF" -e "GITHUB_WORKFLOW_SHA" -e "GITHUB_WORKSPACE" -e "GITHUB_ACTION" -e "GITHUB_EVENT_PATH" -e "GITHUB_ACTION_REPOSITORY" -e "GITHUB_ACTION_REF" -e "GITHUB_PATH" -e "GITHUB_ENV" -e "GITHUB_STEP_SUMMARY" -e "GITHUB_STATE" -e "GITHUB_OUTPUT" -e "RUNNER_OS" -e "RUNNER_ARCH" -e "RUNNER_NAME" -e "RUNNER_ENVIRONMENT" -e "RUNNER_TOOL_CACHE" -e "RUNNER_TEMP" -e "RUNNER_WORKSPACE" -e "ACTIONS_RUNTIME_URL" -e "ACTIONS_RUNTIME_TOKEN" -e "ACTIONS_CACHE_URL" -e "ACTIONS_ID_TOKEN_REQUEST_URL" -e "ACTIONS_ID_TOKEN_REQUEST_TOKEN" -e "ACTIONS_RESULTS_URL" -e GITHUB_ACTIONS=true -e CI=true -v "/var/run/docker.sock":"/var/run/docker.sock" -v "/home/runner/work/_temp/_github_home":"/github/home" -v "/home/runner/work/_temp/_github_workflow":"/github/workflow" -v "/home/runner/work/_temp/_runner_file_commands":"/github/file_commands" -v "/home/runner/work/projectName/projectName":"/github/workspace" e519d1:f229f1572e3f48b3a1f7801d30a01c33 package --verbose -c serverless.yml -p dist --stage qa --param="deployDescription=develop-latest"
Running "serverless" from node_modules

Packaging projectName for stage qa (us-east-1)

Generated requirements from /github/workspace/postback/requirements.txt in /github/workspace/.serverless/postback/requirements.txt
Installing requirements from "/github/home/.cache/serverless-python-requirements/e29a8fbd0127f5ea361fb998fc6a150bec13d9d834f2dad4fb4e6a5d8497bb71_x86_64_slspyc/requirements.txt"
Docker Image: public.ecr.aws/sam/build-python3.10:latest-x86_64
Using download cache directory /github/home/.cache/serverless-python-requirements/downloadCacheslspyc
Running docker run --rm -v /github/home/.cache/serverless-python-requirements/e29a8fbd0127f5ea361fb998fc6a150bec13d9d834f2dad4fb4e6a5d8497bb71_x86_64_slspyc\:/var/task\:z -v /github/home/.cache/serverless-python-requirements/downloadCacheslspyc\:/var/useDownloadCache\:z public.ecr.aws/sam/build-python3.10\:latest-x86_64 /bin/sh -c 'chown -R 0\\:0 /var/useDownloadCache && python3.10 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 0\\:0 /var/task && chown -R 0\\:0 /var/useDownloadCache'...
Environment: linux, node 18.17.1, framework 3.36.0 (local) 3.36.0v (global), plugin 7.1.0, SDK 4.4.0
Error:
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

1 deprecation found: run 'serverless doctor' for more details
Running "docker run --rm -v /github/home/.cache/serverless-python-requirements/e29a8fbd0127f5ea361fb998fc6a150bec13d9d834f2dad4fb4e6a5d8497bb71_x86_64_slspyc:/var/task:z -v /github/home/.cache/serverless-python-requirements/downloadCacheslspyc:/var/useDownloadCache:z public.ecr.aws/sam/build-python3.10:latest-x86_64 /bin/sh -c chown -R 0\:0 /var/useDownloadCache && python3.10 -m pip install -t /var/task/ -r /var/task/requirements.txt --cache-dir /var/useDownloadCache && chown -R 0\:0 /var/task && chown -R 0\:0 /var/useDownloadCache" failed with: ""

Environment information

Running "serverless" from node_modules
Framework Core: 3.36.0 (local) 3.36.0 (global)
Plugin: 7.1.0
SDK: 4.4.0

`"serverless-python-requirements": "^6.0.1",` in my package.json

kvnn commented 9 months ago

@timnoinc Have you been able to solve this for the CI?

iwootten commented 8 months ago

@timnoinc @kvnn Are you using the serverless github action by any chance?

I found that kept hitting silent errors when usin dockerizePip but also was using the above action. That action was erroring with Node 18 which was mismatched from the version I'd installed within setup-node. On closer inspection that actions Dockerfile is using Node 18 and Python 3.10.

Replacing it with an action that installs serverless seems to have resolved my issues:

- run: npm install -g serverless
- name: serverless deploy
  run: "serverless plugin install -n serverless-python-requirements && serverless deploy --stage=prod --verbose"

serverless / serverless-python-requirements