Runtime exited with error with customized Lambda image

[yike5460]

Hi folks

I been exploring our container support in Lambda, for both official container image (public.ecr.aws/lambda/python:3.7) and customized image with RIC/RIE (python:buster). The official image works fine, however customized image failed with error for local invoke curl -X POST "http://localhost:9000/2015-03-31/functions/function/invocations" -d '{"payload":"hello world!"}', or Lambda console invoke. Not sure it's dockerfile compose, RIC/RIE or other issue, anyone had been met the same issue?

local env:

$ docker run -p 9000:8080 local-lambda-python3.8-custom-ocr:latest
time="2020-12-28T04:18:10.654" level=info msg="exec '/usr/local/bin/python' (cwd=/opt, handler=app.handler)"START RequestId: ad5c6337-7132-4619-8165-b7456ea4ae01 Version: $LATEST
time="2020-12-28T04:18:13.918" level=info msg="extensionsDisabledByLayer(/opt/disable-extensions-jwigqn8j) -> stat /opt/disable-extensions-jwigqn8j: no such file or directory"
time="2020-12-28T04:18:13.918" level=warning msg="Cannot list external agents" error="open /opt/extensions: no such file or directory"
Traceback (most recent call last):
  File "/usr/local/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/local/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/opt/awslambdaric/__main__.py", line 21, in <module>
    main(sys.argv)
  File "/opt/awslambdaric/__main__.py", line 17, in main
    bootstrap.run(app_root, handler, lambda_runtime_api_addr)
  File "/opt/awslambdaric/bootstrap.py", line 416, in run
    event_request = lambda_runtime_client.wait_next_invocation()
  File "/opt/awslambdaric/lambda_runtime_client.py", line 76, in wait_next_invocation
    response_body, headers = runtime_client.next()
AttributeError: 'NoneType' object has no attribute 'next'
Executing 'app.handler' in function directory '/opt'
time="2020-12-28T04:18:14.276" level=warning msg="First fatal error stored in appctx: Runtime.ExitError"
time="2020-12-28T04:18:14.276" level=warning msg="Process 16(python) exited: Runtime exited with error: exit status 1"
time="2020-12-28T04:18:14.276" level=error msg="Init failed" InvokeID= error="Runtime exited with error: exit status 1"
time="2020-12-28T04:18:14.276" level=warning msg="Failed to send default error response: ErrInvalidInvokeID"
time="2020-12-28T04:18:14.277" level=error msg="INIT DONE failed: Runtime.ExitError"
time="2020-12-28T04:18:14.277" level=warning msg="Reset initiated: ReserveFail"

Lambda console:

START RequestId: 0737419e-378e-4989-8d59-e54092369fed Version: $LATEST
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/opt/awslambdaric/__main__.py", line 21, in <module>
    main(sys.argv)
  File "/opt/awslambdaric/__main__.py", line 17, in main
    bootstrap.run(app_root, handler, lambda_runtime_api_addr)
  File "/opt/awslambdaric/bootstrap.py", line 416, in run
    event_request = lambda_runtime_client.wait_next_invocation()
  File "/opt/awslambdaric/lambda_runtime_client.py", line 76, in wait_next_invocation
    response_body, headers = runtime_client.next()
AttributeError: 'NoneType' object has no attribute 'next'
Executing 'app.handler' in function directory '/opt'
END RequestId: 0737419e-378e-4989-8d59-e54092369fed
REPORT RequestId: 0737419e-378e-4989-8d59-e54092369fed  Duration: 5804.95 ms    Billed Duration: 5805 ms    Memory Size: 128 MB Max Memory Used: 18 MB  
RequestId: 0737419e-378e-4989-8d59-e54092369fed Error: Runtime exited with error: exit status 1
Runtime.ExitError

My dockerfile:

# Define function directory
ARG LAYER_DIR="/opt"

FROM lambci/lambda-base:build as build-image

# Install aws-lambda-cpp build dependencies
RUN yum -y install \
  g++ \
  make \
  cmake \
  unzip \
  libcurl4-openssl-dev

# Include global arg in this stage of the build
ARG LAYER_DIR
# Create function directory
RUN mkdir -p ${LAYER_DIR}
# Install the runtime interface client
RUN pip install \
        --target ${LAYER_DIR} \
        awslambdaric

# Add layer start
ARG AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-"cn-northwest-1"}
ARG AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-"xxxx"}
ARG AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-"xxxx"}
ENV AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}

RUN yum install -y curl unzip

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install

# layer1
RUN curl $(aws lambda get-layer-version-by-arn --arn arn:aws-cn:lambda:cn-northwest-1:xxxx:layer:xxxx:3 --query 'Content.Location' --output text) --output layer.zip
RUN unzip layer.zip -d ${LAYER_DIR}
RUN rm layer.zip

# layer2
RUN curl $(aws lambda get-layer-version-by-arn --arn arn:aws-cn:lambda:cn-northwest-1:xxxx:layer:xxxx:1 --query 'Content.Location' --output text) --output layer.zip
RUN unzip layer.zip -d ${LAYER_DIR}
RUN rm layer.zip
# Add layer done

# Multi-stage build: grab a fresh copy of the base image, use custom image instead of official one
FROM python:buster

# Include global arg in this stage of the build
ARG LAYER_DIR

# Copy in the build image dependencies
WORKDIR ${LAYER_DIR}
COPY --from=build-image ${LAYER_DIR} .

COPY app.py .

ADD https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie /usr/bin/aws-lambda-rie
RUN chmod 755 /usr/bin/aws-lambda-rie

COPY entry.sh /
RUN chmod 755 /entry.sh

# Production env
ENTRYPOINT [ "/entry.sh" ]
CMD [ "app.handler" ]

And entry.sh embedded

#!/bin/sh
if [ -z "${AWS_LAMBDA_RUNTIME_API}" ]; then
    exec /usr/bin/aws-lambda-rie /usr/local/bin/python -m awslambdaric $1
else
    exec /usr/local/bin/python -m awslambdaric $1
fi

[stefanbotez]

Hello, @yike5460.

I see that in your Dockerfile you are using lambci/lambda-base:build to build the Runtime Interface Client: FROM lambci/lambda-base:build as build-image

The image you are using seems to be using Python 2.7 and this version is not supported in the Runtime Interface Client.

At the moment, RIC only supports Python versions 3.6.x up to and including 3.9.x as stated in the Requirements section.

I suggest trying to update the Python version from the image or use another image with the required Python version.

Feel free to reach out if you are still facing issues.

[yike5460]

@stefanbotez thanks for your reminder, forget to update. Already solved by using public.ecr.aws/bitnami/python:3.7 or python:buster. Also note that both build-image and multi-stage build should be the same, otherwise issues like dependencies missing happens.