Using portable-pypy in AWS Lambda

[KentonParton]

Hello, I was curious if anyone has used pypy or portable-pypy with AWS Lambda custom runtimes? I came across this article but isn't a stable and maintainable solution. I'm sure this would be leveraged extensively by the community if it was available.

[squeaky-pl]

The changes from Portable PyPy were upstreamed to official PyPy downloads (https://www.pypy.org/download.html). This repository won't see new releases because of that.

The article you linked to mentions that they could not get PyPy to run out of the box since it predates merging portable patches to upstream. This is no longer the case.

About stability: For x86_64 PyPy builds to run on a LInux distro it has to provide at least glibc 2.12

The PyPy builds from pypy.org are built against Centos 6 which provides glibc 2.12.

> docker run --rm -ti centos:6
>  rpm -q glibc
glibc-2.12-1.212.el6.x86_64

The lambda images are RedHat-like so they are basically Centos compatible and they ship glibc 2.17 and glibc is always backward compatible.

> docker run --rm -ti --entrypoint bash lambci/lambda
> cat /etc/*-release
NAME="Amazon Linux AMI"
VERSION="2018.03"
ID="amzn"
ID_LIKE="rhel fedora"
VERSION_ID="2018.03"
PRETTY_NAME="Amazon Linux AMI 2018.03"
ANSI_COLOR="0;33"
CPE_NAME="cpe:/o:amazon:linux:2018.03:ga"
HOME_URL="http://aws.amazon.com/amazon-linux-ami/"
VARIANT_ID="201904170905-al2018.03.122.0"
Amazon Linux AMI release 2018.03
> rpm -q glibc
glibc-2.17-260.175.amzn1.x86_64

In fact the binaries from pypy.org run in this setup out of the box:

> docker run --rm -ti --entrypoint bash lambci/lambda
> cd /tmp
> curl https://downloads.python.org/pypy/pypy3.6-v7.3.2-linux64.tar.bz2 --output pypy.tar.bz2
> tar xf pypy.tar.bz2
> pypy3.6-v7.3.2-linux64/bin/pypy
Python 3.6.9 (d38cd66c14b8, Sep 23 2020, 08:01:17)
[PyPy 7.3.2 with GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>>

So basically they are already binary compatible. Everything else is about providing interface between lambda tasks to be consumed and PyPy interpreter. I myself don't have an interest in AWS lambda but it should be fairly easy for somebody who uses it.

[KentonParton]

@squeaky-pl thank you for your insight and for verifying that it will work. Regarding the size of pypy, would it be possible to reduce its size; are there parts that aren't essential? Seeing that pypy and packages are loaded at runtime, the smaller the size the faster the start-up time.

[squeaky-pl]

The size of unpacked PyPy 3 tar is around 133MB.

You can start by removing debugging symbols from bin/libpypy3-c.so.debug saves 15.5 MB You an also remove tests from lib-python/3 saves 21 MB If you are not planning on using pip during the runtime you can probably remove lib-python/3/ensurepip which saves 4 MB You are probably not gonna use tkinter/idle or any GUI support so you can remove lib-python/3/tkinter, lib-python/3/idlelib lib_pypy/_tkinter lib/tcl* lib/tk* which saves 9MB

The standard library could be probably stripped even further depending on exact requirements but it starts getting dangerous from this point on.

[KentonParton]

This is great, thanks @squeaky-pl !