How to install on Amazon Lambda?

[renatoargh]

I have successfully installed audiowaveform on Amazon Linux (which is the same OS for Amazon Lambda), but I can't seem to find a way to create a unique executable file that will include all shared libraries in which audiowaveform depends upon (running ldd audiowaveform shows a lot of them). I can't think of any other way to achieve this. Any hints?

Being able to do this means that I would be able to run it on-demand not worrying about provisioning any EC2 servers (and then saving a lot of $ since my demand is very low).

[ffxsam]

Hey Chris, would you mind posting your CMakeLists.txt file here? In fact, the general steps you took to get this compiling. I'm not sure which parts of the linked article you followed. I can't even get the shared lib version to build at all. Let me know also which packages you installed via yum.

EDIT: Actually I'm not even able to compile audiowaveform normally, without the Lambda stuff. Same errors as before. Boost is being problematic, and I'm curious how you got it to work in this environment.

[chrisn]

Yes, I'll run through the steps again later today, and hopefully produce a Dockerfile that automates everything.

[ffxsam]

Thanks! I tried to do the lifting myself based on the work others started on Ubuntu, so I could open a PR with the Dockerfile & build script. Turned out to be more of a headache than I had anticipated, though!

PS: Static self-contained build is still probably ideal, especially if audiowaveform can't find library files in the same folder as the binary (not sure, haven't tried).

[chrisn]

Here is the Dockerfile I made. The zip file is written to /root/audiowaveform-1.4.0/build/audiowaveform.zip inside the container. The next step is to copy the zip file out of the docker image and try it on Lambda. Over to you!

[ffxsam]

@chrisn Thank you! I tried it and it worked, and it spit out about 20MB worth of stuff. I'll use what you've done here as a basis for a static build Dockerfile and post it here if I get it working.

[chrisn]

A static build will mean compiling most of the dependencies from source, as the -devel rpms generally don't include static libraries. Have you tried the zip file on Lambda? I'm curious whether it works or not.

[ffxsam]

It does indeed work (your version)! I also managed to get a static build made and it's only 2.6MB. My guess is that the shared lib version is including a bunch of system-level libraries that are already present in the Amazon Linux container. (confirmed, I see libstdc++ in there)

Thanks to everyone on this thread for pitching in! I'm happy to share the final Dockerfile and build script if anyone is interested.

[chrisn]

Thanks Sam, the Dockerfile and script would be really helpful. As there's been so much interest here, would you be interested in writing a guide, not just for the build but also how to deploy and run it on Lambda? I could include a link in the documentation, or we could include it here even?

[raedatoui]

@ffxsam def interested and happy to some testing

[ffxsam]

I don't have time at the moment to write a proper guide on Lambda deployment, but I will when I'm able!

I can share my Dockerfile & build script in a bit. Just found an issue or two I need to fix up first.

[ffxsam]

@chrisn @raedatoui Here you go:

audiowaveform.zip

Please feel free to double-check this. Instructions:

1. yarn build to build the container
2. yarn install to fire a container to copy the binary from
3. yarn clean to remove the container and image

I'm using the binary within Lambda and it works great! And now it's just 1.6MB!

Thanks again @ajbarber and @chrisn for your help.

[raedatoui]

@ffxsam confirmed its working on a Lambda function! thank you @chrisn and @ffxsam for your help!

I got it working with Python, here is a snippet if someone else can find this useful

import boto3
import subprocess

LAMBDA_TASK_ROOT = os.environ.get('LAMBDA_TASK_ROOT')
# binary is added to the lambda package
BIN_PATH = '{root}/audiowaveform'.format(root=LAMBDA_TASK_ROOT) 

client = boto3.client('s3')

file_signed_url = client.generate_presigned_url(
    ClientMethod='get_object',
    ExpiresIn=3600,
    Params={
        'Bucket': bucket,
        'Key': input_key
    }
)

get_file = subprocess.Popen(
    ['curl', file_signed_url],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE
)

gen_waveform = subprocess.Popen(
    [
        config.BIN_PATH,
        '--input-format', input_format,
        '--output-format', 'dat',
        '-b', '8', '-z', '256'
    ],
    stdin=get_file.stdout,
    stdout=subprocess.PIPE,
)

get_file.stdout.close()

output, err = gen_waveform.communicate()

client.put_object(Bucket=bucket, Key=output_key, Body=output, ContentType='binary/octet')

There is probably a better way than using curl and signed urls but this works just fine.

[TrevorHinesley]

@ffxsam have you had issues with this throwing an error like the following with flac files:

Failed to read file: filename.flac 
File contains data in an unimplemented format.

The static library on lambda won't seem to process flac files. This is usually due to libsndfile missing I believe, but from your script I see a section with libsndfile in there. Possible it's not getting pushed into the static build properly?

[chrisn]

That error comes from libsndfile (see here). It looks like libsndfile has been built without FLAC support enabled.

[TrevorHinesley]

Hmm any idea how to configure that properly? libsndfile gets made before flac in the above build scripts, so it won't get built with FLAC support enabled. I know there's a cmake option that will force enable those other file types (FLAC included), but if the packages for FLAC, Ogg, and Vorbis exist it should work anyway. Tried rearranging the build sections to have libsndfile build after those packages but right before boost, but it threw an error when I went to use it after building it like that:

RubyError: StandardError: /opt/audiowaveform: error while loading shared libraries: libmvec.so.1: cannot open shared object file: No such file or directory

Everything I've read about this error says it's something to do with glibc, but from what I can tell the amazonlinux:latest Docker image uses a fairly up to date version of that. This is definitely not my area of expertise, so any help is much appreciated :)

[TrevorHinesley]

So the ruby 2.5 runtime (which utilizes amazonlinux:1 in Docker) on Lambda uses an older version of glibc than the amazonlinux:latest docker image (aka amazonlinux:2, which is what the ruby 2.7 runtime on Lambda uses). I'm having some other issues deploying using the ruby 2.7 runtime, so I'm trying to adjust the Dockerfile to just use amazonlinux:1 but I'm running into package issues. Will report back.

[chrisn]

If you can share the Dockerfile you're using, I'll take a look.

[TrevorHinesley]

Thanks so much!

Dockerfiles

Original (from @ffxsam):

FROM amazonlinux:latest
RUN yum -y update
RUN yum -y install make cmake3 autogen automake libtool gcc gcc-c++ wget tar \
  gzip zip libcurl-devel zlib-static libpng-static xz git python-devel \
  bzip2-devel which gd-devel

COPY ./build.sh build.sh
RUN chmod +x build.sh
RUN ./build.sh

Updated:

FROM amazonlinux:1
RUN yum -y update
RUN yum -y install make wget tar gcc gcc-c++
RUN wget https://cmake.org/files/v3.7/cmake-3.7.2.tar.gz && \ 
  tar -xzvf cmake-3.7.2.tar.gz && cd cmake-3.7.2 && \
  ./bootstrap && make && make install
RUN yum -y install autogen automake libtool \
  gzip zip libcurl-devel zlib-static libpng-static xz git python-devel \
  bzip2-devel which gd-devel

COPY ./build.sh build.sh
RUN chmod +x build.sh
RUN ./build.sh

Explanation: switched amazonlinux:latest (which is now amazonlinux:2) to amazonlinux:1 since that's what the ruby 2.5 runtime uses on Lambda. Also, cmake3 isn't working in amazonlinux:1 so I'm trying to build a different version of cmake and use just cmake as the command in build.sh (see the diffs below, not sure if it will work yet... not my specialty lol).

build.sh

Original (from @ffxsam):

#!/bin/sh

set -ex

# Build libid3tag
wget https://netix.dl.sourceforge.net/project/mad/libid3tag/0.15.1b/libid3tag-0.15.1b.tar.gz
tar xzf libid3tag-0.15.1b.tar.gz
cd libid3tag-0.15.1b
sed -i 's/ -fforce-mem//' configure
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libmad
wget https://netix.dl.sourceforge.net/project/mad/libmad/0.15.1b/libmad-0.15.1b.tar.gz
tar xzf libmad-0.15.1b.tar.gz
cd libmad-0.15.1b
sed -i 's/ -fforce-mem//' configure
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libFLAC
wget https://ftp.osuosl.org/pub/xiph/releases/flac/flac-1.3.0.tar.xz
tar xf flac-1.3.0.tar.xz
cd flac-1.3.0
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libogg
wget http://downloads.xiph.org/releases/ogg/libogg-1.3.4.tar.gz
tar xf libogg-1.3.4.tar.gz
cd libogg-1.3.4
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libvorbis
wget http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.gz
tar xf libvorbis-1.3.6.tar.gz
cd libvorbis-1.3.6
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libgd
git clone https://github.com/libgd/libgd.git
cd libgd
mkdir build
cd $_
cmake3 -DBUILD_STATIC_LIBS=1 -DENABLE_PNG=1 ..
make
mv Bin/libgd.a /lib64
cd /

# Build libsndfile (Amazon repo only has earlier 1.0.25 release)
# Requires autogen, automake, and libtool packages
wget https://github.com/erikd/libsndfile/archive/1.0.28.tar.gz
tar xzf 1.0.28.tar.gz
cd libsndfile-1.0.28
sed -i 's/flac >= 1.3.1/flac >= 1.3.0/' configure.ac
./autogen.sh
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build boost
wget https://sourceforge.net/projects/boost/files/boost/1.69.0/boost_1_69_0.tar.gz
tar xf boost_1_69_0.tar.gz
cd boost_1_69_0
./bootstrap.sh --libdir=/lib64 --includedir=/usr/include
./b2 link=static install
cd /

# Build audiowaveform
wget https://github.com/bbc/audiowaveform/archive/1.4.0.tar.gz
tar xzf 1.4.0.tar.gz
mv audiowaveform-1.4.0 audiowaveform
cd $_
mkdir build
cd $_
cmake3 -D ENABLE_TESTS=0 -D BUILD_STATIC=1 ..
make
strip audiowaveform

Updated:

#!/bin/sh

set -ex

# Build libid3tag
wget https://netix.dl.sourceforge.net/project/mad/libid3tag/0.15.1b/libid3tag-0.15.1b.tar.gz
tar xzf libid3tag-0.15.1b.tar.gz
cd libid3tag-0.15.1b
sed -i 's/ -fforce-mem//' configure
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libmad
wget https://netix.dl.sourceforge.net/project/mad/libmad/0.15.1b/libmad-0.15.1b.tar.gz
tar xzf libmad-0.15.1b.tar.gz
cd libmad-0.15.1b
sed -i 's/ -fforce-mem//' configure
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libogg
wget http://downloads.xiph.org/releases/ogg/libogg-1.3.4.tar.gz
tar xf libogg-1.3.4.tar.gz
cd libogg-1.3.4
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libvorbis
wget http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.gz
tar xf libvorbis-1.3.6.tar.gz
cd libvorbis-1.3.6
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libFLAC
wget https://ftp.osuosl.org/pub/xiph/releases/flac/flac-1.3.0.tar.xz
tar xf flac-1.3.0.tar.xz
cd flac-1.3.0
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build libgd
git clone https://github.com/libgd/libgd.git
cd libgd
mkdir build
cd $_
cmake -DBUILD_STATIC_LIBS=1 -DENABLE_PNG=1 ..
make
mv Bin/libgd.a /lib64
cd /

# Build libsndfile (Amazon repo only has earlier 1.0.25 release)
# Requires autogen, automake, and libtool packages
wget https://github.com/erikd/libsndfile/archive/1.0.28.tar.gz
tar xzf 1.0.28.tar.gz
cd libsndfile-1.0.28
sed -i 's/flac >= 1.3.1/flac >= 1.3.0/' configure.ac
./autogen.sh
./configure --disable-shared --libdir=/lib64
make install
cd /

# Build boost
wget https://sourceforge.net/projects/boost/files/boost/1.69.0/boost_1_69_0.tar.gz
tar xf boost_1_69_0.tar.gz
cd boost_1_69_0
./bootstrap.sh --libdir=/lib64 --includedir=/usr/include
./b2 link=static install
cd /

# Build audiowaveform
wget https://github.com/bbc/audiowaveform/archive/1.4.0.tar.gz
tar xzf 1.4.0.tar.gz
mv audiowaveform-1.4.0 audiowaveform
cd $_
mkdir build
cd $_
cmake -D ENABLE_TESTS=0 -D BUILD_STATIC=1 ..
make
strip audiowaveform

Explanation: moved libflac under the other libraries so it would hopefully register them as being installed, and add support for them. Also, cmake3 isn't working in amazonlinux:1 so I'm trying to build a different version of cmake and use cmake (not sure if it will work yet... not my specialty lol).

[chrisn]

I got a working build for amazonlinux:1 using these files.

[TrevorHinesley]

@chrisn man, I can't thank you enough. Unfortunately, libsndfile spits this out during the build:

. . .
checking for flac >= 1.3.1 ... no
checking for ogg >= 1.1.3 ... no
checking for vorbis >= 1.2.3 ... no
checking for vorbisenc >= 1.2.3 ... no

configure: WARNING: *** One or more of the external libraries (ie libflac, libogg and
configure: WARNING: *** libvorbis) is either missing (possibly only the development
configure: WARNING: *** headers) or is of an unsupported version.
configure: WARNING: ***
configure: WARNING: *** Unfortunately, for ease of maintenance, the external libs
configure: WARNING: *** are an all or nothing affair.
. . .

So it's not getting built with support for those three still. I'm going to play around with it and see if I can figure out what's going on.

[TrevorHinesley]

Very strange... can't seem to get it to see the packages. The final summary printed out by libsndfile's build step shows the proper directories from what I can tell (Pkgconfig is where it looks for FLAC and such I believe), but confirms that it's not enabling support for FLAC, Ogg or Vorbis:

-=-=-=-=-=-=-=-=-=-= Configuration Complete =-=-=-=-=-=-=-=-=-=-

  Configuration summary :

    libsndfile version : .................. 1.0.28

    Host CPU : ............................ x86_64
    Host Vendor : ......................... unknown
    Host OS : ............................. linux-gnu

    Experimental code : ................... no
    Using ALSA in example programs : ...... no
    External FLAC/Ogg/Vorbis : ............ no

  Tools :

    Compiler is Clang : ................... no
    Compiler is GCC : ..................... yes
    GCC version : ......................... 4.8.5
    Sanitizer enabled : ................... no
    Stack smash protection : .............. no

  Installation directories :

    Library directory : ................... /lib64
    Program directory : ................... /usr/local/bin
    Pkgconfig directory : ................. /lib64/pkgconfig
    HTML docs directory : ................. /usr/local/share/doc/

[chrisn]

Ah, sorry I spoke to soon... The problem was that the libflac, libvorbis etc pkg-config files were being installed in /usr/lib/pkgconfig and not /usr/lib64/pkgconfig. Setting --prefix=/usr --libdir=/usr/lib64 fixed it. Here's an updated gist.

[TrevorHinesley]

You. Are. A. Saint. That worked! Here is the full setup for the Ruby 2.5 runtime on Lambda for anyone that needs it, with FLAC, Ogg and Vorbis support. Appreciate the work you and @ffxsam did. Here's the steps again for anyone viewing this thread:

1. yarn build to build the container
2. yarn install to fire a container to copy the binary from
3. yarn clean to remove the container and image

OR the build is already zipped in the bin folder :) audiowaveform.zip

[cretueusebiu]

I've put together a repo for building a binary for AWS Lambda (Amazon Linux 2) with Docker https://github.com/flixier/audiowaveform-aws-lambda

[peterj35]

I got this working on Lambda. To save everyone else the heavy lifting, feel free to grab this:

http://ffxsam.s3.amazonaws.com/public/audiowaveform-lambda.zip

When you package up your Lambda function, leave the bin and lib folders intact. Lambda's environment will automatically have /var/task/lib in LD_LIBRARY_PATH so no extra work there is needed. However, you will have to add the bin folder to the path inside your exports.handler function:

process.env.PATH =
    process.env.PATH + ':' + process.env.LAMBDA_TASK_ROOT + '/bin';

Then you can use Node.js to execute audiowaveform.

I know it's not an ideal solution, so if anyone ever figures out how to statically link all those library files into a single binary, please reply to this thread!

I tried this by creating a layer with that zip, but getting the error:

"Command failed: audiowaveform -i fileName --pixels-per-second 20 -b 8 -o output.json\n audiowaveform: error while loading shared libraries: libX11.so.6: cannot open shared object file: No such file or directory\n",

Is the libX11.so.6 binary missing? Is there a way I can install it?

[ffxsam]

This has been fully tested and I'm ready to post it here:

https://github.com/reelcrafter/audiotool-builders

Hope that helps!