Amazon Linux 2023 (AWS Lambda - Node 20)

[garyrutland]

With a lot of trial and error using various snippets from other issues and your own GitHub actions config I was able to rig together the following Dockerfile that built audiowaveform.

FROM amazonlinux:2023

WORKDIR /tmp

RUN dnf update -yq && \
    dnf install -yq make cmake3 automake libtool gcc gcc-c++ wget tar \
        gzip zip libcurl-devel zlib-static libpng-static xz git python python-devel \
        bzip2-devel which gd-devel

RUN wget http://sourceforge.net/projects/mad/files/libid3tag/0.15.1b/libid3tag-0.15.1b.tar.gz && \
    tar xzf libid3tag-0.15.1b.tar.gz && \
    cd libid3tag-0.15.1b && \
    ./configure --enable-static --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make && \
    make install

RUN wget https://netix.dl.sourceforge.net/project/mad/libmad/0.15.1b/libmad-0.15.1b.tar.gz && \
    tar xzf libmad-0.15.1b.tar.gz && \
    cd libmad-0.15.1b && \
    sed -i 's/ -fforce-mem//' configure && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget https://ftp.osuosl.org/pub/xiph/releases/flac/flac-1.3.0.tar.xz && \
    tar xf flac-1.3.0.tar.xz && \
    cd flac-1.3.0 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget http://downloads.xiph.org/releases/ogg/libogg-1.3.4.tar.gz && \
    tar xf libogg-1.3.4.tar.gz && \
    cd libogg-1.3.4 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.gz && \
    tar xf libvorbis-1.3.6.tar.gz && \
    cd libvorbis-1.3.6 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget https://archive.mozilla.org/pub/opus/opus-1.3.1.tar.gz && \
    tar xzf opus-1.3.1.tar.gz && \
    cd opus-1.3.1 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN git clone https://github.com/libgd/libgd.git && \
    cd libgd && \
    mkdir build && \
    cd $_ && \
    cmake3 -DBUILD_STATIC_LIBS=1 -DENABLE_PNG=1 .. && \
    make && \
    mv Bin/libgd.a /lib64

RUN wget https://github.com/libsndfile/libsndfile/archive/refs/tags/1.1.0.tar.gz && \
    tar xzf 1.1.0.tar.gz && \
    cd libsndfile-1.1.0 && \
    mkdir build && \
    cd build && \
    cmake -D ENABLE_MPEG=0 -D ENABLE_TESTING=0 -D BUILD_SHARED_LIBS=0 -D BUILD_REGTEST=0 -D BUILD_PROGRAMS=0 -D BUILD_TESTING=0 -D BUILD_EXAMPLES=0 .. && \
    make && \
    make install

RUN wget https://boostorg.jfrog.io/artifactory/main/release/1.85.0/source/boost_1_85_0.tar.gz && \
    tar xf boost_1_85_0.tar.gz && \
    cd boost_1_85_0 && \
    ./bootstrap.sh --libdir=/lib64 --includedir=/usr/include && \
    ./b2 link=static install

RUN wget https://github.com/bbc/audiowaveform/archive/1.10.1.tar.gz && \
    tar xzf 1.10.1.tar.gz && \
    cd ./audiowaveform-1.10.1 && \
    mkdir ./build && \
    cd ./build && \
    cmake3 -D ENABLE_TESTS=0 -D CMAKE_BUILD_TYPE=Release -D BUILD_STATIC=1 .. && \
    cmake --build . --config Release

After building this using docker build -t audiowaveform . and then running docker run --rm -it audiowaveform sh you could then execute ./audiowaveform-1.10.1/build/audiowaveform --help which would spit out the help information as expected.

However, despite flagging this to build as static if I then update my Dockerfile to the following:

FROM amazonlinux:2023 as builder

# everything from above

FROM amazonlinux:2023

WORKDIR /tmp

COPY --from=builder /tmp/audiowaveform-1.10.1/build/audiowaveform /tmp/audiowaveform

If you then run the build and run commands to get into the container you can see the file is there but when executing ./audiowaveform --help I get the following error:

./audiowaveform: error while loading shared libraries: libicudata.so.67: cannot open shared object file: No such file or directory

We're trying to build this so that it can be used as an AWS Lambda Layer for Node 20 (which used Amazon Linux 2023). I've tried installing icu enabling static but still not luck. Am I missing something to get this working?

We have a fallback to use Node 18 as that is running Amazon Linux 2 and we have a working binary for that, but it would be nice to use Node 20.

Thanks, Gary

[chrisn]

The libicudata dependency is via Boost, audiowaveform doesn't directly depend on ICU.

You can tell Boost not to use ICU using:

RUN wget https://boostorg.jfrog.io/artifactory/main/release/1.85.0/source/boost_1_85_0.tar.gz && \
    tar xf boost_1_85_0.tar.gz && \
    cd boost_1_85_0 && \
    ./bootstrap.sh --without-icu --libdir=/lib64 --includedir=/usr/include && \
    ./b2 --disable-icu --with-program_options --with-filesystem --with-system --with-regex link=static install

This also just compiles the parts of Boost that you need, which should save some time too.

[garyrutland]

Perfect, I can now use audiowaveform with Node 20 on AWS Lambda.

For anyone else looking, this is a bit of a crude way of doing it but the following will build audiowaveform that can be used as a Layer:

# FROM fedora
FROM amazonlinux:2023 as builder

WORKDIR /tmp

RUN dnf update -yq && \
    dnf install -yq libicu-devel make cmake3 automake libtool gcc gcc-c++ wget tar \
        gzip zip libcurl-devel zlib-static libpng-static xz git python python-devel \
        bzip2-devel which gd-devel

RUN wget http://sourceforge.net/projects/mad/files/libid3tag/0.15.1b/libid3tag-0.15.1b.tar.gz && \
    tar xzf libid3tag-0.15.1b.tar.gz && \
    cd libid3tag-0.15.1b && \
    ./configure --enable-static --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make && \
    make install

RUN wget https://netix.dl.sourceforge.net/project/mad/libmad/0.15.1b/libmad-0.15.1b.tar.gz && \
    tar xzf libmad-0.15.1b.tar.gz && \
    cd libmad-0.15.1b && \
    sed -i 's/ -fforce-mem//' configure && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget https://ftp.osuosl.org/pub/xiph/releases/flac/flac-1.3.0.tar.xz && \
    tar xf flac-1.3.0.tar.xz && \
    cd flac-1.3.0 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget http://downloads.xiph.org/releases/ogg/libogg-1.3.4.tar.gz && \
    tar xf libogg-1.3.4.tar.gz && \
    cd libogg-1.3.4 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget http://downloads.xiph.org/releases/vorbis/libvorbis-1.3.6.tar.gz && \
    tar xf libvorbis-1.3.6.tar.gz && \
    cd libvorbis-1.3.6 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN wget https://archive.mozilla.org/pub/opus/opus-1.3.1.tar.gz && \
    tar xzf opus-1.3.1.tar.gz && \
    cd opus-1.3.1 && \
    ./configure --disable-shared --libdir=/lib64 --build=aarch64-unknown-linux-gnu && \
    make install

RUN git clone https://github.com/libgd/libgd.git && \
    cd libgd && \
    mkdir build && \
    cd $_ && \
    cmake3 -DBUILD_STATIC_LIBS=1 -DENABLE_PNG=1 .. && \
    make && \
    mv Bin/libgd.a /lib64

RUN wget https://github.com/libsndfile/libsndfile/archive/refs/tags/1.1.0.tar.gz && \
    tar xzf 1.1.0.tar.gz && \
    cd libsndfile-1.1.0 && \
    mkdir build && \
    cd build && \
    cmake -D ENABLE_MPEG=0 -D ENABLE_TESTING=0 -D BUILD_SHARED_LIBS=0 -D BUILD_REGTEST=0 -D BUILD_PROGRAMS=0 -D BUILD_TESTING=0 -D BUILD_EXAMPLES=0 .. && \
    make && \
    make install

RUN wget https://boostorg.jfrog.io/artifactory/main/release/1.85.0/source/boost_1_85_0.tar.gz && \
    tar xf boost_1_85_0.tar.gz && \
    cd boost_1_85_0 && \
    ./bootstrap.sh --without-icu --libdir=/lib64 --includedir=/usr/include && \
    ./b2 --disable-icu --with-program_options --with-filesystem --with-system --with-regex link=static install

RUN wget https://github.com/bbc/audiowaveform/archive/1.10.1.tar.gz && \
    tar xzf 1.10.1.tar.gz && \
    cd ./audiowaveform-1.10.1 && \
    mkdir ./build && \
    cd ./build && \
    cmake3 -D ENABLE_TESTS=0 -D CMAKE_BUILD_TYPE=Release -D BUILD_STATIC=1 .. && \
    cmake --build . --config Release

FROM amazonlinux:2023

WORKDIR /tmp

COPY --from=builder /tmp/audiowaveform-1.10.1/build/audiowaveform /tmp/audiowaveform

You can confirm it works by running the following:

docker build -t audiowaveform .
docker run --rm -it audiowaveform sh
./audiowaveform --help

This will spit out the help dialog.

To get the binary out to use as in a Layer you could do the following:

docker build -t audiowaveform .
docker run -it audiowaveform sh
exit
docker ps -a
docker cp<ID_OF_CONTAINER>:/tmp/audiowaveform ./audiowaveform

As I say, quite a crude way of doing it but it does work.

[garyrutland]

Correction, while this does seem to work in amazonlinux:2023 it doesn't seem to work in the Lambda itself. For example, if I use amazon/aws-lambda-nodejs:20 locally and mount audiowaveform to /opt/bin I get the following output:

{
    "error": {
        "errno": -2,
        "code": "ENOENT",
        "syscall": "spawnSync /opt/bin/audiowaveform",
        "path": "/opt/bin/audiowaveform",
        "spawnargs": [
            "--help"
        ]
    },
    "status": null,
    "signal": null,
    "output": null,
    "pid": 0,
    "stdout": null,
    "stderr": null
}

The file is definitely there, so not sure exactly what the issue is yet but will keep looking.

[garyrutland]

Have done some more testing and it seems to be specific to the amazon/aws-lambda-nodejs:20 container.

1. Added another binary to the same folder and was able to successfully call it.
2. Tried mounting the same binary in amazonlinux:2023 and worked without issue.

docker run --rm -v ./bin:/tmp -it amazonlinux:2023 sh

sh-5.2# /tmp/audiowaveform --help
AudioWaveform v1.10.1

Usage:
  /tmp/audiowaveform [options]

Options:
  --help                          show help message
  -v [ --version ]                show version information
  -q [ --quiet ]                  disable progress and information messages
  -i [ --input-filename ] arg     input file name (.mp3, .wav, .flac, .ogg, 
                                  .oga, .opus, .dat, .json)
  -o [ --output-filename ] arg    output file name (.wav, .dat, .png, .json)
  --split-channels                output multi-channel waveform data or image 
                                  files
  --input-format arg              input file format (mp3, wav, flac, ogg, raw, 
                                  opus, dat, json)
  --output-format arg             output file format (wav, dat, png, json)
  -z [ --zoom ] arg (=256)        zoom level (samples per pixel)
  --pixels-per-second arg (=100)  zoom level (pixels per second)
  -b [ --bits ] arg (=16)         bits (8 or 16)
  -s [ --start ] arg (=0)         start time (seconds)
  -e [ --end ] arg (=0)           end time (seconds)
  -w [ --width ] arg (=800)       image width (pixels)
  -h [ --height ] arg (=250)      image height (pixels)
  -c [ --colors ] arg (=audacity) color scheme (audition or audacity)
  --border-color arg              border color (rrggbb[aa])
  --background-color arg          background color (rrggbb[aa])
  --waveform-color arg            waveform color (rrggbb[aa])
  --waveform-style arg (=normal)  waveform style (normal or bars)
  --bar-width arg (=8)            bar width (pixels)
  --bar-gap arg (=4)              bar gap (pixels)
  --bar-style arg (=square)       bar style (square or rounded)
  --axis-label-color arg          axis label color (rrggbb[aa])
  --no-axis-labels                render waveform image without axis labels
  --with-axis-labels              render waveform image with axis labels 
                                  (default)
  --amplitude-scale arg (=1.0)    amplitude scale
  --compression arg (=-1)         PNG compression level: 0 (none) to 9 (best), 
                                  or -1 (default)
  --raw-samplerate arg            sample rate for raw audio input (Hz)
  --raw-channels arg              number of channels for raw audio input
  --raw-format arg                format for raw audio input (s8, u8, s16le, 
                                  s16be, s24le, s24be, s32le, s32be, f32le, 
                                  f32be, f64le, f64be)

See audiowaveform(1) for usage examples

I'm not sure what else I can try at this point, but for some reason there seems to be an issue with calling audiowaveform specifically from a Node 20 Lambda.

[chrisn]

Can you build audiowaveform with an amazon/aws-lambda-nodejs:20 container rather than amazonlinux:2023?

In https://github.com/bbc/audiowaveform/issues/201#issuecomment-2078875992, can you capture stdout and stderr to see if any more detail is output?

[garyrutland]

That was the complete output in my previous comment for the following:

const results = spawnSync("/opt/bin/audiowaveform", ["--help"])
console.log({ results })

I updated the builder above to the following:

FROM amazon/aws-lambda-nodejs:20 as builder

# all the same build steps

FROM amazonlinux:2023

WORKDIR /tmp

COPY --from=builder /tmp/audiowaveform-1.10.1/build/audiowaveform /tmp/audiowaveform

This seems to have done the trick as I'm now getting the help dialog from within my Lambda. Weirdly, it now doesn't work from within the amazonlinux:2023 which AWS claim the Lambda is built from but in all honesty I don't really care if it works.

Thanks for the help.

[garyrutland]

I'll keep this issue open until I can confirm it works out in the wild in an actual Lambda, just in case anything else crops up.

[chrisn]

Thanks! This question comes up so often, it would be really helpful to have a complete example project and step by step guide written up somewhere.

[garyrutland]

If useful (and if I can make it more intuitive) we may well do something. Is there a reason you don't include the binaries as part of the releases?

[chrisn]

The main reason is that I don't use audiowaveform with Lambda so I haven't had a need to produce binaries, so there's work involved ;-) If I did create binaries I'd also have to create and test a Lambda to make sure it all works, as well as provide documentation. (I wasn't necessarily suggesting you do that in my previous message, btw.) And as this is now a really personal project, it's not something I've prioritised spending time on.

[garyrutland]

FYI we have created a public repo to create standalone binaries for audiowaveform and currently support the following OS:

• amazonlinux:2023
• amazon/aws-lambda-nodejs:20

It's probably not perfect, but it's a start. Main thing is that you need to probably want to rename the binary once downloaded and then run chmod +x against it.

I haven't tested the amazonlinux:2023 binary but have downloaded the amazon/aws-lambda-nodejs:20 and can confirm it does work.

https://github.com/lickdlabs/audiowaveform https://github.com/lickdlabs/audiowaveform/releases/tag/1.10.1