Leaving the Headers

[bubba-h57]

In order for others to build extensions on top of the BREF layer, we need to leave these headers in the layer.

https://github.com/mnapoli/bref/blob/0.3/runtime/php/php.Dockerfile#L462

[mnapoli]

Thanks for bringing that up. Just to be sure I understand, the files in /opt/bref/include are necessary to:

• compile PHP extensions using the same Docker image?
• or at runtime to load PHP extensions that were compiled separately and are provided by a separate layer?

TBH I have no idea what those files are for but a quick search suggests it's for the first case.

What we need to make sure is that the Bref runtime/layer can support extra extensions added via extra layers.

But the current Dockerfile in Bref is not necessarily meant as an extension point for building custom extensions (at least right now).

[bubba-h57]

It is indeed for compiling additional PHP extensions, but there is no reason anyone requires the Dockerfile. The existence of the layer, with the headers, is all that is required.

[mnapoli]

I don't understand how one would proceed to compile extensions using the layer files. Are you thinking about downloading the zip of the layer, unzipping it and using the header files in there to compile PHP extensions on a development machine? I'm not sure layers can be downloaded directly 🤔

The downside I see is that it would add files to the layer. And if I understand correctly those files are not necessary at runtime.

[prolic]

Maybe the problem really comes from that side, that there is no clear path documented on how to compile and add extra extensions to bref when needed. Struggling with this myself I took a different approach yesterday. I compiled php with the exact same version locally and compiled then my needed extension. This is put simply into the repository of my lambda code and enabled the extension. That's it. Works like a charm. Might not work as good for windows or mac users, but not a problem for me since I work on a linux machine.

On Wed, Jan 30, 2019, 21:57 Matthieu Napoli <notifications@github.com wrote:

I don't understand how one would proceed to compile extensions using the layer files. Are you thinking about downloading the zip of the layer, unzipping it and using the header files in there to compile PHP extensions on a development machine? I'm not sure layers can be downloaded directly 🤔

The downside I see is that it would add files to the layer. And if I understand correctly those files are not necessary at runtime.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/mnapoli/bref/issues/212#issuecomment-458952767, or mute the thread https://github.com/notifications/unsubscribe-auth/AAYEvDepxXl3oWRNevJVhvBHN0WUVsrsks5vIaTngaJpZM4aY8v9 .

[bubba-h57]

I don't understand how one would proceed to compile extensions using the layer files. Are you thinking about downloading the zip of the layer, unzipping it and using the header files in there to compile PHP extensions on a development machine? I'm not sure layers can be downloaded directly

Yes, any layer that can be used in a Lambda Job can also provide the layer package as a downloadable zip.

aws lambda get-layer-version --layer-name arn:aws:lambda:us-east-1:416566615250:layer:php-73 --version-number 1

Will provide a dowload URL in the response, good for 10 minute, allowing me to download our BREF layer. Or better yet, in PHP

#!/usr/bin/env php
<?php
require __DIR__ . '/vendor/autoload.php';

use GuzzleHttp\Client;
use Aws\Lambda\LambdaClient;

$config = yaml_parse_file(__DIR__ . '/config.yml');

$client = LambdaClient::factory([
    'version' => '2015-03-31',
    'region'  => 'us-east-1'
]);

$result = $client->getLayerVersion($config['bref']);
$layer = ($result->get('Content'));

(new Client())->request('GET', $layer['Location'], ['sink' => __DIR__ . '/bref-layer.zip']);

At this point the user has a couple of options for what they want to do. But the simplest is to throw the zip into a docker container, build the extensions they want, in the directory they want, and add that layer on top of the BREF layer.

Here is a partial example of the docker file I use for such things.

# This is the image we use for compiling things for Lambda.
# It already has all the tools in place, and we can just pull
# it from Dockerhub:
# https://cloud.docker.com/u/stechstudio/repository/docker/stechstudio/aws-lambda-build
FROM  stechstudio/aws-lambda-build
LABEL authors="Bubba Hines <bubba@stechstudio.com>"
LABEL vendor="Signature Tech Studio, Inc."
LABEL home="https://github.com/stechstudio/bref-extensions"

WORKDIR /root

# We need a base path for all the sourcecode we will build from.
ENV BUILD_DIR="/tmp/build"

# We need a base path for the builds to install to. This path must
# match the path thatour layer will be unpackaged to in Lambda.
ENV INSTALL_DIR="/opt/sts/"

# We need some default compiler variables setup
ENV PKG_CONFIG_PATH="${INSTALL_DIR}/lib64/pkgconfig:${INSTALL_DIR}/lib/pkgconfig:/opt/bref/lib64/pkgconfig:/opt/bref/lib/pkgconfig:${PKG_CONFIG_PATH}" \
    PKG_CONFIG="/usr/bin/pkg-config" \
    PATH="${INSTALL_DIR}/bin:/opt/bref/bin:${PATH}"

ENV LD_LIBRARY_PATH="${INSTALL_DIR}/lib64:${INSTALL_DIR}/lib:/opt/bref/lib64:/opt/bref/lib:${LD_LIBRARY_PATH}"

# Lets get the bref layer in place.
COPY bref-layer.zip /tmp/bref-layer.zip

RUN mkdir -p /opt \
 && unzip /tmp/bref-layer.zip -d /opt \
 && mkdir -p ${BUILD_DIR}

###############################################################################
# Ghostscript
ENV VERSION_GS=9.26
ENV GS_BUILD_DIR=${BUILD_DIR}/ghostscript

RUN set -xe; \
    mkdir -p ${GS_BUILD_DIR}/bin; \
    curl -Ls https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs${VERSION_GS//./}/ghostpdl-${VERSION_GS}.tar.gz \
  | tar xzC ${GS_BUILD_DIR} --strip-components=1

WORKDIR  ${GS_BUILD_DIR}/

RUN set -xe; \
    ./configure \
     --prefix=${INSTALL_DIR} \
    --disable-compile-inits \
    --enable-dynamic        \
    --with-drivers=FILES \
    --with-system-libtiff \
 && make install \
 && mkdir -p ${INSTALL_DIR}/include/ghostscript \
 && install -v -m644 base/*.h ${INSTALL_DIR}/include/ghostscript \
 && mkdir -p ${INSTALL_DIR}/share/ghostscript \
 && curl -Ls http://downloads.sourceforge.net/gs-fonts/ghostscript-fonts-std-8.11.tar.gz | tar xzC ${INSTALL_DIR}/share/ghostscript \
 && curl -Ls http://downloads.sourceforge.net/gs-fonts/gnu-gs-fonts-other-6.0.tar.gz | tar xzC ${INSTALL_DIR}/share/ghostscript \
 && /usr/bin/fc-cache -v ${INSTALL_DIR}/share/ghostscript/fonts/

###############################################################################
# Imagemagic
ENV VERSION_IMAGEMAGICK=6.9.10-25
WORKDIR  ${INSTALL_DIR}/
RUN set -xe; \
    curl -Ls https://imagemagick.org/download/linux/CentOS/x86_64/ImageMagick-libs-${VERSION_IMAGEMAGICK}.x86_64.rpm \
  | rpm2cpio | cpio -idmv \
 && curl -Ls https://imagemagick.org/download/linux/CentOS/x86_64/ImageMagick-${VERSION_IMAGEMAGICK}.x86_64.rpm \
  | rpm2cpio | cpio -idmv \
 && mv /opt/sts/usr/bin/* /opt/sts/bin

###############################################################################
# Imagic PHP Extension
ENV VERSION_IMAGICK=3.4.3
ENV IMAGICK_BUILD_DIR=${BUILD_DIR}/imagick
RUN set -xe; \
    mkdir -p ${IMAGICK_BUILD_DIR}; \
    curl -Ls https://pecl.php.net/get/imagick-${VERSION_IMAGICK}.tgz\
  | tar xzC ${IMAGICK_BUILD_DIR} --strip-components=1

......

The downside I see is that it would add files to the layer. And if I understand correctly those files are not necessary at runtime.

As you can see in my dockerfile, I build my command line tools and extensions in /opt/sts, not /opt/bref, which means I don't affect the bref extension at all. However, I need the libraries and headerfiles from the bref layer if my layer is going to play nice with it.

As for adding files that are not necessary at runtime. The individual compiling their own layer/extensions would be responsible for what they determine is needed at runtime and packaging that up into their own layer. The 'bref' layer needs do nothing more than provide the requisite information to allow other layers to enteract with it. Currently, our bref layer has everything we need except the header files from the include/ directory,

Including those in the layer will indeed mean adding files to the bref layer that bref itself does not need at runtime. Not providing those files signals that bref does not intend to work well with other layers. As you can see from my sample above (which is from a production project) I personally need a lot more than merely PHP extensions (Ghostscript, Libvips) and some the extensions I need require compiling libraries that the extensions rely on (ie. Imagick Ext needs Imagic compiled and installed).

[bubba-h57]

Maybe the problem really comes from that side, that there is no clear path documented on how to compile and add extra extensions to bref when needed.

We are working to get that documented, there are simply a lot of moving parts with the new release and getting everything written down is a challenge. It was in writing up documenation for it, that I discovered the need for the header files and thus this ticket.

[mnapoli]

:+1: thanks for the details that's clearer now.

Do you reckon it will affect a lot the file size of the layer? Because if it's not that significant we can add them.

[bubba-h57]

They are all text files, it looks like 14M unzipped and 2.7M zipped.

[bubba-h57]

After considering it more, I think what we want is a reference layer. That way we do not mess with the current layers at all. Simply have a new layer, call it the bref-reference layer, and document which runtimes the reference layer is suitable for. Then, anyone can pull down the reference layer to do compiling work for their own executables, binaries, or layers on top of bref?

[nealio82]

That might be useful. I needed to add mysqli the other day (playing with getting Wordpress running), so I ended up just editing php.Dockerfile then running the publish script to generate the layer for me. For newbies that might not be such an easy thing to find / figure out!

[mnapoli]

Yes that's a good idea, that could be doable without adding too much complexity I think. If I understand correctly we would push a new runtime where we don't remove the header files (those could be removed when zipping the php and php-fpm layers).

[mnapoli]

Issue made obsolete by https://github.com/brefphp/extra-php-extensions