Support building native modules i.e. C/C++

[ivanayov]

We would like to stick to a minor build for all templates, but also allow the use of gcc/make/git/musl (Glibc for Alpine), which is currently not supported.

One of the options is to duplicate the templates and have a separate dev build. This will cost extra support efforts.

The other option is to use build-args and make the dev build optional. This allows both having a minor build and supporting native C/C++ modules if required.

Any other suggestions are welcome.

Python and Ruby are with highest priority.

For Python, one can test with the Pillow module. (Also can try NumPy and Pandas)

[rgee0]

+1 for build-args. I think this would allow us a single Dockerfile for both minimal and dev builds

[affix]

@rgee0 @iyovcheva

For Ruby another solution would be

• Install GCC at the start of the build
• Copy the Gemfile
• Run bundle install
• Uninstall GCC

This would keep the same Dockerfile and not require the user to do anything special to build native extensions.

[kenfdev]

+1 for build-args, too. As @iyovcheva says, it's an extra effort to duplicate template for this officially and maintain it. Being able to dynamically add it with build-args would open more possibilities.

One thing is that, it would be better if we can declare it somewhere in the stack.yml and won't need to set the build-arg every time. Same way as docker-compose allows build-args.

functions:
  foo:
    lang:
      context: python3
      args:
        PKG: git
    handler: ./foo
    image: foo

I admit setting it in the lang is strange but I hope you get the point.

[alexellis]

The build arg may not be exposed generically or explicitly to the user. It may more likely be a flag that corresponds to some extra metadata in template.yml.

In this way during the build some arbitrary option could be specified to pull in a flavour or mode.

Let's first see an example of it working without any CLI code changes and then talk about UX. --build-arg is already supported in the CLI

[kenfdev]

Here's a simple PoC using pandas. (if I understand correctly) https://github.com/kenfdev/faas-pandas-example

Added an ARG in the apk add here

[alexellis]

That looks very cool... does ask or apt have the ability to read from a file too? Can we quote the packages to prevent additional arbitrary commands being injected?

[kenfdev]

does ask or apt have the ability to read from a file too?

AFAIK, it doesn't look possible. I checked the options for apk but doesn't look like there's some options for that.

Can we quote the packages to prevent additional arbitrary commands being injected?

I've tried this but the command fails if I surround it with double quotes.

With double quotes:

Dockerfile:

RUN apk --no-cache add curl "${ADDITIONAL_PACKAGE}" \ 

Build command:

faas build --build-arg 'ADDITIONAL_PACKAGE=make automake gcc g++ subversion python3-dev'

Results:

ERROR - Could not execute command: [docker build --build-arg ADDITIONAL_PACKAGE=make automakegcc g++ subversion python3-dev -t faas-pandas .]

Without double quotes and with arbitrary command:

RUN apk --no-cache add curl ${ADDITIONAL_PACKAGE} \ 

Build command:

faas build --build-arg 'ADDITIONAL_PACKAGE=&& echo "This is DANGEROUS"'

Result:

2018/05/19 22:10:51 ERROR - Could not execute command: [docker build --build-arg ADDITIONAL_PACKAGE=&& echo "This is DANGEROUS" -t faas-pandas .]

Is there any other way to inject something?? Thanks :)

[ivanayov]

@kenfdev This error looks like #392 Did you try with 0.6.9? The PR fixing this is merged in the latest CLI

[kenfdev]

@iyovcheva Thank you but I am aware of that issue and have upgraded to the latest CLI (need to check my home env to see the precise version though). I'm assuming the above errors are due to invalid Dockerfile syntax. If I get rid of the double quotes, it builds as expected (which used to fail before I upgraded the CLI as you've mentioned above: FYI)

[alexellis]

Every language has a template.yml file:

language: python3
fprocess: python3 index.py

We could extend this with build-modes or build-options:

language: python3
fprocess: python3 index.py
build_options:
    - name: dev
      packages: 
         - musl-dev
         - gcc
         - make

This internally would be passed via build args to Docker using the solution proposed by Ken.

The stack.yml file would not change but the build args may look like this:

faas build -f stack.yml --build-option=dev

If the stack file had to be extended (not sure we should):

provider:
  name: faas
  gateway: http://127.0.0.1:8080

functions:
  test:
    lang: python3
    handler: ./test
    image: test
    build_option: dev            # optional

I'm using the word option(s) since mode didn't seem to be liked on the Slack thread.

Things I think we should avoid:

• coupling directly to Docker concepts of build-args in the stack file
• having N templates for build/lite/dev for each language and platform

Bonus idea:

We should also look at whether we can use this concept of extending the templates to cover the ARMHF or Windows platform.

[ivanayov]

@alexellis one concern for this approach - if build_option is set to dev, we may not want to build all of the listed packages.

It may also look like:

functions:
  test:
    lang: python3
    handler: ./test
    image: test
    build_options: gcc make glibc          # optional

[alexellis]

Please can you explain what you mean? Also the build_options flag would only specify dev or nothing, it would never specify packages. The packages would be an option or flavor specified in template.yml.

[ivanayov]

As I thought more, your suggestion looks like the best option. There is no meaningful use case in which someone will need to list the modules as parameters.

[alexellis]

There is a use-case where someone may want a native module - for instance mysql-dev for @affix in his example he shared with me. On Debian I think this is called libmysqlclient-dev.

I wanted to try to abstract away the packages into a logical grouping or option like dev or build.

Technically people can fork the templates repo, or convert to a dockerfile language type after running --shrinkwrap.

The solution proposed in this thread is certainly better off than when we started.

[napperley]

How would this affect Kotlin Native projects? In Kotlin Native it is very common to interop with C libs. For example if Debian is used, and SDL is a dependency (in a Kotlin Native project) then the libsdl1.2-dev package would need to be installed in a container. More recently there is an option to interop with C libs via Klib and Gradle.

[alexellis]

I don't think there is any impact to a Kotlin template. Each template maintains its own Dockerfile with a base image (pick Debian if you like).

[alexellis]

The primary use case here is if someone uses Python pip or Ruby gem that needs make/gcc to build itself during the installation. bcrypt is a good example of that.

[ivanayov]

This is resolved for python and ruby templates. Node works without this feature (AFAIR from my last testing).

Needs figuring out what other templates should support this before closing (php?)