Enable different python docker images

pdiazvargas commented 5 years ago

What

While using juniper for build purposes, I am unable to create an artifact for anything other than python3.6. I need to have the ability to specify the python base image the container will use when packaging my lambda functions.

The default image juniper is currently using is: lambci/lambda:build-python3.6

How

Update the manifest file to take a few docker specific parameters. Either add a docker specific section in the manifest file like:

docker:
  image: python:3.6-alpine

functions:
  router:
    include:
      - ./src/router_lambda

Or create a more global set of parameters:

global:
  docker_image: python:3.6-alpine

functions:
  router:
    include:
      - ./src/router_lambda

Why

Having a developer change the runtime of a lambda function to python 3.6 due to juniper's inability to package using python 3.7 is not acceptable.

positiveintent commented 5 years ago

It looks to me like the proposal is trying to solve 2 issues at once:

The ability to specify a docker container for any given function
The ability to specify common parameters applying to ALL functions: docker, requirements, and common includes

I would advise that the two separate concerns be split and tackled individually since there is no concept in Juniper now which addresses hierarchy/inheritance.

For Docker specification (issue part 1), you could take the simplest approach:

functions:
  router:
    image: python:3.6-alpine
    include:
      - ./src/router_lambda
    requirements: ./src/requirements.txt

Documentation should stipulate that the image field is not required, and if not specified, the default value is python:3.6-alpine
Alternatively you can call the property name runtime (and then provide an internal mapping between lambda choices and docker containers).

For common parameter values (issue part 2):

There are 2 approaches you could take:

Specify a global set of parameters (with properties matching those of individual functions)

global:
image: python:3.6-alpine
include:
- ./src/common/constants.py
requirements: ./src/requirements.txt

Introduce hierarchy - this is the one I favor (since you can have groups of lambdas each with a separate set of common properties), and might actually work out-of-the-box with YAML

functions:

  .globals: &global_template
    image: python:3.6-alpine
    requirements: ./src/requirements.txt

  my_function:
    <<: *global_template
    include:
      - ./src/my_function

  your_function:
    <<: *global_template
    include:
      - ./src/your_function

Worth noting is the fact that using inheritance, the parameter values are always OVERWRITTEN. This might not fit well with specifying common includes, which is why GitLab introduced an extends: property. Below is a sample of what that might look like for us - illustrating the ability that my_function will include two directories: (1) common, (2) my_function
```
functions:

.globals: &global_template
image: python:3.6-alpine
include: ./src/common
requirements: ./src/requirements.txt

my_function:
extends: globals
include:
  - ./src/my_function
```

pdiazvargas commented 5 years ago

Taking a mixed approached, for this particular feature, we should implement the global parameter setting with an override at a function level. That way a developer will have the following options:

Build all functions using the default image
Specify a global docker image to build all functions
Override the image at function level

With this in mind the manifest would look like:

global:
  image: python:3.6-alpine

functions:
  edge:
    image: python:3.8-alpine
    requirements: ./src/edge/requirements.txt
    include:
      - ./src/edge/

  sequential-worker:
    include:
      - ./src/worker/sequential_worker

eabglobal / juniper