Proposal: Support multiple-revisions of functions

[alexellis]

Support multiple-revisions of functions

Description

We should consider how to enable / support multiple-revisions of OpenFaaS functions. This is currently done by changing the name of the function before deployment.

If possible we could use a new feature-flag (for backwards compatibility) and conventions on the Docker image tag to support multiple concurrent versions of functions.

Imagine three different git commits and image tags defined in your stack.yml file:

functions:
  # version allows multiple slash commands in one comment
  name: derek
  image: derek:ef5ja

functions:
  # version has rebase command enabled
  name: derek
  image: derek:ffa3j

functions:  
  # version represents latest stable release, indicated via sem-ver
  name: derek
  image: derek:0.6.0 

This would produce 3 endpoints when deployed:

/function/derek-dev
/function/derek-ffa3j
/function/derek-0.6.0

We may want to then add an additional route like /release or /revision which can make use of "smart" aliases:

/release/derek/dev/    latest SHA   (ef5ja)
/release/derek/ffa3j/  other  SHA   (ffa3j)
/release/derek/0.6.0/  specific semver
/release/derek/0.6.1/   specific semver
/release/derek/  latest semver (0.6.1)

When we have two semver versions we can order these using a semver sort so that /release/derek/ points to 0.6.1.

Ideas for feature: Alex / Stefan

[burtonr]

I think it's important to keep the URL path the same for current/latest function versions.

gateway.openfaas.com/function/derek should always point to the most recent version available. I'm not certain how to accomplish this exactly. It could be an additional label latest_function_version: true and the label would need to be removed once another function of the same name is deployed.

For routing to a specific version, we could add the version to the route: gateway.openfaas.com/function/derek/ffa3j in order to "lock" some outside application to a specific version.

The proxy should remove the version from the route when forwarding to the function to avoid forcing the individual functions to know if there is a version attached so that we don't break the path inside the functions.

Behind the scenes, we could use labels to identify the version of the function. That could be created automatically when the function is deployed by taking the label from the container image and creating a label.

There is also the question of how does the deploy work with this in place? Would this mean that the users would need to explicitly remove all the containers themselves? Could we implement some sort of prune command that would clear out all but the latest version of the function? I worry a little about having a lot of containers being deployed while testing/development that may cause issues...

[noppen]

Very much in favour of managed versioning of functions. Ideally the latest on the same endpoint or at least a softlink endpoint that lands you on the last deployed iteration of a function. Under ideal circumstances you would also have a rollback mechanism that allows you to determine a fixed endpoint of a previous version you know your dependent application worked well with. For interoperability as well as migration in our environment this would be extremely useful.

[LucasRoesler]

I like this idea.

With respect to automating smart aliases, i see how the default/latest path could be implemented for semver but not with shas. Wouldn't it be easier to add an alias property to allow the user to manager these aliases. The "default" path would be either the last or the first found in the stack file. So,

functions: 
    # version allows multiple slash commands in one comment 
    name: derek 
    image: derek:ef5ja 

functions: 
    # version has rebase command enabled 
    name: derek 
    image: derek:ffa3j 

functions: 
    # version represents latest stable release, indicated via sem-ver 
    name: derek 
    image: derek:0.6.0

you would get this by default (assuming last in takes the default)

/function/derek-ef5ja 
/function/derek-ffa3j 
/function/derek-0.6.0
/function/derek # aliases /function/derek-0.6.0

but the user could override this using

functions: 
    # version allows multiple slash commands in one comment 
    name: derek 
    image: derek:ef5ja 
    alias:
        - derek-latest

functions: 
    # version has rebase command enabled 
    name: derek 
    image: derek:ffa3j 
    alias:
        - derek-feature-branch

functions: 
    # version represents latest stable release, indicated via sem-ver 
    name: derek 
    image: derek:0.6.0
    alias:
         - derek-stable

/function/derek-ef5ja 
/function/derek-ffa3j 
/function/derek-0.6.0
/function/derek-stable # aliases /function/derek-0.6.0
/function/derek-feature-branch # aliases /function/derek-ffa3j
/function/derek # aliases /function/derek-ef5ja

[alexellis]

Stefan's idea for the priority order for SHA would be the date deployed.

[alexellis]

The aliases idea gets us too far into "configure everything" Vs our style or convention over configuration. When building from a feature branch I think you'd tag the image with the branch name.

@stefanprodan could probably add more of how he does this with customers on Kubernetes with flux.

Alex

[LucasRoesler]

Not adding aliases makes sense

I think I would prefer not automating aliases for shas, because it might be too difficult for someone to consistently understand why and which alias is set. Automting based on "latest tag" and then "semver order" for the default path and then unique names for anything else (shas or branch names) would be pretty clear. You can easily read this directly from the file and predict what will happen.

Using datetime created would mean that the resulting urls would depend on how they were deployed. If you deploy the entire stack file it would be "order of the file". But you could deploy one at a time in a random order.

Note: One downside to semver order is that it does not play well it CI/CD git tag based versions because strict semver does not have a concept of a post-release, i.e. "number of commits after a version". It only has pre-releases, ie "number of versions prior to the next release. See https://github.com/semver/semver/issues/200 and https://github.com/semver/semver/issues/394 This is not an issue, but just something to be aware of. Any strict semver ordering will not be able to properly order nightly or CI/CD builds that use git describe to generate the tag

[stefanprodan]

Why would anyone use sem ver for nightly builds? Usually you would use the short SHA for nightly builds or pre releases and strict sem ver for releases. It's easy to follow this patter and from a CI perspective it's 10 lines of bash: https://github.com/stefanprodan/flagger/blob/master/.travis.yml#L22

[LucasRoesler]

I only brought it up because I have been on multiple teams where the consensus was "we use semver" but then there were unexpected differences in how tools behave because people thought that git describe --tags would produce semver, but this is not a valid semver: v3.2.2-17-g98f8c58. Or it is a semver but is has a different meaning in git and in semver, so the order is not what you expect. This discussion probably isn't so helpful in this thread, we could move it to slack, but it might be something that we need to address in documentation or FAQ at some point.

[imishravmw]

Hi @alexellis , I have some thought about function versioning. Right now, when we do "faas-cli new" we get a yml file where image tag is latest. I suggest to add a new flag "-v " to following CLI faas-cli new faas-cli build faas-cli push faas-cli deploy

Let me explain how this new version flag can behave in all above CLI faas-cli new -v Currently above CLI generates a yml file where image is tagged with :latest label, so when someone gives i.e. "faas-cli new -v v2" the yml file will contain image with label ":v2" , default could remain same as ":latest"

faas-cli build -v This will build docker image having version tag, even if yml file contains image with ":latest" label, so version flag should be given precedence over already existing label of image.

faas-cli push -v This should push docker image with version

faas-cli deploy -v This should deploy docker images with version label, irrespective of what is mentioned in yml file. also we should be adding label to Pod and service for function ( in case of K8) with version , so if any routing or selection needs to be done based on version labels, it can be done.

Using version labels on pod can help in rolling updates or rollback to previous version (this can be discussed later)

Let me know if above proposal suits the purpose.

[Bessonov]

Disclaimer: I'm not OpenFaas user yet. Sorry if I write some dumb stuff :)

This proposal is very valuable for us. We need to support different client versions (an android/ios app) on the same time and from similar (branches/tags) code base.

I'm not sure that inference from an image tag or deployment date is the right path. But I think alias is a good idea. Maybe weighted aliases for canary deployment.

Furthermore, I think there is more power behind the routing. How about to have some API for routing decisions? For example, I can write an small function, parse request, get list of deployed versions and forward request,may be with additional data, to specific version.

Let me know if above proposal suits the purpose.

@imishravmw nice, but I think a possibility to override every yaml part is better.

[alekar]

Versioning functions is important for canary testing. In some FaaS offerings, one can achieve this with an alias which directs a percentage of traffic to a version, which is convenient. It's important to be able to dial traffic both ways on aliases to roll back.

One issue with aliases is they are out of band of the container runtime. Which suggests that the control plane needs it's own persistence layer (database) synchronized with the runtime. I feel that not needing it's own persistence layer is a strength of the current system.

How about allowing deploying a new 'version' of function, with the scaling pinned to a maximum? Then the provider can load balance evenly across containers from all versions of the function. This allows one to canary functions.

To maintain simplicity, faas-cli can require explicitly specifying a version on operations like remove and invoke if it detects more than 1 version of a function is deployed.

Lastly, about 'version'. It is possible to use monotonically increasing number independent of code version for this field. Does openfaas have to be opinionated about which 'deployment version' is which 'code version'? Especially when that can be queried via the info handler if required?

[GabrielAraujo]

One thing to consider is the faas is widely recommended for mobile backend..

If I have an endpoint that is already implemented in the app and due to some design changes we had to change the scheme of the body response I can't just overlap the function with the new one, I have to version it by using /v1/.. and /v2/.. on its path.

The only way it can be done now in openfaas is by using another name which would add a little complexity to the process of deploying new versions and also would not be a good solution.

Can we continue the discussion on this?

[lordvlad]

Using a ci pipeline, its easy to accomplish versioning with current means (excluding smart alias, which I don't like anyhow). Just override the function name to include the version: faas deploy --name $NAME-$VERSION. Only I would like to have the version separated by a slash: faas deploy --name $NAME/$VERSION. Allowing a slash in the version name would already satisfy my needs

[alexellis]

This is largely a solved problem thanks to Service Mesh Interface. Without any changes to OpenFaaS I was able to support almost all the use-cases including traffic weighting and splitting, a full lab is available here: https://github.com/openfaas/openfaas-linkerd-workshop

For a lighter-weight version with URL and domain mapping see FunctionIngress:

https://github.com/openfaas/ingress-operator https://docs.openfaas.com/reference/ssl/kubernetes-with-cert-manager

FunctionIngress can also be used for blue/green and revision support.