Question: Can a function be made ephemeral?

[Th0masL]

I'm new to OpenFaas, and I've been playing with it for couple of days. I'm looking for a way to kill/renew the function/containers after it has been used once.

Expected Behaviour

Everytime a function is called, a new container should be deployed for this function, then do the job, and then get killed. If there's launch delay, 1 or couple containers could be running in advance to limit the launch delay, and once they got killed, a new one is launched for next time. Since this behavior might not be the one most of people want by default, a parameter could be added to enable this behavior on some functions.

Current Behaviour

It seems that right now, the only way I get my container "renewed" is when the autoscaling deploys new containers, and then when the load comes back to normal, then it's killing some of the existing containers to go back to a "low load" state. I think there's another way also do trick the container to die, from within, but it's definitely not clean, and it might create problems and connectivity loss / query errors.

Possible Solution

Adding a flag in the function YML file to define how many time we want to use a container before renewing it (.ie exampe below)

` provider:
name: faas gateway: http://localhost:8080

functions:
hello-python: lang: python handler: ./hello-python image: my_custom_registry/hello-python max_usage: 1 # renew the container after every use `

Steps to Reproduce (for bugs)

1. Deploy OpenFaaS on a Docker Swarm with multiple workers
2. Deploy a function that return the name of the container, ie below (copy of the hello-python function) :

import socket import subprocess def handle(st): print( "You said : " + st ) hostname = socket.gethostname() ip_info = subprocess.Popen("ip a | grep -A2 eth | grep inet | awk '{print $2}' | awk -F '/' '{print $1}' | tr '\n' ' '", shell=True, stdout=subprocess.PIPE).stdout.read() print ( "This query has been proccessed by container " + hostname + " (" + ip_info + ")" )

3. Call the function
4. See that the container ID / IP is always the same (docker ps will also keep returning the same container ID)

Context

What I'm trying to do is to have functions (containers) that can process critical data, and that gets totally killed once they have finish to run. It's for security and confidentiality purposes.

If this functionality already exist, then I've been unlucky with Google, because I've been searching for 3 hours and the only thing that I can find is some weird tricks to fake a failed "HEALTHCHECK" and force docker to kill the container, or using the switch --ephemeral from docker, but it's not working the way I want because I need the function/container to wait for the initial query, and only then get killed.

Example : Customizing the HEALTHCHECK command : HEALTHCHECK --interval=1s CMD [ ! -e /tmp/killme.now ] || exit 1 At the end of the function, add a command that create the file "/tmp/killme.now". The function will run once, and then it will create the file and the HEALTHCHECK will think that the container is not healthy anymore and will kill it. But if this same container was about to receive a new request, then this request will finish in an inconsistent/failed state.

Your Environment

`

• Docker version ( on the Swarm Manager ): Client: Version: 17.12.0-ce API version: 1.35 Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:11:19 2017 OS/Arch: linux/amd64

Server: Engine: Version: 17.12.0-ce API version: 1.35 (minimum version 1.12) Go version: go1.9.2 Git commit: c97c6d6 Built: Wed Dec 27 20:09:53 2017 OS/Arch: linux/amd64 Experimental: false `

• Are you using Docker Swarm or Kubernetes (FaaS-netes)? Docker Swarm

• Operating System and version (e.g. Linux, Windows, MacOS): 3 x Ubuntu 16.04 LTS (1 Swarm Manager + 2 workers)

• Link to your project or a code example to reproduce issue: See function provided above that echo the container ID

[stefanprodan]

This could be covered by scaling down to zero #238

[alexellis]

@sappounet what is the use-case for having the container die for every request? You realise this would be very slow right? What level of traffic are you expecting?

It's for security and confidentiality purposes.

We have plans to introduce an option for a completely read-only filesystem option - I've tested it as a PoC and it works well. I would have thought that would satisfy your constraint.

[Th0masL]

Happy new year guys ! Thanks for your quick answers :)

I'm not sure I've been clear, nor if I'm using OpenFaas properly or not, so I'm open to any solution around this question :)

The point of the process of spinning a new container all the time being slow is quite true, which is why by setting the minimum number of nodes to, let's say 5 ( labels: com.openfaas.scale.min: "5"), Docker will ensure that there are at least 5 fresh-new-readytouse containers ready all the time (that have not processed any data yet), and renew it as soon as it has been used.

In my opinion, the only moment it could be critical is when we have more query arriving than the number of clean-readytouse existing containers (let's say 6 queries, in this example). The 5 first queries will be answered by the 5 different containers, and the 6th one would have to be buffered for a while (couple of milliseconds/second) in order to spin up a new clean container. This could be avoided by :

• having an idea on how many simultaneous request a function get and set the right value for "scale.min"
• tuning the autoscaling so it's smart enough to provision couple of more container than the existing load
• configuring the container deployment so it's starting a new container as soon as a query arrives on the API (instead of waiting for the container to finish to process the query)
• eventually giving the right to the existing container to respond to more than 1 query if the load is too high right now (meaning that we will not kill the container instantly, but wait that we have new available container before killing the used ones)

The goal of this idea is to make the functions/container safe to deal with critical data.

Example : Let's say that we have a function that receive the customer data (credit card info), do some manipulation with it, and then return a result, I want to make sure that even if someone (hacker) breaks into a container, they will not be able to find anything (not in the container's memory, nor on the container's storage), and they will not be able to extract data coming from the next query of this container (since this container will be destroyed as soon as the first query is finished).

Thanks :)

[alexellis]

Can you say why a read only filesystem is not suitable?

[Th0masL]

I guess that read only filesystem would eventually behave pretty much the same way, but the Information Security Manager of my current company would say that as long as the container is running, we can extract data from the previous processes by accessing the RAM.

Yeah, I know, we have crazy security restrictions when working in the financial market and have to be PCI compliant / audited :)

Also, this would permit to recreate automatically any container that became internally faulty for whatever reason but is not detected as faulty by Docker/OpenFaas (if the returned errorlevel from the function is 0, for example).

[alexellis]

I'm not sure I fully understand all the aspects of PCI compliance which mean that a process cannot handle two requests independently. Surely this is what all web-servers, monoliths and microservices do today in the industry? What examples do you have of how you process credit card data in other systems?

Here are a couple of posts I found on PCI / DSS:

• https://www.schellman.com/blog/docker-pci-compliance
• https://www.twistlock.com/2017/01/11/checklist-pci-compliance-containers/

Perhaps it would make sense to set up a call with your information security manager to get some hard requirements on what is absolutely needed to tick the boxes. At that point we could tell you when and if using a serverless approach could satisfy your needs.

In terms of healthchecks and readiness checks we already have this with Swarm and Kubernetes - if you delete the lock file then the container will be re-scheduled.

[Th0masL]

The thing behind security is that we always tend to do our best with what we have (technologically speaking).

I could tell you that all our servers are fully up-to-date, but that would not be true, and I'm pretty sure that most of the people who has been using Linux for a while had, at least one time, to deal with the fact that some monoliths are soooo old and the code is soooo specific to the current versions of installed packages, that it's very difficult to keep them up-to-date with security and packages updates without breaking something.

Right now the data is processed by a Monolith that we are (slowly) splitting into micro-services, so for sure the security concerns are impacting the current system too (and even more now with a Monolith than if we were using a read-only Docker containers).

Anyway, I was just curious to investigate the serverless technology to see what it can provide, and I was thinking that making the container a one-time use was a nice idea to improve the security and making it really-really serverless.

But since you said that deleting the lock file in /tmp will re-schedule the container, then maybe I can start with that to have a working proof of concept.

I can kill the lock file at the end of the function, and it will re-schedule a new container, I will give it a try :)

[JockDaRock]

@Th0masL Just curious how your testing went with deleting the lock file?

[Th0masL]

Hey guys !

I finally had some time to test more what happen when we simply killing the lock file ( /tmp/.lock ) at the end of the function.

We have to keep in mind that that's not exactly an expected behavior (to kill the lock file on purpose), but more some kind of "test" or "quick hack" to see if we can make a container/function ephemeral.

And for a "quick hack", it seems to be working quite well, but there are couple of things that I can notice :

1) If the traffic load is too high, sometime some container can have the time to respond to more than only one request. Also, even if the auto-scaling settings specify that the minimum number of containers allowed is 5, it can go as low as 1 if the number of requests per seconds is too high.

// Example of multiple response : macbook-thomas:Vagrant $ ./simulate_traffic.sh This query has been processed by container 123d4715ed3e (10.0.0.75 172.18.0.24 ) This query has been processed by container 1738324eab86 (10.0.0.46 172.18.0.3 ) This query has been processed by container 1fd89b4bd6fc (10.0.0.26 172.18.0.14 ) This query has been processed by container 40f73fdff7f4 (10.0.0.12 172.18.0.6 ) This query has been processed by container 0cb4480f0c8e (10.0.0.76 172.18.0.15 ) This query has been processed by container 1738324eab86 (10.0.0.46 172.18.0.3 ) This query has been processed by container 0cb4480f0c8e (10.0.0.76 172.18.0.15 )

So in this example, the container 0cb4480f0c8e responded two times, and 1738324eab86 too

2) At some point, I have the feeling that some containers are not killed properly and remain in an inconsistent state (to be investigated).

Example : Below, two containers that have "lost their NAME", and that have not been killed (I used the "grep -v :" command to remove the containers that still have a valid NAME value)

root@faas-manager1:/home/ubuntu# docker ps | grep -v ":" CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 1fd89b4bd6fc 2b5c228c3684 "fwatchdog" About an hour ago Up About an hour (healthy) weather_forecast.2.cpxcjzzdcpz4o4rvi845yif5p 40f73fdff7f4 a2be8da1f485 "fwatchdog" About an hour ago Up About an hour (healthy) hello-python.4.h9jylejdq17lir7n5r395l0an

3) It also seems that I killed the faas-manager by sending too many requests (or forcing him to trigger too many new containers). The two functions that I'm using are very simple, one is just the hello-python, the second one is a "weather_forecast" function that returns the weather.

Example :

// Unable to kill the crashed container, the command docker kill is not responding root@faas-manager1:/home/ubuntu# docker kill 1fd89b4bd6fc ^C

// Cannot restart docker service either, so obliged to kill the service root@faas-worker1:/home/ubuntu# pkill -f docker

// Then I restart the service root@faas-manager1:/home/ubuntu# service docker restart

// But even after the reboot of the docker service, the openfaas containers are not coming back up root@faas-manager1:/home/ubuntu# docker ps | grep "weather|hello" | wc -l 0

// Then I decide to deploy the functions faas-cli deploy hello-python.yml faas-cli deploy weather_forecast.yml

// And only then my containers are back both on manager and on worker, with the right minimum number of containers (5 for each function) root@faas-manager1:/home/ubuntu# docker ps | grep "weather|hello" | wc -l 4 root@faas-worker1:/home/ubuntu# docker ps | grep "weather|hello" | wc -l 6

// And then I "flood" again with requests (around 3 requests per second), I managed to get approximately 20 to 30 seconds of response without any errors, and then it killed the dockers containers on both manager and worker root@faas-manager1:/home/ubuntu# docker ps | grep "weather|hello" | wc -l 0 root@faas-worker1:/home/ubuntu# docker ps | grep "weather|hello" | wc -l 0

// This time I don't have to kill the docker service, because there are no containers that are in an inconsistent state (they are basically all gone :D ) so I redeploy using the faas-cli command faas-cli deploy hello-python.yml faas-cli deploy weather_forecast.yml

And the containers are back.

So ... to sum up : Deleting the lock file to make a function ephemeral can work, but we're getting a complicated/random behavior when the traffic is getting too high on the function. I guess this could be handled by allocating more resources to the docker hosts, and also increasing the number of minimum containers that we want per function for the auto-scaling, but at the same time I still think that the behavior is too random to be able to use that in a Production environment :)

[alexellis]

You should not delete the lock file. This is a health-check mechanism for recovering from a bad container / application crash.

One container per request is not a supported use-case but as I've said I think a root r/o filesystem will satisfy the constraints of your security officer. If not then perhaps we should have a call together rather than going by conjecture.

[abhishiv]

Hey @alexellis, another use case would be running a multi-tennant paas. Very much like heroku or aws lambda. However in that case the router also needs to think about a grace period instead of just one container per request.

[hveiras]

@abhishiv I think that if you are looking for a solution to that scenario at the same time you want to self-host your solution then Apache Openwhisk is the best option out there today.

[alexellis]

Derek close: inactivity

[jmolivas]

@Th0masL did you found a solution for this?

@hveiras can you elaborate more on what you mentioned?

Apache Openwhisk is the best option out there today.

@alexellis are you willing to follow the discussion somewhere either here or slack?

[rabbah]

In reference to openwhisk - what @hveiras might have meant is that when a function exits, its container is expunged and the next execution uses a fresh container automatically.

[alexellis]

Hi @jmolivas

Please can you explain how your existing micro services or monolithic applications operate? Are you telling me you restart Flask, Ruby on Rails, Java, ASP.NET or whatever it is after every request? If so that sounds incredibly inefficient.

Folks saying "a scenario could be" doesn't sound like this is a hard or actual requirement at present. Can you start with your specific requirements and then maybe we can work backwards from there?

Alex

[VizorKit]

@alexellis I think i know whats going on and why there are two different camps. If I can get a guarantee that my "function" is destroyed after every use, I can give any of my clients, customers, myself a reasonable guarantee that I wont have any significant memory leaks, I can't get into any weird state scenario in my app.

I remember when I originally was told about functions a year or two ago. I was ecstatic to eventually get to work with them. And I am just now starting to dig into them, and I'll look into openwhisk.

A container is quite simple really. Here is a really good talk on what a container is, she even creates one written in go,container.

It's a fully isolated environment, basically.

Your point could be valid about inefficiency. But, how fast can we create a completely isolated environment? Probably in the grand scheme of things it's not that bad.

Startup times of your container are the enemy. I would be willing to bet, at scale, a function written in c, or anything without a runtime that is destroyed on every request can easily out scale a .net, java, and several other framework based applications.

Next up is code execution. The actual task your function requires takes time.

So! We have three time buckets when dealing with functions. And then point 4 and 5.

1. Containerization Time.
   • how long it takes to make and destroy isolated environments.
2. StartUp Time.
   • how much program needs to be loaded into .text, .bss, .data, the stack, the heap.
   • how many instructions needs to be executed to get a framework/runtime working.
3. Task Time.
   • how long the functions intended task takes.
4. Human Factor.
   • how paranoid the developers are about state.
   • how paranoid the developers are about security.
5. Cost Factor
   • how well do you need to scale.
   • how cheap do you need to run at downtime.
   • how well can you predict peak usage.

So although I can agree that the point about efficiency is correct. It doesn't matter under these scenarios.

1 and 2 are basically nothing. 3 is really long. anyone happens to fall into camp 4. any of 5 really. I'm pretty sure this is a known problem you deal with every day helping people properly scale, setting max replicas, min replicas, etc.

[alexellis]

You'll find that OpenWhisk is not ephemeral, it uses "stem cells" to side-load code AFAIK.

You would be best using a cron job automated from the Kubernetes API if you really do have a hard requirement for "one-shot" processing. This will be extremely slow, rate-limited, and limited by concurrency.

Alternatively, you could automate the OpenFaaS API to deploy one function per request and remove it after each invocation, or keep one function per customer. This would be much faster and more efficient, and would allow you to use async invocations and other features of the project.

Btw. I don't think you need to explain to me what a container is? I wasn't sure how to interpret that 🤔

[rabbah]

openwhisk stem cells are an optimization - invisible to the user. They allow the control plane to allocate new resources ahead of time and avoid all container startup overheads which can be significant especially under high load.

openwhisk keeps a container around for the same function from the same user for a programmable (at deployment of the system) grace period - for example the default is 10 minutes; this allows for container reuse of the same function (alongside automatic scaling which will allocate more containers for a given function on demand/as needed).

The function container resources are automatically reclaimed after the grace period. During the grace period the container is paused until it’s needed again.

So the containers/resources in openwhisk are ephemeral in that they’re provisioned on demand and destroyed shortly after use.

It is possible to configure the system for a use once model - and compensating for potential startup overheads by provisioning the deployment with more stems cells.

I hope this helps clear up how openwhisk works. We’ve found stem cells to be incredibly important in production for short running functions where you can’t afford more than a few Milliseconds of slack in the control plane.