Suggestion: Export deployed functions

maiermic commented 6 years ago

I'd like to be able to save all deployed functions and (re-) deploy them to another OpenFaas platform/gateway.

Expected Behaviour

save deployed functions
stop and remove OpenFaas platform (e.g. Docker stack) from server (all function services should be stopped and removed)
deploy OpenFaas platform on another server
deploy all saved functions

Current Behaviour

You can not save functions that have been deployed using the API in a format that can be redeployed. Functions that are defined in a docker-compose.yml file can be redeployed, but some of them might have been deleted using the API.
If you stop and remove the Docker stack of the OpenFaas platform then only functions are stopped and removed that are defined in a docker-compose.yml file. The services of functions that have been deployed using the API still run and might prevent deleting the network of the OpenFaas platform.
deploy OpenFaas platform on another server works using the same docker-compose.yml as the one of the old server
Functions that are defined in a docker-compose.yml file can be redeployed automatically. Functions that have been deployed using the API have to be (saved and) deployed manually.

Context

I'd like to be able to move the OpenFaas platform to another server with all deployed functions. The functions are deployed using the API or using a docker-compose.yml file.

Your Environment

Docker version docker version (e.g. Docker 17.0.05 ):


Client:
Version:      18.03.1-ce
API version:  1.37
Go version:   go1.9.5
Git commit:   9ee9f40
Built:        Thu Apr 26 07:17:20 2018
OS/Arch:      linux/amd64
Experimental: false
Orchestrator: swarm

Server: Engine: Version: 18.03.1-ce API version: 1.37 (minimum version 1.12) Go version: go1.9.5 Git commit: 9ee9f40 Built: Thu Apr 26 07:15:30 2018 OS/Arch: linux/amd64 Experimental: false


* Docker Swarm

* Linux

alexellis commented 6 years ago

Hi Michael,

I strongly recommend against using docker-compose.yml for your functions.

When working with a git-oriented workflow you should always be deploying functions from a CI server using the stack.yml file that makes up your functions.

I'd suggest keeping a Git repo with the functions you want so that you can regenerate this in a declarative way.

If you are working your way towards the above but are not quite there yet then you could write a query or some custom code to query which functions are deployed and then use that to mirror that into another environment. This is not a recommended approach because it's stateful and imperative - I recommend you approach this problem declaratively.

In your specific example how many functions do you have deployed?

Regards,

Alex

maiermic commented 6 years ago

It is a topic I will be confronted with in the future. I've just set up a test environment/server yet. I will have to move this test environment to a production server sometime. I don't like to move all functions manually if it can be avoided even though it might not be that many (maybe 5 or 10).

The functions are defined in different repositories and should be deployed from a CI server using the stack.yml file of each repository. There is no single Git repository with a stack.yml that contains all deployed functions. It would have to be kept synchronous with every deployment from the CI server.

Do you suggest to redeploy each function (manually) using the stack.yml of each repository?

tarunmangukiya commented 6 years ago

I think migration is one time task and most of the migrations do require human actions. Plus even with current OpenFaaS & faas-cli, I am able to re-deploy all functions anywhere within 5 minutes (this includes creating new Docker VM).

We are having around 26 functions for different processing tasks.
Also, I don't use OpenFaaS UI to delete functions, so all the required functions are in stack.yml only. And other functions are commented out.
I suggest you to have same repo for functions as all of them are going to be deployed at the same location. You can better manage multiple docker registry if you have multiple devs who are writing functions.

I think you can deploy functions from multiple stack.yml files to the same gateway using CI. @alexellis can we?

Let me know on Slack what do you think about this.

LucasRoesler commented 6 years ago

This is just an idea about how to build a migration or backup based on the current feature set: functions already supports labels and the concept of annotations is coming soon. I wonder if adding the project name and git repo to the annotations could be helpful here. You could ask OpenFaaS for the functions and generate the list of projects that are deployed. It would then be easy to checkout and redeploy those to another cluster. This would of course miss any functions that were deployed via other means (e.g. the store).

alexellis commented 6 years ago

Is there any data-loss between creating functions via a stack.yml file and querying them back via the /system/functions endpoint?

If not, then we could add some kind of CLI command faas-cli export --gateway https://... > stack.yml

Alex

alexellis commented 6 years ago

@rgee0 @ivanayov any thoughts?

rgee0 commented 6 years ago

Not sure environment variables are stored in /system/functions nor /system/function/fn

LucasRoesler commented 6 years ago

According to our documentation, https://github.com/openfaas/faas/blob/7a2a3542da5d5f67af001e6cd9948672945a0495/api-docs/swagger.yml#L409-L442 it looks like a env variables, secrets, limits, requests, and constraints are all missing

We would need the api to match this struct, right? https://github.com/openfaas/faas-cli/blob/85e2bd1d2b1cd8682d45a156f68ab3e518d5392c/stack/schema.go#L14-L59

alexellis commented 6 years ago

What’s in the code in faas-swarm? Swagger may lack some details but is updated on a best effort basis.

LucasRoesler commented 6 years ago

From faas-swarm https://github.com/openfaas/faas-swarm/blob/be442da69feaeff3e19be1ecbc18c1c3b2ef1ff5/handlers/reader.go#L39 it returns a requests.Function

https://github.com/openfaas/faas/blob/7a2a3542da5d5f67af001e6cd9948672945a0495/gateway/requests/requests.go#L61-L78

alexellis commented 6 years ago

This work is blocked by and related to: #849

An export would be useful for any operators someone wants to write for use with OpenFaaS and for backup scripts as suggested by @maiermic

Alex

alexellis commented 3 years ago

/lock: inactivity

openfaas / faas