search

access-company / antikythera

Elixir framework to build your own in-house PaaS
Apache License 2.0
218 stars 19 forks source link

Lacking guide to production environment deployment #73

Open fire opened 5 years ago

fire commented 5 years ago

https://hexdocs.pm/antikythera/development_environment.html

Development guides for antikythera exist but there lacks a guide for production deployments.

What cicd is needed to deploy?

How does the antikythera code get merged with the gears code in a docker image?

How does ACCESS deploy: VMs? Docker? Kubernetes?

What environmental variables and other settings are required?

aMasakiTakahashi commented 5 years ago

Sorry, we don't have guides for production deployments. We will add the necessary documents in the future.

For your reference, we(ACCESS company) briefly introduce how we construct Antikythera cluster for production. We have built Antikythera on AWS. (we don't use docker and so on, so currently build each Erlang node on EC2(Amazon Elastic Compute Cloud) directly).

Each Erlang VM nodes runs on EC2, Furthermore, we use an EC2 node for Jenkins server and use EFS(Amazon Elastic File System). Jenkins server checks gear code repositories periodically. If it finds a new version of gear, it compiles gear and stores the compiled object on EFS. Each Erlang VM node monitors EFS periodically. (This is implemented by VersionSynchronizer If a node finds the new compiled object of gear, It loads new version of gear with hot code upgrading.

Upgrading mechanism of Antikythera core is also the same as gears.

[Note] Antikythera has been designed not to depend on the specific cloud provider. In other words, modules depending on some infrastructure are completely divided from Antikythera core codes. We offer these modules as pluggable modules of Antikythera. AWS Example is here.

What environmental variables and other settings are required?

You have to setup build server(ex, Jenkins) to compile gear and file server(ex, EFS) to store compile gear object.

fire commented 5 years ago

You can close this when documentation is written.

fire commented 5 years ago

Are you using EFS or S3?

EFS is scsi based filesystem emulation that has infinite space.

S3 is an object store.

EFS is proprietary, while S3 has multiple independent implementations.

My goal is to run this system on any other cloud, like GKE or Digital Ocean's Kubernetes.

Both has S3 emulations, but EFS is secret technology.

fire commented 5 years ago

A common problem converting vms to docker / kubernetes is that secrets need to be moved into environment variables, so everything configuration wise that is written to the executable needs to be changed.

For example a previous example would have javascript that encodes the endpoint of the elixir server. When the server endpoint is changed, it causes the javascript to fail.

aMasakiTakahashi commented 5 years ago

Are you using EFS or S3?

We use EFS, but Antikythera doesn't depend on EFS. Antikythera requires only a file server. Each ErlangVM node should be mounted on a file server. But It is OK that this file server is not EFS.

A common problem converting vms to docker / kubernetes is that secrets need to be moved into environment variables, so everything configuration wise that is written to the executable needs to be changed. For example a previous example would have javascript that encodes the endpoint of the elixir server. When the server endpoint is changed, it causes the javascript to fail.

We use Jenkins to build Antikythera and embed some secret info at the same time. After that, we move compiled Antikythera to linux server and run it. So, I think you can control settings of Antikythera with environmental variables.

fire commented 5 years ago

It is difficult to have a filesystem store, it is much easier to use s3 which is an object store.

In Kubernetes I can give each ErlangVM a disk but they are separate from each other and small. There can be a way to reclaim disk space, but it is not access to a file system that has near unlimited size.

But It is OK that this file server is not EFS.

What do you mean?

aMasakiTakahashi commented 5 years ago

In Kubernetes I can give each ErlangVM a disk but they are separate from each other and small. There can be a way to reclaim disk space, but it is not access to a file system that has near unlimited size.

Currently, Antikythera need file system to share some files with each node. I think that it is also good to use S3 instead of a file server. But our priority to support S3 for that purpose is low.

But It is OK that this file server is not EFS. What do you mean?

It means that Antikythera doesn't depend on AWS service.