search

dwyl / learn-devops

🚧 Learn the craft of "DevOps" (Developer Operations) to Deploy your App and Monitor it so it stays "Up"!
GNU General Public License v2.0
418 stars 172 forks source link

EPIC: Continuous Deployment of Phoenix Apps to Any VPS #59

Open nelsonic opened 4 years ago

nelsonic commented 4 years ago

At present we don't have a reliable way of doing Continuous Deployment to Linode for our Hits App. https://github.com/dwyl/hits ... Because it's been infrequent, I have done the deploy manually. 🤦

We have setup CI/CD for Client Phoenix Apps in the past e.g: https://github.com/dwyl/learn-microsoft-azure#how But so far we have stuck to using Heroku for @dwyl Apps because it's so easy and cheap (Free!)

Now we need to do it with a fresh pair of eyes and with the benefit of all the enhancements that have occurred in the Phoenix/Elixir/Erlang community over the past couple of years.

Despite running Cowboy (Erlang) under the hood, Heroku still places limits on concurrency and connections to preserve service quality for all users. Also, when you factor in a PostgreSQL database (min $9/month), Heroku gets quite expensive for "hobby" or "side project" apps like Hits. And given that Hits has way more than 10M rows (the limit for the $9/month DB) at this point, we would probably need to pay the $200/month for the "standard" Postgres instance ... 💸 see: https://elements.heroku.com/addons/heroku-postgresql

Todo

Related:

@SimonLab given that you prefer to focus on "back end" part of your "full stack" skills, I feel like your name is missing from this list: https://github.com/dwyl/learn-devops/graphs/contributors 😉 I've added you to the Linode account using your gmail address. Please add 2FA to enhance security: https://www.linode.com/docs/security/authentication/two-factor-authentication/linode-manager-security-controls And feel free to play with a Linode VPS on my account.

P.S: I'm not married to Linode by any means. I just prefer to use an independent service that is not VC funded or pouring money into the pocket of the world's richest person 🙄 The reason the title of this issue is "Any VPS" is precisely because we want a generic solution to deploying our Phoenix Apps.

nelsonic commented 4 years ago

@SimonLab please give a ballpark estimate for how long you think this will take. ⏳ If the issue description does not have enough detail, please let me know what is unclear. 💭 Thanks! ☀️

SimonLab commented 4 years ago
SimonLab commented 4 years ago
SimonLab commented 4 years ago

My goal is to see how easy it is to run a small Phoenix application with Linode. I'll try to install the following app: https://github.com/simonlab/phx . It doesn't contain Ecto so I don't have to worry with setting up Postgres at the moment (see https://github.com/dwyl/learn-devops/issues/58 Linode provide the Postgres as application which means that it will create a Debian which will host the application. However this means having two linodes, one for the app and one for Postgres, ie paying twice).

The steps I'm going to try is to

So far I've:

got the following error:

15:17:52.358 [error] Could not find static manifest at "/root/phx_release/lib/phx-0.1.0/priv/static/cache_manifest.json". Run "mix phx.digest" after building your static files or remove the configuration from "config/prod.exs
SimonLab commented 4 years ago

I've installed posstgres on the Ubuntu server and at the moment running mix phx.server (without creating a realease): image

nelsonic commented 4 years ago

@SimonLab have you had a chance to read Chapter 11 "Deploy Your Application to Production" of the Real-Time Phoenix book?

image

I think it might be highly relevant to this Epic/Quest as we 100% want to have WebSockets/Channels in our App (that's the biggest selling point of using Elixir/Phoenix! 😉 )

SimonLab commented 4 years ago

I just rapidly read the deployment chapter:

A production server can be run with mix phx.server or by using releases.

Releases package BEAM (Erlang virtual machine), provide scripts (for example to monitor the running application), allow BEAM customisation with flags and preload some code to make the initial responses to client requests faster.

You can decide to deploy the application on a "Platform as a Service" PaaS (e.g. Heroku, Gigalixir, Render) or on a virtual private server VPS (e.g. Linode). Make sure the server/platform can handle a large amount of concurrent connection and can use websocket.

To manage the requests load you can use a load balancer which manage where the requests are sent to (e.g. HAProxy, nginx)

Redeploying an application can be done with rolling deployment where servers are restarted after each others. The blue-green deployment switch between an old cluster (running current applications) to a new cluster (running the new applications). When deploying the live connection will be closed however the load balancer should manage the reconnection to the new servers.

nelsonic commented 4 years ago

@SimonLab thanks for sharing this good summary of your reading. 👍 As discussed on our Standup call on Monday, please add a lot more detail for everything you have researched and links you have followed in your quest to do Continuous Deployment (CD) on a VPS.

Ideally we would like to have a "how to guide" that gives the exact steps to setup CD for Elixir/Phoenix Apps (as per the issue description). And a set of (version controlled) scripts for actually executing the DevOps. Have you started creating a markdown file with your notes of what you have learned? (please push to GitHub)

OpenBSD is a (really) "nice to have" at this point, definitely not a "requirement" right now. Given that we cannot use backups on Linode with OpenBSD, I'd say we should "park" the idea of using BSD on Linode (which is targeted at Linux, the clue is in the name). The goal with considering OpenBSD is "high security" as described in https://github.com/dwyl/learn-security/issues/73 🔐 (please leave a comment on that issue with your experience so far of using BSD... 🙏)

Just the fact that there are considerably fewer people using OpenBSD than Linux means that there are fewer people who understand how to attack/hack it. From a security perspective that's a major plus. It's something Mac had a few years ago that Windows never did. (fewer users means hackers don't bother trying!) Now that Mac is 9% of the desktop market and Hackers know that higher value users are on Mac (e.g. Developers who need to build iOS Apps and pretentious rich brats who buy $2k laptops to do their word processing and web surfing! 🙄 ), Macs are increasingly targeted with malware.

To clarify: I would much rather figure out CD fast using Ubuntu (which will have many more answers on StackExchange, tutorials, blog posts, etc.) than spend days learning BSD so we can deploy the "Hits" (Phoenix) App ASAP.

Linux is "fine" for deploying "Hits" to Linode with backups enabled. The only data that can be considered PII that we store is IP Address and we can easily add Fields.IpAddressEncrypted to encrypt the data at rest thus minimising the effect of a potential breach.

For the purposes of deploying the DWYL APP, we need to make a time-value tradeoff assessment and determine how much more effort is needed to figure out how to get Zero-downtime CD on Linode. The DWYL APP is our "crown jewel" and we cannot afford to have a "hacky" deployment pipeline. We need something with an SLA and at least "five nines" of uptime. If the amount of effort required to set this up now is another 5 days of your time, that's enough cash to pay for Heroku or Gigalixir for a Year. By which point we will either have a few thousand paying customers or have run out of funds to continue working on the App ... 💸

Notes

We still need to get better at Estimating Tasks before starting the work. https://github.com/dwyl/learn-devops/issues/59#issuecomment-632678545 ⬆️ We need to get better at holding ourselves accountable to those estimates to avoid spending time on activities that are not building "features" people using the App want/need. I hope that our App will help with this. 🤞

To be clear: I consider data security to be a "feature" but I don't know how many customers will use our product because of the security. It's more of a "hygiene" factor than a "motivator".

As much as I want to have our own Continuous Delivery with Zero-downtime Deploys to a VPS, I feel that investing more time into this quest is not wise right now. The reason I asked for an estimate above was to help with prioritising the task.

Deploying our App to an existing (PaaS) provider is "enough" security during our MVP. Once we have 1000 paying customers we can re-visit deploying to a VPS with BSD. 👍

SimonLab commented 4 years ago

Thanks for your comment above @nelsonic I've just created a PR (https://github.com/dwyl/learn-devops/pull/61) where I try to recap what I've learn so far while searching how to deploy Elixir/Phoenix. I will need to add more detailed steps later on

I feel confident that for a simple Phoenix application a VPS could be a nice solution, however I still have some aspect that I will need to research (backup, deployment without any downtime, elixir cluster) which can take a at least a few days of reading/testing. With that in mind I also think at the moment a PaaS is a good solution while the application is getting build and user tested.

As much as I want to have our own Continuous Delivery with Zero-downtime Deploys to a VPS, I feel that investing more time into this quest is not wise right now. :+1:

SimonLab commented 4 years ago

I'm currently reviewing the changes that need to be made on #61 As mention in the PR the idea is to use Travis and mix release to test and build the application, once the build is done to store it on S3 to keep an history of the different build versions then to deploy the build on Linode.

I'm reading/searching the following

travis-supported-providers

Other questions I want to also clarify:

nelsonic commented 4 years ago

@SimonLab we can safely skip the S3 step for the Hits Application. Just deploy it directly from Travis-CI to Linode. 🚀

I had to create specific deployment keys to add to Travis. Obvs documented it: https://github.com/dwyl/learn-travis/blob/master/encrypted-ssh-keys-deployment.md If anything unclear, please open an issue on learn-travis/issues and link to it here.

SimonLab commented 4 years ago

Testing the script step on a simple Phoenix application, where the .travis.yml file is:

language: elixir
elixir:
  - 1.10.3
otp_release:
  - 22.1.8
env:
  - MIX_ENV=test
script:
  - mix test
cache:
  directories:
  - _build
  - deps

deploy:
  provider: script
  script: bash deploy.sh

And a dummy deploy.sh file:

#!/bin/bash
echo "#########################"
echo "DEPLOYMENT SCRIPT RUNNING"
echo "#########################"
exit 0

~The script is not run as:~ image

expanding information of the deployment section, we can see the script is run: image

SimonLab commented 4 years ago

Using Distillery and Edeliver we can use hot code upgrade (updating the code to a new version while the server is still running). However mix release doesn't provide hot code upgrade, from https://hexdocs.pm/mix/Mix.Tasks.Release.html#module-hot-code-upgrades hot-code-upgrade-release

The following thread is the current problem I'm trying to solve: https://elixirforum.com/t/graceful-restart-of-an-elixir-v1-9-release/24211/5

The idea is to have the current and new application running at the same time and to use a load balancer to transfer the requests to the new application. There are two methods, the rolling and blue-green deployments which help to switch to the newest application version:

nelsonic commented 4 years ago

@SimonLab for the Hits App a few seconds of downtime once a month when there is a new release is perfectly acceptable. Please don’t worry about that. As long as we have a deployment we are fine. We can revisit the “zero downtime” deploys if/when we use the script for our “real” app (later).

nelsonic commented 4 years ago

Continue:

th0mas commented 4 years ago

http://dokku.viewdocs.io/dokku/ is an excellent open-source Heroku-style PaaS. - It uses Heroku build scripts under the hood so porting apps from Heroku to Dokku should be reasonably painless.

You host it on your own VPS so it is a bit more involved than Heroku but you have control over your own infrastructure and allows for a transition to container-based solution in the future

For running small applications with CD pipelines its probably one of the best options, only needing a git push in .travis.yml to deploy your application.

nelsonic commented 4 years ago

@th0mas yeah, we used Dokku a couple of years ago to deploy Node.js Apps: See: https://github.com/dwyl/learn-devops/blob/master/nodejs-digital-ocean-centos-dokku.md Agree 100% that it's very good in most usecases. 👍

The reason we decided not to use Dokku at the time for our Phoenix Apps is because WebSockets are not supported: https://github.com/dokku/dokku/issues/3480 😞 And WebSockets (Phoenix Channels) is one of the biggest reasons we addopted Elixir https://github.com/dwyl/learn-elixir/issues/102 If WebSocket support is available now in 2020, happy to reconsider. 💡