CI/CD pipeline - DigitalOcean, Terraform, DockerHub, Actions

[Romes8]

CD pipeline

Resources:

• https://tecadmin.net/creating-a-digitalocean-droplet-with-terraform/
• https://developer.hashicorp.com/terraform/tutorials/applications/digitalocean-provider\

Workflow:

1. Create a docker file that will build our application and deploy it to image repository (Docker hub)
2. Terraform Config

Terraform config: This section creates droplet in Digital Ocean

• set up provider - digital ocean
• pass token
• create a droplet with its specs and appropriate name

This section creates an MiniTwit app in the droplet. Needs to be SSHd into

• we need to create SSH connection to the droplet
• pull docker file from Docker Hub - need a log in into a docker hub
• output the IP address

We need to create 2 droplets. One for main app (one with UI) and one for API service

3. Trigger a GitHub Action for testing PR is created for main.
4. Trigger a GitHub Action when there is a push in main(needs to be push and not a PR) This way merge will happen after code review and tests were successfully passed.

Action steps:

• run tests (either locally with “go test” or with docker compose)
• if test pass continue else fail the action and give output
• log into the docker hub (if needed) and build docker image
• initialize terraform
• apply the terraform config

Extension and clarification for this task - #82 Related issue: #83

[danielgron]

This sounds very close to what I would suggest. I propose we have tests run before review so we can fail fast and not have to wait until someone spends time to review the code. We also debated Friday if it was better to not rebuild the droplet between each release, to prevent downtime, but if we are able to run multiple servers and load balance I am not opposed to this suggestion.

Note that we don't actually need two droplets to deploy both applications, but no harm in doing it.

[Hopeberg]

Made a PR for the continous-deployment.yml action file from the exercise. Maybe we could work from this file and configure it the way we want it to be. to be honest, I don't feel like I am learning the things that I want to regarding docker setup and github actions, so would be nice that we build it from scratch this week so everyone has an idea of how it works. The exercises form a good basis to that, so I suggest we work from there.

[Hopeberg]

Looks good the way the action works :)

[Romes8]

@danielgron , I have updated the frow in the text. We run test also when PR is created and also when we merge with main. When it comes to keeping the droplets - sure we can do it but than action needs to check weather there are droplets created and than SSH into them. Also with adding load balancer it can work but we are adding complexity. As for now I am not sure how the connection should be but we can work on it if we agree.

[rasmus-bn]

While I am generally fan of not having persistent server, I think it could be simple enough to do changeover like this:

1. SSH into the machine.
2. Pull the latest docker image
3. Kill the running docker container
4. Start the container using the new image

I imagine this being done from the GH action, assuming that we have the private key stored as a GH secret. It could maybe even be a bash script that always lives on the server. Then we only need to SSH into the server and execute that script :)

It would propably be better to have a load balancer and kill/spin up servers behind, but as you mentioned @Romes8 that will also increase complexity. Maybe that could be next step though :)

[danielgron]

I imagine this being done from the GH action, assuming that we have the private key stored as a GH secret. It could maybe even be a bash script that always lives on the server. Then we only need to SSH into the server and execute that script :)

I think it will be more transparent to run the scripts from the gh actions. Otherwise we have manual steps in case we want to update the script

[rasmus-bn]

I imagine this being done from the GH action, assuming that we have the private key stored as a GH secret. It could maybe even be a bash script that always lives on the server. Then we only need to SSH into the server and execute that script :)

I think it will be more transparent to run the scripts from the gh actions. Otherwise we have manual steps in case we want to update the script

Good point!

[rasmus-bn]

Just a heads up regarding terraform. When you deploy resources with terraform it creates a state file. We need to find a way to manage this state file as it needs persist between deployments. I havent looked into it but propably there's some way to store it in either digital ocean or in GitHub. https://developer.hashicorp.com/terraform/language/state

[Romes8]

While I am generally fan of not having persistent server, I think it could be simple enough to do changeover like this:

1. SSH into the machine.
2. Pull the latest docker image
3. Kill the running docker container
4. Start the container using the new image

I imagine this being done from the GH action, assuming that we have the private key stored as a GH secret. It could maybe even be a bash script that always lives on the server. Then we only need to SSH into the server and execute that script :)

It would propably be better to have a load balancer and kill/spin up servers behind, but as you mentioned @Romes8 that will also increase complexity. Maybe that could be next step though :)

Sure, we can try to build it like this as well. However we just need to check weather the droplet is existing and if yes than SSH to it and if it does not exist we need to create first and than SSH

[danielgron]

Should the the check before review be created as separate issue? I assume it will need a separate definition containing only a build/test step triggered by pr for main?

[Romes8]

Should the the check before review be created as separate issue? I assume it will need a separate definition containing only a build/test step triggered by pr for main?

Yes, this can be separate issue for sure.

[janlishak]

I think the proposed pipline seems doable, I need to look more in depth into how github actions, docker hub, terraform.. work together. #82 I'll do some research today and get back to this issue later when I know more.

[Romes8]

Status update

DONE:

• Pipeline builds docker images and submits them on DockerHub
• initializes terraform
• applies terraform config file
• Droplets get created in Digital Ocean

TODO:

• SSH into droplets and start the apps. Both in API app and main APP
• Make fixed IP address for both of the apps or have a domain
• fix #106 so that we can have stable apps