save github builds and deploy them to server

[cypressf]

The problem

Deploys take a long time, and are complex, because we always build from scratch on svante.

The solution

I'm already building the frontend and the backend in github actions on merge to main. Zip those builds and save them as releases in github. Download and unzip a build on svante to deploy.

Note: we'll need to also bundle the sql migrations with the backend build, so we can run sql migrations on svante.

[cypressf]

I can use the upload-artifact github action to upload the build outputs for frontend and backend.

[cypressf]

It successfully uploaded frontend, backend, and backend sql migrations: https://github.com/cypressf/climate-risk-map/actions/runs/1694996032

you can see the artifacts at the bottom of that page. frontend contains minified html + css. backend contains a binary webservice, and the sqlx migrations files.

[cypressf]

github artifacts don't last forever, and I'm unsure if there's a convention for their url, so it might be best to scp the artifacts to svante after building them. I could use something like this scp-files-action when anything is pushed or merged to main.

After that, I could ssh into svante and run a command to restart the pod, pointing it to the latest artifact

[cypressf]

@mjbludwig what do you think of automatically uploading artifacts from github to svante via scp when I merge to main? I can store an ssh key for svante3 and svante4 in github secrets, then automatically copy the artifacts to svante3 whenever I PR a branch, and copy artifacts to svante4 when I merge something to main.

[cypressf]

after that, we need a way to swap out the old backend with the new one, and the old frontend with the new one. I was thinking each artifact could get put in a folder named after its git hash, then when we restart the container on svante, we could give it a folder name to launch from, and it runs the backend executable, and serves the frontend html from that folder.

[cypressf]

We might be able to store artifacts as github packages, for example, in their container registry as an OCI container. https://docs.github.com/en/packages/learn-github-packages/introduction-to-github-packages

[mjbludwig]

Very interesting! I will have to read up a bit more but this would serve as a nice CI/CD system

I think first on the docket should be moving the build system to github so the containers no longer need to be rebuilt each time.

[cypressf]

great! it's already moved to github (it builds every time I merge to the main branch)

[cypressf]

@mjbludwig if you want to change the scripts, here's how the artifacts look: https://github.com/cypressf/climate-risk-map/actions/runs/1694996032

frontend has all the html and other static assets backend contains a migrations folder with all the sql files that sqlx runs on. it also contains an executable at target/release/climate_risk_map.

How's this sound?

frontend container

• run apache to serve the static frontend assets

backend container

• run sqlx to do the latest migrations, then
• run the backend executable

[mjbludwig]

Yeah that sounds so much cleaner than the current setup. I will work on reformatting the podman bits/scripts for this new workflow.

[cypressf]

I'm unsure how this would work when I upload a new backend. Would I be able to execute a command to stop running the previous backend, run the database migrations on the new one, and run the new backend?

[cypressf]

I'm going to try to scp the artifact to svante3 when a PR is merged.

I can make a deploy action that only occurs on push to main via this conditional: https://github.community/t/how-do-i-specify-job-dependency-running-in-another-workflow/16482/2

[cypressf]

I'm having trouble scp-ing the files. I keep getting Permission denied even though I believe I'm storing the correct secret in this repo. @mjbludwig could you help me troubleshoot?

[mjbludwig]

Can you meet at 3?

[cypressf]

Yeah! See you then

[cypressf]

@mjbludwig and I figured out what was going wrong with the scp action. I will push a fixed version of it. I'll scp to svante3 when a PR is submitted, and scp to svante4 when it's merged to main.

The copied file path will be ~/builds/{git hash} and it will contain the frontend and backend artifacts for that build.

Morgan will change the containers:

• [ ] remove the frontend container, instead symlink the pre-built frontend directory to svante's apache directory
• [ ] change the backend container to not build, but instead only run sqlx on the migrations, and start the pre-built service binary

At first, to deploy, I will manually change the symlink to the latest frontend and backend, and restart the backend container. Next, I will add that step to the github actions, so that it will auto-redeploy to our production server when changes merge to main, and auto-deploy to our development server when changes are PR'd.

[cypressf]

I've started working on copying over builds to server. The last attempt hung on the scp step for some reason (https://github.com/cypressf/climate-risk-map/runs/4903079080?check_suite_focus=true) and I had to cancel it because it's the end of the workday and I didn't want to get charged for github overages. This happened after https://github.com/cypressf/climate-risk-map/commit/350ce7ad44f8f589f17bd2c56ecf8363a702cb73. I noticed the backend artifact (https://github.com/cypressf/climate-risk-map/actions/runs/1731136951) somehow inflated to 2.44 GB, so I'm doing something wrong....

[cypressf]

Okay, going back to working on this, I'm going to debug exactly what it's including in the tar to make it so big.

[cypressf]

tar -cvf backend.tar backend on my laptop creates a 4.91GB tar file. This is because it's including all the dependencies and extra code in backend/. I only want to include backend/target/release/climate_risk_map and backend/migrations.

[cypressf]

tar -cvf backend.tar backend/target/release/climate_risk_map backend/migrations results in a 31.5MB file. That's what I want!

[cypressf]

Okay, now I need to extract the tar before scp to the server.

[mjbludwig]

I think once you get that setup I can put sym links in place and we can try migrating to the new container system.

[cypressf]

Sweet. I'm about to push a commit that sets that up. I'll @ you when I'm done.

[cypressf]

@mjbludwig

The files are copied to svante3 and svante4 in the following structure:

~/builds/{git hash}/frontend/
~/builds/{git hash}/backend/climate_risk_map
~/builds/{git hash}/backend/migrations/

for the static frontend directory, the executable service, and the migrations directory

[cypressf]

(I didn't include the date in the directory name because I can just use the last modified times if I want to find the latest one)