Continuous Deployment with GitLab #280

Closed gabor-boros closed 2 years ago

Proposal

In this example project I’ll show how to integrate a simple Python microservice with GitLab’s CI/CD solution to deploy both preprod and production services to Fly. This project could be a base of an article as well

The CI/CD workflow

Installing dependencies
Running unit tests
Creating a docker image
Uploading the image to registry
Deploying the microservice to preprod
Running integration tests against preprod
Deploying the microservice to production

About the microservice

This is an open-ended question what would be the microservice. I think a currency converter with fixed currencies and rates could be great for demonstration purposes, since the interesting part is the workflow itself.

References

This project could be combined with Kong API Gateway on Fly by the users easily to get a better experience

Yes this would be awesome. For good CI examples, we want to make sure deploys use local Docker builds, if it says "Docker not available, doing remote build" that's not great in CI.

For now, any app will work. You don't need to build a microservice, though. Demonstrating CI with our "Hello Fly" app will work just fine if you don't have anything else convenient: https://fly.io/docs/hands-on/create-app/

@mrkurt Sure thing.

@mrkurt Demonstrating the whole workflow (step 2) is not possible with the Hello Fly image. Isn't it an issue for you?

Ah that's a pretty good point. Demonstrating tests is a good thing. I mainly just didn't want you spend a lot of time on an example project just for this! But that does make sense, I'm cool with either a tiny service that has the tests + setup or using one of our existing examples and adding some test, whatever's simplest.

Then I'll go with a minimalist service which is not a big effort, which can demonstrate the whole workflow 😊 😇

@gabor-boros we have a standard app to use for these kinds of examples now (and it includes tests!): https://github.com/fly-examples/flygreeting

Hello @mrkurt, Thanks for letting me know! I'm almost done with the whole thing. Should I go with your app instead? (I went with a simple hello world like api service in python tho).

Yeah use ours instead.

Right,I think it will be done tomorrow.

@mrkurt Is that contains integration tests too?

Yep, if you clone that project and run go test you'll see their output:

➜  flygreeting git:(main) go test
[GIN-debug] [WARNING] Running in "debug" mode. Switch to "release" mode in production.
 - using env:   export GIN_MODE=release
 - using code:  gin.SetMode(gin.ReleaseMode)

Wed, 29 Jul 2020 13:48:49 CDT - [GET] "200 71.346µs "
Wed, 29 Jul 2020 13:48:49 CDT - [GET] "200 122.838µs "
Wed, 29 Jul 2020 13:48:49 CDT - [GET] "200 10.033µs "
PASS
ok      flygreeting 0.326s

Hello @mrkurt ,

I just checked the tests in the repo. What I can see is that the repo has tests, but not against the preprod environment but an http test server. This means step 6. cannot be done. Here is what I would suggest (to modify this tasks description):

Install dependencies
Run the unit tests
Build and upload docker image to registry
Deploying the app to the environment defined in fly.toml

The reason I reduced the number of steps is that the Dockerfile you sent does multistage docker build.

With the flygreetings application it would make more sense. What do you think?

That sounds good to me.

@mrkurt Could you please help me with a weird issue?

The workflow on GitLab CI works perfectly until it reaches the deploy part. At that point, flyctl deploy -i <IMAGE TAG> says: 401 Unauthorized.

I did login with flyctl auth docker, the FLY_ACCESS_TOKEN is set, the image is pulled, the docker daemon knows about the fly.io registry, the credentials are extended and so on. Still, I cannot deploy from GitLab docker container. When I try to use repository:tag with -i flag the result is the same.

To me it seems that the CLI tool runs into the https://github.com/superfly/flyctl/blob/0861671ea17e0fea34fc24eb58bafbb906fff259/docker/deploy.go#L137 branch, but there I lost the track.

Could someone help me to debug this issue? Here are the details below (ofc, the sensitive data is filtered out):

$ /root/.fly/bin/flyctl auth docker
Authentication successful. You can now tag and push images to registry.fly.io/{your-app}
$ docker pull "$DOCKER_IMAGE"
00ff9dda: Pulling from flygreeting-gitlabci
df20fa9351a1: Pulling fs layer
c237c00e8766: Pulling fs layer
f4bcc4c628bd: Pulling fs layer
5a837f3b7fc6: Pulling fs layer
ff785fb4a1fe: Pulling fs layer
5a837f3b7fc6: Waiting
ff785fb4a1fe: Waiting
c237c00e8766: Verifying Checksum
c237c00e8766: Download complete
f4bcc4c628bd: Verifying Checksum
f4bcc4c628bd: Download complete
df20fa9351a1: Verifying Checksum
df20fa9351a1: Download complete
5a837f3b7fc6: Verifying Checksum
5a837f3b7fc6: Download complete
ff785fb4a1fe: Verifying Checksum
ff785fb4a1fe: Download complete
df20fa9351a1: Pull complete
c237c00e8766: Pull complete
f4bcc4c628bd: Pull complete
5a837f3b7fc6: Pull complete
ff785fb4a1fe: Pull complete
Digest: sha256:721248a5a85732836900cb2eda51bac46dbee1dd481f77980698c121460c114f
Status: Downloaded newer image for registry.fly.io/flygreeting-gitlabci:00ff9dda
registry.fly.io/flygreeting-gitlabci:00ff9dda
$ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
registry.fly.io/flygreeting-gitlabci   00ff9dda            f839038c0494        39 seconds ago      20.7MB
$ cat ~/.docker/config.json
{
    "auths": {
        "registry.fly.io": {
            "auth": "<FILTERED>"
        }
    },
    "HttpHeaders": {
        "User-Agent": "Docker-Client/19.03.12 (linux)"
    }
}$ /root/.fly/bin/flyctl deploy -i "$CI_COMMIT_SHORT_SHA"
Deploying flygreeting-gitlabci
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 8080 
Deploying image: 00ff9dda
Error Unable to access image 00ff9dda: 401 Unauthorized

@gabor-boros Try this: /root/.fly/bin/flyctl deploy -i "registry.fly.io/flygreeting-gitlabci:$CI_COMMIT_SHORT_SHA"

@mrkurt Unfortunately, I tried it already. I mean, I started with that:

$ docker images
REPOSITORY                             TAG                 IMAGE ID            CREATED             SIZE
registry.fly.io/flygreeting-gitlabci   ae5f0ff7            6eca018f31b3        28 seconds ago      20.7MB

[...]

$ /root/.fly/bin/flyctl deploy -i "$DOCKER_IMAGE"
Deploying flygreeting-gitlabci
==> Validating App Configuration
--> Validating App Configuration done
Services
TCP 80/443 ⇢ 8080 
Deploying image: registry.fly.io/flygreeting-gitlabci:ae5f0ff7
Error Unable to access image registry.fly.io/flygreeting-gitlabci:ae5f0ff7: 401 Unauthorized
ERROR: Job failed: exit code 1

Can you put the source somewhere?

It might be simpler to just do flyctl deploy and let it handle the docker build. There's not much reason to push to fly and then deploy separately.

I already tried to deploy with the image id parsed from docker images output. The result is the same.

I tried it locally and it worked so I'm slowly convinced that there should be a bug somewhere on GitLab's side, though I still don't get it, since I can see the image listed.

Is it possible that flyctl cannot find images, when docker-in-docker is used?

Can you put the source somewhere?

It might be simpler to just do flyctl deploy and let it handle the docker build. There's not much reason to push to fly and then deploy separately.

Sure, I'll do it tomorrow morning. Just wanted to avoid building the image with the deploy command.

Anyway, it is suspicious that the image is listed and not found by the deploy command.

@mrkurt I want able to wait. If I don't try to pass the -I flag and let deploy build the image it works

I'd say just finish the example with that and we can worry about why it can't push an built image later.

Hello @mrkurt ,

Today morning I figured out what the issue is. The problem was setting TLS for docker daemon in a dind setup. I have no idea (yet) why was this an issue.

Anyway, I made some adjustments and being able to deploy a locally built image (also, the cli tool now finds the docker image).

Can you put the source somewhere? It might be simpler to just do flyctl deploy and let it handle the docker build. There's not much reason to push to fly and then deploy separately.

I put it to: https://gitlab.com/gabor-boros/flygreeting-gitlab-ci

Also, removed the step which built the image and merged it to the deploy part. My reasoning behind the original setup: I prepared the pipeline to have an integration test against a preprod environment which became invalid since no integration tests exists for the greeting app.

The flow now looks like this:

download dependencies
run tests + run vet
build and deploy (on master build)

In case you think having a separate step for preprod testing and deploying to production after preprod testing passed, let me know and I'll adjust the pipeline based on that. Otherwise, comments are welcomed. 🙏

The pipeline: https://gitlab.com/gabor-boros/flygreeting-gitlab-ci/-/pipelines/174018840

Any observations @mrkurt ? 😇

@mrkurt / @codepope

I just added some documentation to the example as stated here. You can check the docs at: https://gitlab.com/gabor-boros/flygreeting-gitlab-ci/-/blob/master/README.md

Could you please give some feedback on the example? Thank you in advance.

Hi @gabor-boros. Sorry for the slow response on this, we've been working on some guidelines for CI articles. I will send you an email tonight or tomorrow.

@mrkurt Sure, I'll be around

@mrkurt or @codepope should I change the readme? Is there any specific format you would like to have?

superfly / fly