Makefile?

[kaihendry]

Just trying to follow from https://youtu.be/z6pYfz_q9tc?t=2006 and I'm a little confused how you:

1. Setup the cluster
2. What commands you ran to deploy

Perhaps a Makefile or README would help?

Ideally you could have a Github workflow to boot?

[kpenfound]

Hey! Thanks for asking! I will update the readme with more info, but here's the high level:

1. The cluster was deployed with https://eksctl.io/ , It's a super easy single command to configure everything needed for eks. eksctl create cluster --name hello-eks --region us-east-1
2. To run the dagger pipeline, I just ran go run ci/main.go. The entire pipeline is defined here: https://github.com/kpenfound/hello-eks/blob/main/ci/main.go

[kaihendry]

Failed with:

Status: Downloaded newer image for ghcr.io/dagger/engine@sha256:41f71f0036167fcc8b926792b079d840c52066eec093f3e177881468fd122c83
                                 Error: buildkit failed to respond
panic: EOF: Error: buildkit failed to respond

I am running:

❯ docker ps
CONTAINER ID   IMAGE                          COMMAND                  CREATED              STATUS              PORTS     NAMES
9c172890339a   ghcr.io/dagger/engine:v0.3.5   "buildkitd --oci-wor…"   About a minute ago   Up
About a minute             dagger-engine-41f71f0036167fcc

Subsequent run was:

❯ go run ci/main.go
#1 resolve image config for docker.io/library/golang:latest
#1 ERROR: failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/latest": net/http: TLS handshake timeout
------
 > resolve image config for docker.io/library/golang:latest:
------
panic: input:1: container.from failed to do request: Head "https://registry-1.docker.io/v2/library/golang/manifests/latest": net/http: TLS handshake timeout

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
        /home/hendry/tmp/kpenfound/hello-eks/ci/main.go:67 +0x1439
exit status 2

Third run is appears stuck at:

#5 extracting sha256:195ea6a58ca87a18477965a6e6a8623112bde82c5b568a29c56ce4581b6e6695 1.0s done
#5 extracting sha256:52908dc1c386fab0271a2b84b6ef4d96205a98a8c8801169554767172e45d8c7
#5 extracting sha256:52908dc1c386fab0271a2b84b6ef4d96205a98a8c8801169554767172e45d8c7 1.0s done
#5 extracting sha256:a2b47720d601b6c6c6e7763b4851e25475118d80a76be466ef3aa388abf2defd

Not sure how I'm supposed to make sense of that! 😬

[kaihendry]

Fourth run after 9m15s:

Not sure what Cloudflare has to do with https://github.com/kpenfound/hello-eks/blob/main/ci/main.go

Fifth run:

#13 copy /hello /bin/hello
#13 DONE 0.1s

#14 exporting to image
#14 exporting layers
#14 exporting layers 0.7s done
#14 exporting manifest sha256:57d97a485b1da60b550d5cf793c6d141b4284cdf5f3bfa11155f7194dbafbac8 done
#14 exporting config sha256:3a624da7056db2584010c4aa621180add2ac9781d8cbc2b6528040a5402d8694 done
#14 exporting manifest sha256:fe03730a1be27684bfccaede87f5d135caff4528926c45200d9306c6207a7987 done
#14 exporting config sha256:0d47e445990d227f92c57fd785977925128e3e3f4e551ccdc81828eda3898a02 done
#14 exporting manifest list sha256:7d53aa79649f16f11891a87d4ab775bf527f4bd03140df17aa58fa183f545712 done
#14 pushing layers
#14 pushing layers 2.3s done
#14 ERROR: unexpected status: 401 Unauthorized
------
 > exporting to image:
------
panic: input:1: container.publish failed to solve: unexpected status: 401 Unauthorized

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
        /home/hendry/tmp/kpenfound/hello-eks/ci/main.go:67 +0x1439
exit status 2

I suspect it's a region issue.

Hmmm, still same problem after tweaking region:

#14 pushing layers 1.5s done
#14 ERROR: unexpected status: 401 Unauthorized
------
 > exporting to image:
------
panic: input:1: container.publish failed to solve: unexpected status: 401 Unauthorized

Please visit https://dagger.io/help#go for troubleshooting guidance.

goroutine 1 [running]:
main.main()
        /home/hendry/tmp/kpenfound/hello-eks/ci/main.go:67 +0x1439
exit status 2

hello-eks on  main [!] via 🐹 v1.19.3 on ☁  tw (ap-southeast-1) took 1m29s
❯ git diff
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
modified: ci/main.go
───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
@ ci/main.go:107 @ func rollingDeployment(ctx context.Context, clientset *kubernetes.Clientset, ima
func getKubeClient(ctx context.Context) (*kubernetes.Clientset, error) {
    // Get EKS service
    sess := session.Must(session.NewSession(&aws.Config{
        Region: aws.String("us-east-1"),
        Region: aws.String("ap-southeast-1"),
    }))
    eksSvc := eks.New(sess)

I don't understand where it's trying to push to.

Oh I see, I need to change your repo to mine... and then login aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/...

Finally!

#13 copy /hello /bin/hello
#13 CACHED

#14 exporting to image
#14 exporting layers done
#14 exporting manifest sha256:c20032dd8875a68c43b05e354cee858f6bf12882e7d04c44edb4a370ecddd1df done
#14 exporting config sha256:f30cdadc0013bc1a51e001ea826dca80a75290b6760cdfdbca8fff0896f049a0 done
#14 exporting manifest sha256:43fe448ae4c90d72b7453ce9342b6ad9ea6ec758379ab2966ee04520f8f5bada done
#14 exporting config sha256:ec73d78320626dd04dfb573839675df321dc4ab7c65322ff9a6eabc29d5236ba done
#14 exporting manifest list sha256:d6b764ebef6b9198dab8d9f2ff50dca041f1b660275deccc6c3afdba998c95a7 done
#14 pushing layers
#14 pushing layers 8.0s done
#14 pushing manifest for public.ecr.aws/z6b8a8c7/hello:dev@sha256:d6b764ebef6b9198dab8d9f2ff50dca041f1b660275deccc6c3afdba998c95a7
public.ecr.aws/z6b8a8c7/hello:dev@sha256:d6b764ebef6b9198dab8d9f2ff50dca041f1b660275deccc6c3afdba998c95a7
2022/11/28 14:23:35 here
#14 pushing manifest for public.ecr.aws/z6b8a8c7/hello:dev@sha256:d6b764ebef6b9198dab8d9f2ff50dca041f1b660275deccc6c3afdba998c95a7 3.1s done
#14 DONE 11.2s
Error deploying hello-eks: deployments.apps "hello-eks" not foundUpdated hello-eks deployment

I don't understand that last error message: Error deploying hello-eks: deployments.apps "hello-eks" not foundUpdated hello-eks deployment

Maybe a path issue to https://github.com/kpenfound/hello-eks/tree/main/kubernetes

[kaihendry]

I think I managed to get it working if I manually kubectl apply.

Another comment, why isn't go test run automatically?

[kpenfound]

Glad you got it working! It looks like some of the issues may have been dockerhub rate limiting 🤔

Another comment, why isn't go test run automatically?

Short answer - this demo repo is meant to be a simple example of deploying an application to EKS, and not necessarily a production-grade system. For a better idea of how dagger is used in production, check out how we're using it for the dagger repo itself here: https://github.com/dagger/dagger/tree/main/.github/workflows

For more DIY examples, check out https://github.com/dagger/examples or any of the guides at https://docs.dagger.io/

[kaihendry]

I don't quite understand Dagger. I thought the idea is that it gives you a vendor independent CI/CD workflow, i.e. not defined by a company like Github or Gitlab's .gitlab-ci.yml. Perhaps I've misunderstood the promise of Dagger?

[kpenfound]

The headline is: Dagger is a programmable CI/CD engine that runs your pipelines in containers.

At a certain level, it moves your cicd pipeline definition from platform-specific yaml to portable code. Meaning you can run the same pipeline locally or in your CI platform of choice.

More details on the promise of dagger here: https://docs.dagger.io/

[kaihendry]

"platform-specific yaml to portable code" <-- can you please give an example of this, because platform-specific yaml != Github workflows, right? :grimacing:

[kpenfound]

Yeah that's a fair question 😄 Today, that means you're no longer using platform specific yaml to define your pipeline, however it still defines when and how your dagger pipeline runs.

A practical example of this is the test pipeline here It runs go test on a matrix of 2 different Go versions and 2 different platforms. Using platform-specific yaml for this task, you'd define a matrix and within the matrix run go test. The drawbacks of that are that this yaml configuration can't be run as-is outside of the CI platform, and the syntax to accomplish this is different between platforms.

For the above dagger pipeline, its executed in both Github Actions and CircleCI. In each there's just some boilerplate to define when the pipeline should run, setup qemu (which this pipeline uses, but not a requirement of dagger in general), and sets up the runtime. The amount of boilerplate is something we'd ideally like to reduce to 0, and there are ways to reduce it more than the above example, but it's doing the "important" parts in code which is the point.

The pipeline is a pretty simple one, but in general yaml gets really complex when it's leaned on for defining matrices, environments/secrets, dependencies/runtimes, branching logic, platform-specific extensions (actions/orbs/plugins). The more complex the yaml, the harder it is to reproduce the CICD pipeline outside of the CI platform, and that's where Dagger comes in.

[kaihendry]

I see, I guess it needs to be more mature for me to readily use it. It appears to be a problem worth solving, though tbh most my pipelines are pretty simple. Not need for matrixes et al.

Thank you!

[kpenfound]

My pleasure! If you're not facing the challenges described on the docs site, then definitely no reason to change anything! 😄