kratix `./scripts/quick-start.sh --git` does not ever run

[abangser]

When running the quick start script with just git, the system never reconciles. This is because the worker does not have the right secret for gitea.

Example output:

quick_start_with_platform_destination
⬇️
~/dev/syntasso/kratix ~/dev/syntasso/kratix-demos/backstage
running: ./scripts/quick-start.sh --git --recreate
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
Loading KinD images... ✓
No kind clusters found.
No kind clusters found.
Creating worker destination...
Creating platform destination...
 \ ✓
Finished creating platform destination...
Loading images in platform destination...
 ✓
Finished creating worker destination...
 ✓
Finished loading images in platform destination...
Setting up platform destination...
 ✓
Finished setting up platform destination...
Setting up worker destination...
 ✓
Finished setting up worker destination...
Waiting for local repository to be running... ✓
Waiting for system to reconcile... -

It's taking longer than usual for the system to reconcile.
You can check the pods on the platform and worker Destinations for debugging information.
This script will continue to wait. You can kill it with CTRL+C.

Waiting for local repository to be running... /^C/Users/abbybangser/dev/syntasso/kratix/scripts/utils.sh: line 88: kill: (-7773)

state of secrets in worker cluster:

$ k get secrets -A --context kind-worker
NAMESPACE     NAME                     TYPE                            DATA   AGE
kube-system   bootstrap-token-abcdef   bootstrap.kubernetes.io/token   6      15m

state of secrets in platform cluster:

$ k get secrets -A --context kind-platform
NAMESPACE                NAME                      TYPE                            DATA   AGE
cert-manager             cert-manager-webhook-ca   Opaque                          3      15m
default                  gitea-credentials         Opaque                          4      14m
flux-system              gitea-credentials         Opaque                          4      14m
gitea                    gitea                     Opaque                          1      15m
gitea                    gitea-credentials         Opaque                          4      14m
gitea                    gitea-init                Opaque                          2      15m
gitea                    gitea-inline-config       Opaque                          7      15m
gitea                    gitea-postgresql          Opaque                          2      15m
kratix-platform-system   webhook-server-cert       kubernetes.io/tls               3      14m
kube-system              bootstrap-token-abcdef    bootstrap.kubernetes.io/token   6      16m

[abangser]

Another thing to note, if you create with --git-and-minio it installs both state store applications, but only creates destinations / flux configurations for minio. Not sure if this is what people expect from this.

[cmoulliard]

FYI: I did a local test ./scripts/quick-start.sh --recreate on a macos machine running podman, no errors are reported if I list the pods on the platform or worker but script hangs at line

./scripts/quick-start.sh --recreate      
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
No kind clusters found.
No kind clusters found.
Creating worker destination...
Creating platform destination...
 \Finished creating worker destination ✓                                                                                                                                                                Finished creating platform destination ✓
Setting up platform destination... ✓
Setting up worker destination... ✓
Waiting for local repository to be running... -

[cmoulliard]

The script ./scripts/quick-start.sh --git fails as git client is not installed

 ./scripts/quick-start.sh --recreate --git
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
No kind clusters found.
No kind clusters found.
Creating platform destination...
Creating worker destination...
 \Finished creating worker destination ✓                                                                                                                                                                Finished creating platform destination ✓
Setting up platform destination... ✗
Combined output:
namespace/cert-manager created
customresourcedefinition.apiextensions.k8s.io/certificaterequests.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/certificates.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/challenges.acme.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/issuers.cert-manager.io created
customresourcedefinition.apiextensions.k8s.io/orders.acme.cert-manager.io created
serviceaccount/cert-manager-cainjector created
serviceaccount/cert-manager created
serviceaccount/cert-manager-webhook created
configmap/cert-manager-webhook created
clusterrole.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrole.rbac.authorization.k8s.io/cert-manager-view created
clusterrole.rbac.authorization.k8s.io/cert-manager-edit created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrole.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrole.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-cainjector created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-issuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-clusterissuers created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificates created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-orders created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-challenges created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-ingress-shim created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-approve:cert-manager-io created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-controller-certificatesigningrequests created
clusterrolebinding.rbac.authorization.k8s.io/cert-manager-webhook:subjectaccessreviews created
role.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
role.rbac.authorization.k8s.io/cert-manager:leaderelection created
role.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
rolebinding.rbac.authorization.k8s.io/cert-manager-cainjector:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager:leaderelection created
rolebinding.rbac.authorization.k8s.io/cert-manager-webhook:dynamic-serving created
service/cert-manager created
service/cert-manager-webhook created
deployment.apps/cert-manager-cainjector created
deployment.apps/cert-manager created
deployment.apps/cert-manager-webhook created
mutatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
validatingwebhookconfiguration.admissionregistration.k8s.io/cert-manager-webhook created
deployment.apps/cert-manager condition met
deployment.apps/cert-manager-cainjector condition met
deployment.apps/cert-manager-webhook condition met
namespace/gitea created
deployment.apps/gitea-memcached created
service/gitea-memcached created
serviceaccount/gitea-memcached created
secret/gitea-postgresql created
statefulset.apps/gitea-postgresql created
service/gitea-postgresql-headless created
service/gitea-postgresql created
secret/gitea-inline-config created
secret/gitea created
service/gitea-http created
secret/gitea-init created
service/gitea-ssh created
statefulset.apps/gitea created
job.batch/gitea-create-repository created
role.rbac.authorization.k8s.io/gitea-create-repository created
rolebinding.rbac.authorization.k8s.io/gitea-create-repository created
serviceaccount/gitea-create-repository created
gitea cli not found; run 'make gitea-cli' to download it
go: downloading go1.22 (darwin/arm64)
go: download go1.22 for darwin/arm64: toolchain not available
chmod: /Users/cmoullia/code/kratix/kratix/bin/gitea: No such file or directory
make: *** [gitea-cli] Error 1

 failed

Gitea client is well installed on my machine

 ~/code/kratix  gitea --version
Gitea version 1.21.5 built with GNU Make 3.81, go1.21.6 : bindata, sqlite, sqlite_unlock_notify

[kirederik]

i could not reproduce the errors on latest main (same results with or without recreate)

./scripts/quick-start.sh --git --recreate
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
Loading KinD images... ✓
No kind clusters found.
No kind clusters found.
Creating platform destination...
Creating worker destination...
  Finished creating platform destination ✓
Finished creating worker destination ✓
Loading images in platform destination... ✓
Setting up platform destination... ✓
Setting up worker destination... ✓
Waiting for local repository to be running... ✓
Waiting for system to reconcile... ✓
Kratix installation is complete!

The script ./scripts/quick-start.sh --git fails as git client is not installed

the script should download the gitea cli to the kratix/bin directory; odd that it hasn't. Can you run make gitea-cli and check that it gets in there? nonetheless, i pushed a fix to use the system gitea client if that's available in the host machine

FYI: I did a local test ./scripts/quick-start.sh --recreate on a macos machine running podman, no errors are reported if I list the pods on the platform or worker but script hangs at line

will try this one next

[cmoulliard]

the script should download the gitea cli to the kratix/bin directory; odd that it hasn't. Can you run make gitea-cli and check that it gets in there? nonetheless, i pushed a fix to use the system gitea client if that's available in the host machine

Can you please check if the folder bin exists and create it if this is not the case otherwise users will got this issue

make gitea-cli
chmod: /Users/cmoullia/code/kratix/kratix/bin/gitea: No such file or directory
make: *** [gitea-cli] Error 1

mkdir bin
make gitea-cli
...
ls -la bin
total 296072
drwxr-xr-x@  3 cmoullia  staff         96 May  8 12:16 .
drwxr-xr-x@ 34 cmoullia  staff       1088 May  8 12:16 ..
-rwxr-xr-x@  1 cmoullia  staff  136926322 May  8 12:16 gitea

[cmoulliard]

i could not reproduce the errors on latest main (same results with or without recreate)

Gitea's pod fails to start on kind-platform when we execute

./scripts/quick-start.sh --git --recreate       
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
...

Pod's log

k -n gitea logs gitea-0 -c configure-gitea
==== BEGIN GITEA CONFIGURATION ====
2024/05/08 10:26:30 cmd/migrate.go:34:runMigrate() [I] AppPath: /usr/local/bin/gitea
2024/05/08 10:26:30 cmd/migrate.go:35:runMigrate() [I] AppWorkPath: /data
2024/05/08 10:26:30 cmd/migrate.go:36:runMigrate() [I] Custom path: /data/gitea
2024/05/08 10:26:30 cmd/migrate.go:37:runMigrate() [I] Log path: /data/log
2024/05/08 10:26:30 cmd/migrate.go:38:runMigrate() [I] Configuration file: /data/gitea/conf/app.ini
2024/05/08 10:26:30 ...e/cli@v1.22.9/app.go:277:Run() [I] PING DATABASE postgres
2024/05/08 10:26:30 cmd/migrate.go:41:runMigrate() [F] Failed to initialize ORM engine: dial tcp 10.96.165.193:5432: connect: connection refused
Gitea migrate might fail due to database connection...This init-container will try again in a few seconds

[cmoulliard]

Why don't you use the idpbuilder which is provisioning a kind cluster + gitea + ingress + argocd OOTB instead of redeveloping scripts installing gitea, etc ? @abangser

[kirederik]

Why don't you use the idpbuilder which is provisioning a kind cluster + gitea + ingress + argocd OOTB instead of redeveloping scripts installing gitea, etc ? @abangser

our quick-start predates idpbuilder; it's also under active development and, last time we looked into it, it didn't quite work well for our usecase; will take a look at it again to see if its a bit more stable.

will check it with podman and see if that might be why its failing on your side 🤔

[cmoulliard]

will check it with podman and see if that might be why its failing on your side 🤔

I use this podman version on macos

podman version 5.0.2

[jessesanford]

I would like to see this on idpbuilder! Happy to help.

[kirederik]

@cmoulliard tested on podman and i could reproduce the error; pushed a fix and it should now work; could you give it a try?

[abangser]

Why don't you use the idpbuilder which is provisioning a kind cluster + gitea + ingress + argocd OOTB instead of redeveloping scripts installing gitea, etc ? @abangser

Hey @cmoulliard Thanks again for sharing your context here. As @kirederik shared, our quick-start has been under development from before the idp-builder was ready to go so this is a matter of investing in a migration at this point.

I have done some quick research and while Kratix runs easily on top of the current idp builder, there are a number of things we need to consider before moving to use idp builder as a dependency for our project:

1. The github warning WORK IN PROGRESS: This tool is in a pre-release stage and is under active development. is concerning. Do you have a sense of when you may release a stable v1?
2. We depend heavily on a multi-cluster setup. I believe this has been discussed, but I did not find a corresponding issue in your repo. If we move to idp builder now, we will likely be managing the clusters with different setups which is not ideal for our maintenance.
3. Based on our customers, we need to be able to equally use MinIO in lieu of Gitea as a state store, and Flux in lieu of ArgoCD for GitOps. Again, I know that more flexibility in the packages provided is in the works, but I believe this is not yet fully designed/implemented. We can likely work around this by just double installing, but this is a consideration before switching.

I don't believe any of these are long term blockers, but hopefully they provide a bit of transparency as to why it may not be an immediate switch. We are actively thinking about this and will keep you posted and of course continue to excitedly follow the idp builder project!

[abangser]

@jessesanford it would be great to hear how you might use Kratix on an IDP Builder built cluster as I believe there are no blockers here. As I mentioned in the above message to Charles, use in the project for our local testing may need a bit more research/investment, but for use by others we just need to provide an Argo App that could be an additional package for you. I have tested it by manually installing Kratix with the following commands and it appeared healthy:

version=$(curl -Ls -o /dev/null -w %{url_effective} https://github.com/cnoe-io/idpbuilder/releases/latest)\nversion=${version##*/}\ncurl -L -o ./idpbuilder.tar.gz "https://github.com/cnoe-io/idpbuilder/releases/download/${version}/idpbuilder-$(uname | awk '{print tolower($0)}')-$(uname -m | sed 's/x86_64/amd64/').tar.gz"\ntar xzf idpbuilder.tar.gz\n\n./idpbuilder version\n# example output\n# idpbuilder 0.3.0 go1.21.5 linux/amd64
./idpbuilder create
kubectl apply --filename https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.yaml
kubectl apply --filename https://github.com/syntasso/kratix/releases/latest/download/install-all-in-one.yaml
kubectl apply --filename https://github.com/syntasso/kratix/releases/latest/download/config-all-in-one.yaml

Let me know if this unblocks what you are up to, or if you are looking to move further, for example adding Kratix as an example additional package and of course we can collaborate to help with whatever you need!

[cmoulliard]

2. We depend heavily on a multi-cluster setup. I believe this has been discussed, but I did not find a corresponding issue in your repo. If we move to idp builder now, we will likely be managing the clusters with different setups which is not ideal for our maintenance.

You can use your own kind config file when you create a cluster using the following idpbuilder command - https://github.com/cnoe-io/idpbuilder/tree/main?tab=readme-ov-file#example-commands

./idpbuilder create --build-name local --kind-config ./my-kind.yaml

[cmoulliard]

3. Based on our customers, we need to be able to equally use MinIO in lieu of Gitea as a state store, and Flux in lieu of ArgoCD for GitOps. Again, I know that more flexibility in the packages provided is in the works, but I believe this is not yet fully designed/implemented

This is correct and I'm also in favor to offer such "customization" even for the idpbuilder core components. We could adopt your approach as developed here to switch from one type of store: git vs bucket - https://github.com/syntasso/kratix/blob/1b285391656bd5a4c6dd2abaf772be660867ffcc/controllers/shared.go#L111-L140

Remark: I don't think that you created an APi to configure the CD engine to be used - correct ?

[cmoulliard]

FYI: I did a local test ./scripts/quick-start.sh --recreate on a macos machine running podman, no errors are reported if I list the pods on the platform or worker but script hangs at line

I did a new test from the latest commit sha: 2700053 pushed on the main branch and I was able to run the script using podman rootful without issues

./scripts/quick-start.sh --recreate
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
No kind clusters found.
No kind clusters found.
Creating worker destination...
Creating platform destination...
 \ ✓
Finished creating worker destination ✓
 ✓
Finished creating platform destination ✓
Setting up platform destination... ✓
Setting up worker destination... ✓
Waiting for local repository to be running... ✓
Waiting for system to reconcile... ✓
Kratix installation is complete!

If you are following the docs available at kratix.io, make sure to set the following environment variables:
export PLATFORM=kind-platform
export WORKER=kind-worker

[cmoulliard]

Gitea's pod fails to start on kind-platform when we execute

This is fixed too using main branch commit 27000538 and podman (5.0.2 on macos) & rootfull

./scripts/quick-start.sh --git --recreate
Looking for KinD... ✓
Looking for kubectl... ✓
Looking for docker... ✓
Looking for distribution/kratix.yaml...distribution/kratix.yaml not found; downloading latest version...
 ✓
Deleting pre-existing clusters... ✓
No kind clusters found.
No kind clusters found.
Creating worker destination...
Creating platform destination...
 \ ✓
Finished creating worker destination ✓
 ✓
Finished creating platform destination ✓
Setting up platform destination... ✓
Setting up worker destination... ✓
Waiting for local repository to be running... ✓
Waiting for system to reconcile... ✓
Kratix installation is complete!

If you are following the docs available at kratix.io, make sure to set the following environment variables:
export PLATFORM=kind-platform
export WORKER=kind-worker

[kirederik]

Awesome, thanks @cmoulliard

I'm closing this issue since it looks like the problem was resolved;

As for using idpbuilder, i moved the chat to a discussion thread https://github.com/syntasso/kratix/discussions/133