search

devspace-sh / devspace

DevSpace - The Fastest Developer Tool for Kubernetes ⚡ Automate your deployment workflow with DevSpace and develop software directly inside Kubernetes.
https://devspace.sh
Apache License 2.0
4.19k stars 351 forks source link

Pull secrets are not created for dependant projects even with `ensureSecrets` #2463

Open gustaff-weldon opened 1 year ago

gustaff-weldon commented 1 year ago

What happened?

I have a main API gateway project, with multiple services it depends on, and my devspace setup reflects that ie. gateway devspace depends on other service's devspaces.

All of my projects deploy to their own namespaces and pull secret is the same for all of them. (I think this might be a crucial info)

If I devspace deploy from the main project, secrets are ONLY created for the main project namespace. They are not created for the dependency namespaces.

I believe ensure_secrets does not take namespace of the subproject into account and always uses the top project namespace, so only first ensure_secrets call creates a secret and rest of calls just check for a secret (in wrong, main namespace) and skip creating it.

More info

When I run devspace deploy from gateway (main project with deps), I see output related to secrets for subprojects eg.

cargo pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
booking pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
contracts pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...

What is interesting:

contracts pullsecret:aws-ecr Created image pull secret graphql-gateway/devspace-pull-secrets

The above looks like a bug, as graphql-gateway is the namespace of the main project not the contracts.

If I go to single dependency folder and do devspace deploy there, secrets are created:

pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
pullsecret:aws-ecr Created image pull secret cargo/devspace-pull-secrets

What did you expect to happen instead? I expected the pull secrets to be created for main project and its dependencies, each of the pull secret in the proper project namespace.

How can we reproduce the bug? (as minimally and precisely as possible)

Use the reproduction repo: https://github.com/gustaff-weldon/devspace-dependency-pull-secrets

To reproduce issue you need to configure the docker registry that needs auth in both devspace files and update image url

I also configured docker config in ~/.docker/config.json similar to:

{
    "auths": {
        "123456789.dkr.ecr.eu-west-1.amazonaws.com": {},
    },
    "credHelpers": {
        "123456789.dkr.ecr.eu-west-1.amazonaws.com": "ecr-login"
    }
}

Now, try deploying main project:

❯ cd ../main-project
❯ devspace deploy

You will get output similar to:

❯ devspace deploy
info Using namespace 'main-project'
info Using kube context 'docker-desktop'
sub-project pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
sub-project pullsecret:aws-ecr Created image pull secret main-project/devspace-pull-secrets
sub-project deploy:sub-project Deploying chart /Users/edited/.devspace/component-chart/component-chart-0.8.5.tgz (sub-project) with helm...
sub-project deploy:sub-project Deployed helm chart (Release revision: 1)
sub-project deploy:sub-project Successfully deployed sub-project with helm
pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
deploy:main-project Deploying chart /Users/edited/.devspace/component-chart/component-chart-0.8.5.tgz (main-project) with helm...
deploy:main-project Deployed helm chart (Release revision: 1)
deploy:main-project Successfully deployed main-project with helm

Check the secrets for the main project:

❯ kubectl get secrets --namespace main-project
NAME                                    TYPE                             DATA   AGE
devspace-cache-main-project             devspace.sh/remote-cache         1      3m5s
devspace-cache-sub-project              devspace.sh/remote-cache         1      3m7s
devspace-pull-secrets                   kubernetes.io/dockerconfigjson   1      92s
sh.helm.release.v1.main-project.v1   helm.sh/release.v1               1      89s

And the sub-project:

❯ kubectl get secrets --namespace sub-project
NAME                                TYPE                 DATA   AGE
sh.helm.release.v1.sub-project.v1   helm.sh/release.v1   1      118s
If you try to deploy just the project a:

Now, if you deploy the subproject on its own:

❯ cd sub-project
❯ devspace deploy
❯ devspace deploy

warn Are you using the correct namespace?
warn Current namespace: 'sub-project'
warn Last    namespace: 'devspace'

? Which namespace do you want to use? sub-project
info Using namespace 'sub-project'
info Using kube context 'docker-desktop'
pullsecret:aws-ecr Ensuring image pull secret for registry: 123456789.dkr.ecr.eu-west-1.amazonaws.com...
pullsecret:aws-ecr Created image pull secret sub-project/devspace-pull-secrets
deploy:sub-project Deploying chart /Users/edited/.devspace/component-chart/component-chart-0.8.5.tgz (sub-project) with helm...
deploy:sub-project Deployed helm chart (Release revision: 2)
deploy:sub-project Successfully deployed sub-project with helm

You will see the devspace-pull-secrets secret was created:

❯ kubectl get secrets --namespace sub-project
NAME                                TYPE                             DATA   AGE
devspace-cache-sub-project          devspace.sh/remote-cache         1      35s
devspace-pull-secrets               kubernetes.io/dockerconfigjson   1      37s
sh.helm.release.v1.sub-project.v1   helm.sh/release.v1               1      3m29s
sh.helm.release.v1.sub-project.v2   helm.sh/release.v1               1      35s

Local Environment:

lizardruss commented 1 year ago

Thank you for all the detail and the example repository. The fix for the issue of pull secrets not being created in the correct namespaces for dependent projects would be to modify this configuration to:

dependencies:
  sub-project:
    path: ../sub-project
    namespace: sub-project

This will instruct DevSpace to configure its client to use the correct namespace when running the dependency (and all the steps within).

As you've pointed out, you'd like to pin each sub-project to different namespaces whether they are dependencies or not. This has led to workarounds in the example repository such as setting the namespace for deployments and modifying DEVSPACE_FLAGS

gustaff-weldon commented 1 year ago

Thanks @lizardruss the workaround works, but means projects no longer are self-contained. In our scenario, a dozen of projects that eg. depend on a shared postgresql, have to know its namespace name, and we need to make sure those are kept in sync and no typos occur between devspace.yamls.

I appreciate the workaround, but if there could be a way to define namespace for pull secrets on a project level (even if that would be on pullSecrets.namespace, that would work better in our scenario.