search

shopware / shopware-operator

Operator manages Shopware installations in a Kubernetes cluster.
Apache License 2.0
9 stars 0 forks source link

Stuck in setup #65

Open mstingl opened 2 days ago

mstingl commented 2 days ago

Steps to reproduce

Deployment of a Shop through shopware helm chart

What should happened?

The setup job should pass

What actually happened?

The setup job is instantly deleted (and the pod killed) after creation, leaving the operator stuck in the setup stage. Repeating every 10 seconds

Relevant log output

(Namespace, deployment name and URLs are changed)

2024-11-23 12:31:03    [INFO]    my-shop    Wait for setup to finish    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "8407e3f2-d04c-494c-aca0-5d30fab0911d", "state":
2024-11-23 12:31:03    [INFO]    Reconcile finished    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "8407e3f2-d04c-494c-aca0-5d30fab0911d"}                           
2024-11-23 12:31:03    [ERROR]    job not found in container: shopware-setup    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "8407e3f2-d04c-494c-aca0-5d30fab0911d", "
github.com/shopware/shopware-operator/internal/job.IsJobContainerDone                                                                                                                                                                                                                                                                                              
    /home/runner/work/shopware-operator/shopware-operator/internal/job/util.go:89                                                                                                                                                                                                                                                                                  
github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).stateSetup                                                                                                                                                                                                                                                                            
    /home/runner/work/shopware-operator/shopware-operator/internal/controller/status.go:232                                                                                                                                                                                                                                                                        
github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).reconcileCRStatus                                                                                                                                                                                                                                                                     
    /home/runner/work/shopware-operator/shopware-operator/internal/controller/status.go:63                                                                                                                                                                                                                                                                         
github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).Reconcile.func1                                                                                                                                                                                                                                                                       
    /home/runner/work/shopware-operator/shopware-operator/internal/controller/store_controller.go:107                                                                                                                                                                                                                                                              
github.com/shopware/shopware-operator/internal/controller.(*StoreReconciler).Reconcile                                                                                                                                                                                                                                                                             
    /home/runner/work/shopware-operator/shopware-operator/internal/controller/store_controller.go:132                                                                                                                                                                                                                                                              
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile                                                                                                                                                                                                                                                                                     
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119                                                                                                                                                                                                                                                       
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler                                                                                                                                                                                                                                                                              
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316                                                                                                                                                                                                                                                       
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem                                                                                                                                                                                                                                                                           
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266                                                                                                                                                                                                                                                       
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2                                                                                                                                                                                                                                                                                 
    /home/runner/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227                                                                                                                                                                                                                                                       
2024-11-23 12:31:03    [INFO]    Update store status    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "8407e3f2-d04c-494c-aca0-5d30fab0911d", "status": {"state":"setup
2024-11-23 12:31:03    [INFO]    my-shop    Do reconcile on store    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "4fcdcc02-a9de-4a6b-b347-93525cc532c6", "state": "s
2024-11-23 12:31:03    [INFO]    my-shop    reconcile app secrets    {"controller": "store", "controllerGroup": "shop.shopware.com", "controllerKind": "Store", "Store": {"name":"my-shop","namespace":"my-shop-namespace"}, "namespace": "my-shop-namespace", "name": "my-shop", "reconcileID": "4fcdcc02-a9de-4a6b-b347-93525cc532c6", "state": "s
2024-11-23 12:31:03    [DEBUG]    events    Update Store my-shop setup job in namespace my-shop-namespace. Diff hash    {"type": "Normal", "object": {"kind":"Store","namespace":"my-shop-namespace","name":"my-shop","uid":"0cd29409-5b25-4e5e-ac71-89e86a48dc7b","apiVersion":"shop.shopware.com/v1","resourceVersion":"511498361"}, "reason": "Dif

Your custom resource

# configured through helm chart shopware
# values:
percona:
  enabled: false

minio:
  enabled: false

store:
  host: shop...
  serviceAccountName: null

  disableJobDeletion: true

  network:
    enabled: false

  database:
    user: 'root'
    host: 'my-shop-mariadb'
    passwordSecretRef:
      name: 'my-shop-mariadb'
      key: 'mariadb-root-password'

  s3Storage:
    endpointURL: https://s3...
    accessKeyRef:
      name: s3-credentials
      key: AWS_ACCESS_KEY
    secretAccessKeyRef:
      name: s3-credentials
      key: AWS_SECRET_KEY
TrayserCassa commented 1 day ago

Hi thank you for the issue 👍

This happens when the job resource is modified from other operators or in some way. That's mostly true if you see the diff hash log/event. 

Update Store my-shop setup job in namespace my-shop-namespace. Diff hash

If the job resource is not the same as it was before, we delete the job and recreate it. Because jobs can't be patched and so we delete and recreate it.

The first issue 

job not found in container: shopware-setup

happens when the setup job is found but the name of the setup container inside the job is not shopware-setup. You can't modify the name, because it's hard coded in the code so I guess the issue is a timing issue with the api and the operator.

We can fix this in the operator by ignoring fields for the hash generation. So if you can tell me what get's attached after the job is created, I can fix it. Annotations should be fine, but labels for example would trigger a recreate of the job.