Clouddriver fails on startup: Destroy method on bean with name 'operationsController' threw an exception: java.lang.IllegalArgumentException: Field (null) is not contained in Row

andrijaperovic commented 2 years ago

Issue Summary:

Spinnaker clouddriver fails on startup: 2022-06-23 21:43:19.462 WARN 1 --- [extShutdownHook] .s.c.a.CommonAnnotationBeanPostProcessor : Destroy method on bean with name 'operationsController' threw an exception: java.lang.IllegalArgumentException: Field (null) is not contained in Row ("task_states"."id", "task_states"."task_id", "task_states"."created_at", "task_states"."state", "task_states"."phase", "task_states"."status", "task_states"."task_id", "task_states"."created", "tasks"."id", "tasks"."request_id", "tasks"."owner_id", "tasks"."created_at", "tasks"."saga_ids")

Cloud Provider(s):

Azure (AKS v1.22.6)

Environment:

Running Spinnaker on top of AKS using OSS spinnaker operator: https://github.com/armory/spinnaker-operator/releases/tag/v1.3.0-rc.19

Feature Area:

Clouddriver

Description:

Clouddriver should boot up successfully instead of going into CrashLoopBackOff.

Steps to Reproduce:

Install armory/spinnaker-operator and apply SpinnakerService CRD:

apiVersion: spinnaker.io/v1alpha2
kind: SpinnakerService
metadata:
  name: spinnaker
  namespace: spinnaker
spec:
  expose:
    service:
      type: ClusterIP
    type: service
  kustomize:
    clouddriver: {}
    deck: {}
    echo: {}
    front50: {}
    gate: {}
    igor: {}
    orca: {}
    rosco: {}
  spinnakerConfig:
    config:
      artifacts:
        s3:
          accounts:
          - awsAccessKeyId: awsAccessKeyId
            awsSecretAccessKey: awsSecretAccessKey
            name: name
            region: region
          enabled: true
      canary: {}
      ci: {}
      deploymentEnvironment: {}
      features: {}
      metricStores: {}
      notifications: {}
      persistentStorage:
        persistentStoreType: s3
        s3:
          accessKeyId: accessKeyId
          bucket: bucket
          region: region
          rootFolder: front50
          secretAccessKey: secretAccessKey
      providers:
        kubernetes:
          accounts:
          - cacheThreads: 1
            cachingPolicies: []
            checkPermissionsOnStartup: true
            configureImagePullSecrets: true
            customResources: []
            dockerRegistries: []
            kinds: []
            kubeconfigFile: kubeconfigFile
            liveManifestCalls: false
            name: name
            namespaces: []
            oAuthScopes: []
            omitKinds: []
            omitNamespaces: []
            onlySpinnakerManaged: true
            permissions: {}
            providerVersion: V2
          enabled: true
          primaryAccount: primaryAccount
      pubsub:
        enabled: true
      repository: {}
      security:
        apiSecurity:
          overrideBaseUrl: overrideBaseUrl
        authn:
          enabled: true
          oauth2:
            client:
              accessTokenUri: accessTokenUri
              clientId: clientId
              clientSecret: clientSecret
              scope: user:email
              userAuthorizationUri: userAuthorizationUri
            enabled: true
            provider: OTHER
            resource:
              userInfoUri: userInfoUri
            userInfoMapping:
              email: email
              firstName: name
              lastName: ""
              username: login
        authz:
          enabled: true
          groupMembership:
            github:
              accessToken: accessToken
              baseUrl: baseUrl
              organization: organization
              roleProviderType: GITHUB
            service: GITHUB
        uiSecurity:
          overrideBaseUrl: overrideBaseUrl
      spinnaker: {}
      version: 1.27.0
    files:
      kubeconfigFile: ""
    profiles:
      clouddriver:
        redis:
          cache:
            enabled: false
          enabled: false
          scheduler:
            enabled: false
          taskRepository:
            enabled: false
        sql:
          cache:
            enabled: true
            readBatchSize: 500
            writeBatchSize: 300
          connectionPools:
            default:
              default: true
              jdbcUrl: jdbcUrl
              password: password
              user: user
            tasks:
              jdbcUrl: jdbcUrl
              password: password
              user: user
          enabled: true
          migration:
            jdbcUrl: jdbcUrl
            password: password
            user: user
          scheduler:
            enabled: true
          taskRepository:
            enabled: true
      echo:
        logging:
          level:
            com.netflix.spinnaker.echo.pubsub.aws: DEBUG
        pubsub:
          amazon:
            enabled: true
            subscriptions:
            - messageFormat: S3
              name: name
              queueARN: queueARN
              topicARN: topicARN
          enabled: true
      front50: {}
      gate: {}
      igor: {}
      orca:
        executionRepository:
          redis:
            enabled: false
          sql:
            enabled: true
        keiko:
          queue:
            redis:
              enabled: false
            sql:
              enabled: true
        monitor:
          activeExecutions:
            redis: false
        queue:
          pendingExecutionService:
            redis:
              enabled: false
            sql:
              enabled: true
          zombieCheck:
            enabled: true
        sql:
          connectionPool:
            connectionTimeout: 5000
            jdbcUrl: jdbcUrl
            maxLifetime: 30000
            maxPoolSize: 50
            password: password
            user: user
          enabled: true
          migration:
            jdbcUrl: jdbcUrl
            password: password
            user: user
      rosco: {}
      settings-local.js: {}
      spinnaker: {}
      spinnakerconfig: {}
    service-settings:
      clouddriver: {}
      echo:
        env:
          AWS_ACCESS_KEY_ID: AWS_ACCESS_KEY_ID
          AWS_SECRET_KEY: AWS_SECRET_KEY
      front50: {}
      gate: {}
      igor: {}
      orca: {}
      rosco: {}
      spinnaker: {}
  validation:
    providers:
      kubernetes:
        enabled: true

Additional Details:

orca also goes into CrashLoopBackOff and fiat fails because it cannot connect to clouddriver.

andrijaperovic commented 2 years ago

@ichi0915 @dogonthehorizon filing a ticket in this repo as well in case it needs to be tracked independently: https://github.com/spinnaker/spinnaker/issues/6679

andrijaperovic commented 2 years ago

Once I set clouddriver.sql.taskRepository to enabled: false the error seems to go away. Also, looks like the CrashLoopBackOff itself might have been linked to lack of compatibility of startupProbe and readinessProbe in the deployments of clouddriver & orca to k8s v1.22 (have removed those for now, I believe Armory team is looking at how these need to be addressed).

armory / spinnaker-operator