unable to set some parameters in the DbClusterParameterGroup

[luis-alen]

Describe the bug

I'm unable to set parameters such as slow_query_log, long_query_time and log_queries_not_using_indexes in DbClusterParameterGroup objects. I get an unknown parameter error message:

Name:         luis
Namespace:    rc
Labels:       <none>
API Version:  rds.services.k8s.aws/v1alpha1
Kind:         DBClusterParameterGroup
Metadata:
  Creation Timestamp:  2023-07-13T18:40:54Z
  Finalizers:
    finalizers.rds.services.k8s.aws/DBClusterParameterGroup
  Generation:        7
  Resource Version:  593698696
  UID:               e62bcf3f-cbb9-482f-99d3-adb241686b24
Spec:
  Description:  luis aurora-mysql8.0 parameter group managed by ACK
  Family:       aurora-mysql8.0
  Name:         luis
  Parameter Overrides:
    character_set_client:      utf8mb4
    character_set_connection:  utf8mb4
    character_set_database:    utf8mb4
    character_set_results:     utf8mb4
    character_set_server:      utf8mb4
    collation_connection:      utf8mb4_unicode_ci
    collation_server:          utf8mb4_unicode_ci
    slow_query_log:            1
Status:
  Conditions:
    Message:               unknown parameter: slow_query_log
    Status:                True
    Type:                  ACK.Terminal
    Last Transition Time:  2023-07-14T13:02:08Z
    Message:               Resource not synced
    Reason:                resource is in terminal condition
    Status:                False
    Type:                  ACK.ResourceSynced
Events:                    <none>

I'm able to set parameters such as character_set_*, but the not the ones I mentioned above. I confirm that such parameters are available and are modifiable on the db cluster parameter group using the aws console.

I also confirm that it does work for the DbInstance kind. Maybe it's somehow related to https://github.com/aws-controllers-k8s/community/issues/869#issuecomment-1380728485?

Steps to reproduce Try to create a new DbClusterParameterGroup or try updating an existing one (with family aurora-mysql8.0) overwriting the slow_query_log parameter.

Expected outcome Parameter should be set accordingly.

Environment

• Kubernetes version 1.23
• Using EKS (yes/no), if so version?
• AWS service targeted (S3, RDS, etc.) RDS
• aws-rds-controller version v.1.1.2

[a-hilaly]

Hi @luis-alen , the rds-controller uses a kinda specific cache implementation inside the controller. Can you try to restart the controller? if the problem is solved them we might have an issue with the cache implementation :)

[luis-alen]

Hey, @a-hilaly. Unfortunately it didn't help. I turned on debug log level for the controller and I see that it captures the change, but is unable to apply it.

2023-07-25T10:18:39.211Z    DEBUG   ackrt   > r.Sync    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   >> r.resetConditions    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   << r.resetConditions    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   >> rm.ResolveReferences {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   << rm.ResolveReferences {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   >> rm.EnsureTags    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   << rm.EnsureTags    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   >> rm.ReadOne   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.211Z    DEBUG   ackrt   >>> rm.sdkFind  {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.410Z    DEBUG   ackrt   <<< rm.sdkFind  {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.411Z    DEBUG   ackrt   << rm.ReadOne   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.411Z    DEBUG   ackrt   >> r.updateResource {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.411Z    INFO    ackrt   desired resource state has changed  {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "diff": [{"Path":{"Parts":["Spec","ParameterOverrides"]},"A":{"character_set_client":"utf8mb4","character_set_connection":"utf8mb4","character_set_database":"utf8mb4","character_set_results":"utf8mb4","character_set_server":"utf8mb4","collation_connection":"utf8mb4_unicode_ci","collation_server":"utf8mb4_unicode_ci","slow_query_log":"1"},"B":{}}]}
2023-07-25T10:18:39.411Z    DEBUG   ackrt   >>> rm.Update   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.411Z    DEBUG   ackrt   >>>> rm.customUpdate    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.411Z    DEBUG   ackrt   >>>>> rm.syncParameters {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   >>>>>> rm.modifyParameters  {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   <<<<<< rm.modifyParameters  {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "error": "unknown parameter: slow_query_log"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   <<<<< rm.syncParameters {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "error": "unknown parameter: slow_query_log"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   <<<< rm.customUpdate    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "error": "unknown parameter: slow_query_log"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   <<< rm.Update   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "latest": {}, "error": "resource is in terminal condition"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   << r.updateResource {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "error": "resource is in terminal condition"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   >> r.ensureConditions   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   >>> rm.IsSynced {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   <<< rm.IsSynced {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   << r.ensureConditions   {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   < r.Sync    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "error": "resource is in terminal condition"}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   > r.patchResourceStatus {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.412Z    DEBUG   ackrt   >> kc.Patch (status)    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.511Z    DEBUG   ackrt   patched resource status {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7, "json": "{\"metadata\":{\"resourceVersion\":\"601235537\"},\"spec\":{\"tags\":null},\"status\":{\"conditions\":[{\"message\":\"unknown parameter: slow_query_log\",\"status\":\"True\",\"type\":\"ACK.Terminal\"},{\"lastTransitionTime\":\"2023-07-25T10:18:39Z\",\"message\":\"Resource not synced\",\"reason\":\"resource is in terminal condition\",\"status\":\"False\",\"type\":\"ACK.ResourceSynced\"}]}}"}
2023-07-25T10:18:39.511Z    DEBUG   ackrt   << kc.Patch (status)    {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}
2023-07-25T10:18:39.511Z    DEBUG   ackrt   < r.patchResourceStatus {"account": "xxxxxxxxxx", "role": "", "region": "eu-central-1", "kind": "DBClusterParameterGroup", "namespace": "rc", "name": "rc-aurora-mysql8-0", "is_adopted": false, "generation": 7}

[riddhi2910]

@a-hilaly I also verified the same with debug logs, and see that DbClusterParameterGroup object goes into not synced mode and throws an error

'unknown parameter: long_query_time', We also see the error on the binlog_format, and read_only kind parameters.

[riddhi2910]

Also, Debug logs don't show much information other than what is mentioned in the https://github.com/aws-controllers-k8s/community/issues/1847#issuecomment-1649556139

[riddhi2910]

@a-hilaly further investigating the issue:

the error is only coming for the parameters which are https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Reference.ParameterGroups.html#AuroraMySQL.Reference.Parameters.Instance instance level.

However, I need to check how the status is updated on DBClusterParameterGroup since it shows Instance instance level parameters in the status section.

[cogren1]

We're using controller version 1.1.10. We just encountered this with idle_in_transaction_session_timeout for Aurora Postgres. It worked fine for the instance parameter groups for our non-Aurora databases, but for the cluster parameter groups, we see the following:

    - message: 'unknown parameter: idle_in_transaction_session_timeout'
      status: 'True'
      type: ACK.Terminal

I removed this parameter from the spec, and restarted the controller, but the error still persists.

[riddhi2910]

@cogren1 https://github.com/aws-controllers-k8s/rds-controller/pull/167 will fix the issue that you are facing.

However please make sure, DbClusterParameterGroup should have Cluster Level Parameters https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/AuroraMySQL.Reference.ParameterGroups.html#AuroraMySQL.Reference.Parameters.Instance

[cogren1]

@riddhi2910 Thank you for linking that fix. I look forward to it getting in!

idle_in_transaction_session_timeout is a parameter that exists in cluster parameter groups for Aurora Postgres.

[riddhi2910]

@cogren1 Yeah that isn't very clear, the console UI can do much more than the API call to fetch the supported cluster-level parameters.

[salimadr]

Hello, I am experiencing the same problem, any solutions?

  conditions:
    - message: 'unknown parameter: max_connections'
      status: 'True'
      type: ACK.Terminal