SystemsManagerParameterStoreBackend unable to retrieve values

Checks

[X] I have checked for existing issues.
[X] This report is about the User-Community Airflow Helm Chart.

Chart Version

8.8.0

Kubernetes Version

Client Version: version.Info{Major:"1", Minor:"27", GitVersion:"v1.27.2", GitCommit:"7f6f68fdabc4df88cfea2dcf9a19b2b830f1e647", GitTreeState:"clean", BuildDate:"2023-05-17T14:20:07Z", GoVersion:"go1.20.4", Compiler:"gc", Platform:"darwin/arm64"}
Kustomize Version: v5.0.1
Server Version: version.Info{Major:"1", Minor:"29+", GitVersion:"v1.29.1-eks-b9c9ed7", GitCommit:"07600c74de018baffb16c82771a48adcb843a932", GitTreeState:"clean", BuildDate:"2024-03-02T03:46:35Z", GoVersion:"go1.21.6", Compiler:"gc", Platform:"linux/amd64"}

Helm Version

using terraform
provider registry.terraform.io/hashicorp/helm v2.12.1

Description

Note, I am using airflow 2.7.3-python3.10

I was trying to use AWS parameter store as the backend for my airflow variables but nothing is getting sync. I am following the documentation on AWS Connection Option 3

I attempted to use two different roles, one with only access with the necessary permissions including the correct namespace and the one with total admin access to my EKS cluster. Nothing is getting synced which leads me to believe this may be a bug? I can't say for certain because there does not seem to be any logs indicating something went wrong and I have no idea where to look. All pods are running fine.

Using airflow config list I am getting the correct arguments

[secrets]
backend = airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
backend_kwargs = {"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}

I have SSM vars that exist with the correct prefix airflow/variables/test but airflow variables list shows nothing

My service account exist and has the correct annotations and all my pods are assigned to this service account

eks.amazonaws.com/role-arn=arn:aws:iam::XXXXXXXXXX:role/<<MY_ROLE_NAME>>

I have no way of validating that airflow is able to connect to AWS Parameter store. Any help would be greatly appreciated. Also these same configurations worked with helm chart 2.5.3 and airflow 2.2.3

Relevant Logs

None, everything looks good

Custom Helm Values

airflow:
  image:
    repository: ${airflow_image}
    tag: "${airflow_version}"
    pullSecret: creds-secret
  executor: KubernetesExecutor
  connections:
  - id: default
    type: aws
    description: AWS connection
    extra: |
      {
        "region_name": "us-east-1"
      }
  config:
    AIRFLOW__SECRETS__BACKEND: airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
    AIRFLOW__SECRETS__BACKEND_KWARGS: '{"connections_prefix": "/airflow/connections", "variables_prefix": "/airflow/variables"}'
  serviceAccount:
    name: airflow
    annotations:
      eks.amazonaws.com/role-arn: "arn:aws:iam::XXXXXXXXXX:role/<<MY_ROLE_NAME>>" # My role - obviously

airflow-helm / charts