Terraform plan provider credentials error (AWS)

Terraform plan refuses to work following a successful terraform init.

My init and plan steps as follows:

    - task: TerraformCLI@0
      name: TerraformInit
      displayName: Terraform init
      inputs:
        command: 'init'
        workingDirectory: 'Terraform/environments/staging/eu-west-2'
        backendType: 'aws'
        allowTelemetryCollection: false
        backendServiceAws: 'ScreeningSwitchAccount'
        backendAwsRegion: eu-west-2

    - task: TerraformCLI@0
      name: TerraformPlan
      displayName: Terraform plan
      inputs:
        command: 'plan'
        commandOptions: '-out=tfplan -var-file $(Pipeline.Workspace)/testbuild/tfvars-artifact/ami-build.tfvars -target aws_autoscaling_group.frontend -target aws_launch_template.frontend -target aws_autoscaling_group.worker -target aws_launch_template.worker'
        workingDirectory: 'Terraform/environments/staging/eu-west-2'
        publishPlanResults: 'tfplan'
        allowTelemetryCollection: false
        backendServiceAws: 'ScreeningSwitchAccount'
        backendAwsRegion: eu-west-2

And my terraform config as follows:

terraform {
  required_version = "~> 1.4"

  backend "s3" {
    bucket         = "screening-staging-eu-west-2-tf-state"
    dynamodb_table = "screening-staging-eu-west-2-tf-state"
    encrypt        = true
    key            = "staging/main.tfstate"
    region         = "eu-west-2"
    role_arn       = "arn:aws:iam::449762975869:role/OrganizationAccountAccessRole"
  }

  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5.1.0" # latest 5.1.x 
    }
  }
}

provider "aws" {
  region  = "eu-west-2"
  assume_role {
    role_arn = "arn:aws:iam::XXXX:role/YYYY"
  }
}

The init succeeds and the error comes at the plan stage. If I replace the TerraformCLI plan step with an AWSShellScript that runs terraform plan (with the same arguments), it works ok with no error so I think is something to do with the TerraformCLI task?

I think this was working fine with older terraform provider version (3.x), could be related to new requirements around provider credentials introduced for 4.x? https://registry.terraform.io/providers/hashicorp/aws/latest/docs/guides/version-4-upgrade#changes-to-authentication

jason-johnson / azure-pipelines-tasks-terraform

Terraform plan provider credentials error (AWS) #349