AWS Provider Endpoints not working on 3.64.1

gombosg commented 2 years ago

Community Note

Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment

Terraform CLI and Terraform AWS Provider Version

Terraform v1.0.10 on linux_amd64

provider registry.terraform.io/hashicorp/aws v3.64.1

Affected Resource(s)

aws

Terraform Configuration Files

Please include all Terraform configurations required to reproduce the bug. Bug reports without a functional reproduction may be closed without investigation.

provider "aws" {
  access_key                  = "localstack-key"
  region                      = "us-west-2"
  s3_force_path_style         = true
  secret_key                  = "localstack-key"
  skip_credentials_validation = true
  skip_metadata_api_check     = true
  skip_requesting_account_id  = true

  endpoints {
    apigateway     = "http://localhost:4566"
    cloudformation = "http://localhost:4566"
    cloudwatch     = "http://localhost:4566"
    dynamodb       = "http://localhost:4566"
    es             = "http://localhost:4566"
    firehose       = "http://localhost:4566"
    iam            = "http://localhost:4566"
    kinesis        = "http://localhost:4566"
    lambda         = "http://localhost:4566"
    route53        = "http://localhost:4566"
    redshift       = "http://localhost:4566"
    s3             = "http://localhost:4566"
    secretsmanager = "http://localhost:4566"
    ses            = "http://localhost:4566"
    sns            = "http://localhost:4566"
    sqs            = "http://localhost:4566"
    ssm            = "http://localhost:4566"
    stepfunctions  = "http://localhost:4566"
    sts            = "http://localhost:4566"
  }
}

Debug Output

https://gist.github.com/gombosg/4b39279cced18b0f60fc1a16e33cf96f

Expected Behavior

Works

Actual Behavior

Error

Steps to Reproduce

terraform validate

Important Factoids

Works in 3.63.0.

References

https://github.com/hashicorp/terraform-provider-aws/issues/21637

ewbankkit commented 2 years ago

╷
│ Error: Unsupported argument
│ 
│   on localstack.tf line 27, in provider "aws":
│   27:     es             = "http://localhost:4566"
│ 
│ An argument named "es" is not expected here. Did you mean "fsx"?
╵
╷
│ Error: Unsupported argument
│ 
│   on localstack.tf line 40, in provider "aws":
│   40:     stepfunctions  = "http://localhost:4566"
│ 
│ An argument named "stepfunctions" is not expected here.
╵

ewbankkit commented 2 years ago

This is the prior to #20000 / #21306 (v3.63.0 and earlier) set of attribute names for the provider's endpoints configuration block:

https://github.com/hashicorp/terraform-provider-aws/blob/fce7062f70caa92a78bca629ffe441c088456418/aws/provider.go#L1375-L1538

and this is the new (v3.64.1 and later) set:

https://github.com/hashicorp/terraform-provider-aws/blob/77bad27d9f23679e3e2e56046030c24043969a19/internal/conns/conns.go#L288-L561

Although the documentation lists alternatives for some services (e.g. es or stepfunctions), we are only creating attribute names for the canonical name of the service.

gombosg commented 2 years ago

Thanks! Is this a bug or intended functionality (i.e. a breaking config change)?

ewbankkit commented 2 years ago

@gombosg Definitely a bug - we only (intend to) make breaking changes at a major version and v4.0.0 is scheduled for later this year. Even then we would have to go through a deprecation cycle before removing such significant attributes.

ewbankkit commented 2 years ago

package main

import (
    "fmt"
)

func main() {
    old := []string{
        "accessanalyzer",
        "acm",
        "acmpca",
        "amplify",
        "apigateway",
        "appconfig",
        "applicationautoscaling",
        "applicationinsights",
        "appmesh",
        "apprunner",
        "appstream",
        "appsync",
        "athena",
        "auditmanager",
        "autoscaling",
        "autoscalingplans",
        "backup",
        "batch",
        "budgets",
        "chime",
        "cloud9",
        "cloudcontrolapi",
        "cloudformation",
        "cloudfront",
        "cloudhsm",
        "cloudsearch",
        "cloudtrail",
        "cloudwatch",
        "cloudwatchevents",
        "cloudwatchlogs",
        "codeartifact",
        "codebuild",
        "codecommit",
        "codedeploy",
        "codepipeline",
        "codestarconnections",
        "cognitoidentity",
        "cognitoidp",
        "configservice",
        "connect",
        "cur",
        "dataexchange",
        "datapipeline",
        "datasync",
        "dax",
        "detective",
        "devicefarm",
        "directconnect",
        "dlm",
        "dms",
        "docdb",
        "ds",
        "dynamodb",
        "ec2",
        "ecr",
        "ecrpublic",
        "ecs",
        "efs",
        "eks",
        "elasticache",
        "elasticbeanstalk",
        "elastictranscoder",
        "elb",
        "emr",
        "emrcontainers",
        "es",
        "firehose",
        "fms",
        "forecast",
        "fsx",
        "gamelift",
        "glacier",
        "globalaccelerator",
        "glue",
        "greengrass",
        "guardduty",
        "iam",
        "identitystore",
        "imagebuilder",
        "inspector",
        "iot",
        "iotanalytics",
        "iotevents",
        "kafka",
        "kinesis",
        "kinesisanalytics",
        "kinesisanalyticsv2",
        "kinesisvideo",
        "kms",
        "lakeformation",
        "lambda",
        "lexmodels",
        "licensemanager",
        "lightsail",
        "location",
        "macie",
        "macie2",
        "managedblockchain",
        "marketplacecatalog",
        "mediaconnect",
        "mediaconvert",
        "medialive",
        "mediapackage",
        "mediastore",
        "mediastoredata",
        "memorydb",
        "mq",
        "mwaa",
        "neptune",
        "networkfirewall",
        "networkmanager",
        "opsworks",
        "organizations",
        "outposts",
        "personalize",
        "pinpoint",
        "pricing",
        "qldb",
        "quicksight",
        "ram",
        "rds",
        "redshift",
        "resourcegroups",
        "resourcegroupstaggingapi",
        "route53",
        "route53domains",
        "route53recoverycontrolconfig",
        "route53recoveryreadiness",
        "route53resolver",
        "s3",
        "s3control",
        "s3outposts",
        "sagemaker",
        "schemas",
        "sdb",
        "secretsmanager",
        "securityhub",
        "serverlessrepo",
        "servicecatalog",
        "servicediscovery",
        "servicequotas",
        "ses",
        "shield",
        "signer",
        "sns",
        "sqs",
        "ssm",
        "ssoadmin",
        "stepfunctions",
        "storagegateway",
        "sts",
        "swf",
        "synthetics",
        "timestreamwrite",
        "transfer",
        "waf",
        "wafregional",
        "wafv2",
        "worklink",
        "workmail",
        "workspaces",
        "xray",
    }

    new := []string{
        "accessanalyzer",
        "acm",
        "acmpca",
        "alexaforbusiness",
        "amplify",
        "amplifybackend",
        "apigateway",
        "apigatewayv2",
        "appautoscaling",
        "appconfig",
        "appflow",
        "appintegrations",
        "applicationcostprofiler",
        "applicationdiscovery",
        "applicationinsights",
        "appmesh",
        "appregistry",
        "apprunner",
        "appstream",
        "appsync",
        "athena",
        "auditmanager",
        "augmentedairuntime",
        "autoscaling",
        "autoscalingplans",
        "backup",
        "batch",
        "braket",
        "budgets",
        "chime",
        "cloud9",
        "cloudcontrol",
        "clouddirectory",
        "cloudformation",
        "cloudfront",
        "cloudhsm",
        "cloudhsmv2",
        "cloudsearch",
        "cloudsearchdomain",
        "cloudtrail",
        "cloudwatch",
        "cloudwatchevents",
        "cloudwatchlogs",
        "codeartifact",
        "codebuild",
        "codecommit",
        "codedeploy",
        "codeguruprofiler",
        "codegurureviewer",
        "codepipeline",
        "codestar",
        "codestarconnections",
        "codestarnotifications",
        "cognitoidentity",
        "cognitoidp",
        "cognitosync",
        "comprehend",
        "comprehendmedical",
        "computeoptimizer",
        "config",
        "connect",
        "connectcontactlens",
        "connectparticipant",
        "costexplorer",
        "cur",
        "customerprofiles",
        "dataexchange",
        "datapipeline",
        "datasync",
        "dax",
        "detective",
        "devicefarm",
        "devopsguru",
        "directconnect",
        "dlm",
        "dms",
        "docdb",
        "ds",
        "dynamodb",
        "dynamodbstreams",
        "ec2",
        "ec2instanceconnect",
        "ecr",
        "ecrpublic",
        "ecs",
        "efs",
        "eks",
        "elasticache",
        "elasticbeanstalk",
        "elasticinference",
        "elasticsearch",
        "elastictranscoder",
        "elb",
        "elbv2",
        "emr",
        "emrcontainers",
        "finspace",
        "finspacedata",
        "firehose",
        "fis",
        "fms",
        "forecast",
        "forecastquery",
        "frauddetector",
        "fsx",
        "gamelift",
        "glacier",
        "globalaccelerator",
        "glue",
        "gluedatabrew",
        "greengrass",
        "greengrassv2",
        "groundstation",
        "guardduty",
        "health",
        "healthlake",
        "honeycode",
        "iam",
        "identitystore",
        "imagebuilder",
        "inspector",
        "iot",
        "iot1clickdevices",
        "iot1clickprojects",
        "iotanalytics",
        "iotdataplane",
        "iotdeviceadvisor",
        "iotevents",
        "ioteventsdata",
        "iotfleethub",
        "iotjobsdataplane",
        "iotsecuretunneling",
        "iotsitewise",
        "iotthingsgraph",
        "iotwireless",
        "ivs",
        "kafka",
        "kendra",
        "kinesis",
        "kinesisanalytics",
        "kinesisanalyticsv2",
        "kinesisvideo",
        "kinesisvideoarchivedmedia",
        "kinesisvideomedia",
        "kinesisvideosignalingchannels",
        "kms",
        "lakeformation",
        "lambda",
        "lexmodelbuilding",
        "lexmodelsv2",
        "lexruntime",
        "lexruntimev2",
        "licensemanager",
        "lightsail",
        "location",
        "lookoutequipment",
        "lookoutforvision",
        "lookoutmetrics",
        "machinelearning",
        "macie",
        "macie2",
        "managedblockchain",
        "marketplacecatalog",
        "marketplacecommerceanalytics",
        "marketplaceentitlement",
        "marketplacemetering",
        "mediaconnect",
        "mediaconvert",
        "medialive",
        "mediapackage",
        "mediapackagevod",
        "mediastore",
        "mediastoredata",
        "mediatailor",
        "memorydb",
        "mgn",
        "migrationhub",
        "migrationhubconfig",
        "mobile",
        "mobileanalytics",
        "mq",
        "mturk",
        "mwaa",
        "neptune",
        "networkfirewall",
        "networkmanager",
        "nimblestudio",
        "opsworks",
        "opsworkscm",
        "organizations",
        "outposts",
        "personalize",
        "personalizeevents",
        "personalizeruntime",
        "pi",
        "pinpoint",
        "pinpointemail",
        "pinpointsmsvoice",
        "polly",
        "pricing",
        "prometheus",
        "proton",
        "qldb",
        "qldbsession",
        "quicksight",
        "ram",
        "rds",
        "rdsdata",
        "redshift",
        "redshiftdata",
        "rekognition",
        "resourcegroups",
        "resourcegroupstagging",
        "robomaker",
        "route53",
        "route53domains",
        "route53recoverycontrolconfig",
        "route53recoveryreadiness",
        "route53resolver",
        "s3",
        "s3control",
        "s3outposts",
        "sagemaker",
        "sagemakeredgemanager",
        "sagemakerfeaturestoreruntime",
        "sagemakerruntime",
        "savingsplans",
        "schemas",
        "secretsmanager",
        "securityhub",
        "serverlessapprepo",
        "servicecatalog",
        "servicediscovery",
        "servicequotas",
        "ses",
        "sesv2",
        "sfn",
        "shield",
        "signer",
        "simpledb",
        "sms",
        "snowball",
        "sns",
        "sqs",
        "ssm",
        "ssmcontacts",
        "ssmincidents",
        "sso",
        "ssoadmin",
        "ssooidc",
        "storagegateway",
        "sts",
        "support",
        "swf",
        "synthetics",
        "textract",
        "timestreamquery",
        "timestreamwrite",
        "transcribe",
        "transcribestreaming",
        "transfer",
        "translate",
        "waf",
        "wafregional",
        "wafv2",
        "wellarchitected",
        "workdocs",
        "worklink",
        "workmail",
        "workmailmessageflow",
        "workspaces",
        "xray",
    }

    for _, o := range old {
        found := false

        for _, n := range new {
            if o == n {
                found = true
                break
            }
        }

        if !found {
            fmt.Printf("%q not found in new endpoint list\n", o)
        }
    }
}

"applicationautoscaling" not found in new endpoint list
"cloudcontrolapi" not found in new endpoint list
"configservice" not found in new endpoint list
"es" not found in new endpoint list
"lexmodels" not found in new endpoint list
"resourcegroupstaggingapi" not found in new endpoint list
"sdb" not found in new endpoint list
"serverlessrepo" not found in new endpoint list
"stepfunctions" not found in new endpoint list

YakDriver commented 2 years ago

This should only affect the subset of endpoints whose HCL "key" differs from the service package name. However, it is a bug and I'm working on a fix now.

lorengordon commented 2 years ago

🤞 hoping for a patch release for this one!

YakDriver commented 2 years ago

We apologize for this inconvenience! A point release has been approved.

github-actions[bot] commented 2 years ago

This functionality has been released in v3.64.2 of the Terraform AWS Provider. Please see the Terraform documentation on provider versioning or reach out if you need any assistance upgrading.

For further feature requests or bug reports with this functionality, please create a new GitHub issue following the template. Thank you!

gombosg commented 2 years ago

New version 3.64.2 works fine, thank you so much for fixing this so fast! 🚀

YakDriver commented 2 years ago

@gombosg Thank you for reporting back!

github-actions[bot] commented 2 years ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-aws