search

hashicorp / terraform-provider-aws

The AWS Provider enables Terraform to manage AWS resources.
https://registry.terraform.io/providers/hashicorp/aws
Mozilla Public License 2.0
9.8k stars 9.15k forks source link

[Bug]: RDS: ModifyDBInstance, unsupported protocol scheme "" on apply since 4.42.0 #28994

Open griggi-ws opened 1 year ago

griggi-ws commented 1 year ago

Terraform Core Version

0.13.7

AWS Provider Version

4.51.0,4.50.0,4.49.0,4.48.0,4.47.0,4.46.0,4.45.0,4.44.0,4.43.0,4.42.0

Affected Resource(s)

Expected Behavior

A change should have been applied (in this case, simply enablement of all cloudwatch log exports on a pair of RDS instances)

Actual Behavior

A non-specific error for "unsupported protocol scheme" upon apply

Relevant Error/Panic Output Snippet

Error: updating RDS DB Instance (productname-env-region): operation error RDS: ModifyDBInstance, unsupported protocol scheme ""

  on ../../../modules/aws/rds/main.tf line 278, in resource "aws_db_instance" "rds_instance":
 278: resource "aws_db_instance" "rds_instance" {

Error: updating RDS DB Instance (otherproductname-env-region): operation error RDS: ModifyDBInstance, unsupported protocol scheme ""

  on ../../../modules/aws/rds/main.tf line 278, in resource "aws_db_instance" "rds_instance":
 278: resource "aws_db_instance" "rds_instance" {

Terraform Configuration Files

I made a sanitization effort and encrypted with hashicorp public key as well, I don't believe I snipped anything critical but let me know what more you need. Unfortunately github isn't liking tgzs with encrypted contents, nor a tgz that was then encrypted, so remote host. https://drive.google.com/file/d/1CwwZCU_fG02e4fK2pMKUD1gQ4pBfJqyt/view?usp=sharing

Steps to Reproduce

Start with/create an RDS instance with some semblance of the following state:

# module.rds["client"].aws_db_instance.rds_instance[0]:
resource "aws_db_instance" "rds_instance" {  
    address                               = (sensitive value)
    allocated_storage                     = 300
    allow_major_version_upgrade           = false
    apply_immediately                     = false
    arn                                   = (sensitive value)
    auto_minor_version_upgrade            = true  
    availability_zone                     = "us-east-1a"
    backup_retention_period               = 7
    backup_window                         = "05:07-05:37"
    ca_cert_identifier                    = "rds-ca-2019"
    copy_tags_to_snapshot                 = false
    customer_owned_ip_enabled             = false
    db_subnet_group_name                  = "generic-subnet-group"
    delete_automated_backups              = true
    deletion_protection                   = true
    enabled_cloudwatch_logs_exports       = []
    endpoint                              = (sensitive value)
    engine                                = "mysql"
    engine_version                        = "5.7.38"                                                                                                                                                            
    engine_version_actual                 = "5.7.38"                                                                                                                                                        
    final_snapshot_identifier             = (sensitive value)
    hosted_zone_id                        = (sensitive value)                                                                                                                                                         
    iam_database_authentication_enabled   = false
    id                                    = (sensitive value)
    identifier                            = (sensitive value)
    instance_class                        = "db.m5.large"
    iops                                  = 0
    kms_key_id                            = (sensitive value)
    latest_restorable_time                = "2023-01-17T22:15:00Z"
    license_model                         = "general-public-license"
    maintenance_window                    = "sat:05:45-sat:06:15"
    max_allocated_storage                 = 0
    monitoring_interval                   = 0
    multi_az                              = false
    network_type                          = "IPV4"
    option_group_name                     = "default-mysql-options"
    parameter_group_name                  = "default-mysql-params"
    password                              = (sensitive value)
    performance_insights_enabled          = false
    performance_insights_retention_period = 0
    port                                  = 3306
    publicly_accessible                   = false
    replicas                              = []
    resource_id                           = (sensitive value)
    security_group_names                  = []
    skip_final_snapshot                   = false
    status                                = "available"
    storage_encrypted                     = true
    storage_throughput                    = 0
    storage_type                          = "standard"                                   
}

Change enabled_cloudwatch_logs_exports parameter to ["audit","error","general","slowquery"]

Apply

Debug Output

https://drive.google.com/file/d/1D6HOOEB232xP2wjkcoWbxRTKLzyl72f9/view?usp=share_link

Panic Output

No response

Important Factoids

Applies without issue with provider version 4.41.0. Each version since (up to and including 4.51.0) output the same error. After successfully updating cloudwatch exports with 4.41.0, I attempted to apply another change back on 4.50.0 to enable RDS performance insights, and that also failed with the same error. Reverting to 4.41.0 once again was successful.

Also worth noting, we are using the fips endpoints.

References

No response

Would you like to implement a fix?

None

github-actions[bot] commented 1 year ago

Community Note

Voting for Prioritization

Volunteering to Work on This Issue

AdamTylerLynch commented 1 year ago

@justinretzolk can you pull down the debug and terraform config files? They are encrypted using Hashicorp public keys.

@griggi-ws are you experiencing this issue in GovCloud, Commercial, or another partition? Can you specify which region you are using?

griggi-ws commented 1 year ago

This is in the standard commercial partition, utilizing the fips endpoints. It occurred across multiple regions, eu-central-1 and us-east-1 being the two I recall explicitly testing prior to rolling back to the earlier version. The debug output was from a run in us-east-1, and the config files are for the same.

verbecee commented 1 year ago

@AdamTylerLynch we are seeing this issue as well, in GovCloud in us-gov-west-1. We are on version 4.59.0 of the aws provider.

AdamTylerLynch commented 1 year ago

Upon source code review, there are 3 locations where this specific error message could be raised (not associated with Blue/Green Deployments). They all occur after an AWS API call to ModifyDBInstance().

My hypothesis is that a AWS server side validation routine is rejecting an input value. Most likely something defined in the CloudwatchLogsExportConfiguration attribute.

A few paths exist to get a root cause:

  1. If you have AWS Support, you can open a Support Chat/Ticket and request an investigation. Be specific that you are modifying an existing DBInstance via the AWS GOSDK and ModifyDBInstance() API call. The support team can look at the internals of the service to see what is being rejected. Let support know that they can engage awstyler@ for help internally.
  2. There should be additional details in your CloudTrail logs that might point at the root cause.
gdavison commented 10 months ago

Hi @griggi-ws, do you still see this error with a more recent version of the provider?

If so, how are you setting the FIPS endpoints? Are you using use_fips_endpoint on the provider configuration block, or setting the endpoint directly? If you're setting the endpoint directly, are you setting just the hostname, or a full URL?