bug: Aurora Global DB -- Secondary Cluster Promotion

lek7hj commented 8 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

On a global aurora db set up with a primary cluster and a secondary cluster, running awslocal rds promote-read-replica-db-cluster --db-cluster-identifier test-db-aurora-us-west-2 --debug results in an internal error, and fails to promote the secondary cluster.

Expected Behavior

The secondary cluster should be promoted to a standalone cluster.

How are you starting LocalStack?

With a docker-compose file

Steps To Reproduce

How are you starting localstack (e.g., `bin/localstack` command, arguments, or `docker-compose.yml`)

docker-compose.yml

version: '3.8'

services:
  localstack:
    image: localstack/localstack-pro:3.0.2
    ports:
      - "127.0.0.1:4566:4566"            # LocalStack Gateway
      - "127.0.0.1:4510-4559:4510-4559"  # external services port range
      - "127.0.0.1:53:53"                # DNS config (required for Pro)
      - "127.0.0.1:443:443"              # LocalStack HTTPS Gateway (required for Pro)
      - "127.0.0.1:53:53/udp"
    environment:
      - SERVICES=s3,sns,dynamodb,ecr,lambda,cloudformation,cloudwatch,sts,iam,apigateway,route53,elbv2,ecs,ec2,logs,events,apigatewayv2,application-autoscaling,cloudtrail,rds,kms
      - PERSISTENCE=0
      - PORT_WEB_UI=18080
      - DOCKER_HOST=unix:///var/run/docker.sock
      - AWS_DEFAULT_REGION=us-east-1
      - SKIP_SSL_CERT_DOWNLOAD=1
      - OUTBOUND_HTTP_PROXY=${HTTP_PROXY}
      - OUTBOUND_HTTPS_PROXY=${HTTP_PROXY}
      - LAMBDA_RUNTIME_ENVIRONMENT_TIMEOUT=60
    volumes:
      - "../../shared/localstack:/var/lib/localstack"
      - "/var/run/docker.sock:/var/run/docker.sock"      
      - "./init-npm.sh:/etc/localstack/init/boot.d/init-npm.sh"
    networks:
      - "local"

networks:
  local:
    driver: "bridge"

exec docker-compose -f ./tests/features/docker-compose.yml -p localstack up -d;

Client commands (e.g., AWS SDK code snippet, or sequence of "awslocal" commands)

awslocal cloudformation create-stack --stack-name aurora-db --template-body file://tests/features/templates/aurora/aurora-global.yml --region us-east-1

awslocal cloudformation create-stack --stack-name aurora-west --template-body file://tests/features/templates/aurora/aurora-west.yml --region us-west-2

Environment

- OS: macOS Sonoma 14.4
- LocalStack: localstack/localstack-pro:3.0.2

Anything else?

No response

localstack-bot commented 8 months ago

Welcome to LocalStack! Thanks for reporting your first issue and our team will be working towards fixing the issue for you or reach out for more background information. We recommend joining our Slack Community for real-time help and drop a message to LocalStack Pro Support if you are a Pro user! If you are willing to contribute towards fixing this issue, please have a look at our contributing guidelines and our contributing guide.

lek7hj commented 8 months ago

aurora global yaml

AWSTemplateFormatVersion: '2010-09-09'
Description: 
  A stack to create Aurora DB global clusters.
Resources:
  GlobalCluster:
    Type: 'AWS::RDS::GlobalCluster'
    Properties:
      GlobalClusterIdentifier: test-db-aurora-global
      SourceDBClusterIdentifier: !Ref RDSCluster
  RDSCluster:
    Type: 'AWS::RDS::DBCluster'
    Properties:
      MasterUsername: very
      MasterUserPassword: notsecure
      DBClusterIdentifier: test-db-aurora-us-east-1
      DBClusterParameterGroupName: default.aurora-postgresql14
      Engine: aurora-postgresql
      EngineVersion: '14.8'
  RDSDBInstance1:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      Engine: aurora-postgresql
      DBClusterIdentifier: !Ref RDSCluster
      DBInstanceIdentifier: test-db-aurora-00
      DBParameterGroupName: default.aurora-postgresql14
      PubliclyAccessible: 'true'
      DBInstanceClass: db.r3.xlarge
  RDSDBInstance2:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      Engine: aurora-postgresql
      DBClusterIdentifier: !Ref RDSCluster
      DBInstanceIdentifier: test-db-aurora-01
      DBParameterGroupName: default.aurora-postgresql14
      PubliclyAccessible: 'true'
      DBInstanceClass: db.r3.xlarge

lek7hj commented 8 months ago

aurora west.yml

AWSTemplateFormatVersion: '2010-09-09'
Description: 
  A stack to add a region to the existing global cluster.
Resources:
  RDSCluster:
    Type: 'AWS::RDS::DBCluster'
    Properties:
      GlobalClusterIdentifier: test-db-aurora-global
      DBClusterIdentifier: test-db-aurora-us-west-2
      DBClusterParameterGroupName: default.aurora-postgresql14
      Engine: aurora-postgresql
      EngineVersion: '14.8'
  RDSDBInstance1:
    Type: 'AWS::RDS::DBInstance'
    Properties:
      Engine: aurora-postgresql
      DBClusterIdentifier: !Ref RDSCluster
      DBInstanceIdentifier: test-db-aurora-00
      DBParameterGroupName: default.aurora-postgresql14
      PubliclyAccessible: 'true'
      DBInstanceClass: db.r3.xlarge

steffyP commented 8 months ago

Hi @lek7hj,

thanks for the report! Could you give us some more details on your use case?

For the current implementation of global clusters we do have some limitations, that are outlined in the docs.

One of them includes:

[...] clusters removed from a global database lose their ability to function as standalone clusters, differing from their intended behavior on AWS.

Meaning with the current approach on LocalStack, we only simulate the creation of global clusters. We don't actually create several databases that are synced, rather we have on single database, and the endpoints of the clusters and instances all point to this single database. For the customers that initially requested this feature, this simulation was sufficient for their use case.

That's why we would like to learn a little bit more about your specific use case and how you intend to use the promoted and the original database afterwards. 🙏

lek7hj commented 8 months ago

Meaning with the current approach on LocalStack, we only simulate the creation of global clusters. We don't actually create several databases that are synced, rather we have on single database, and the endpoints of the clusters and instances all point to this single database. For the customers that initially requested this feature, this simulation was sufficient for their use case.

Hi @steffyP

We're currently using localstack to test a tool we've created for failing over RDS (along with some other services). Our tooling runs through the sequence of

promoting the secondary regional cluster
creating a new global cluster with the promoted cluster
add a read replica to the global cluster

We're looking to validate that these events happen (ex the west cluster gets removed from global, new global cluster gets created). We don't need the promoted database to be a new database with it's own unique endpoint.

What we do need is

for the promote promote-read-replica-db-cluster to remove the regional cluster from the global cluster (does not matter if the endpoint is the same, and is not actually functioning as a standalone)

localstack-bot commented 3 months ago

Hello 👋! It looks like this issue hasn’t been active in longer than five months. We encourage you to check if this is still an issue in the latest release. In the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or upvote with a reaction on the initial post to prevent automatic closure. If the issue is already closed, please feel free to open a new one.

localstack / localstack