search

adaptiveteam / adaptive

Umbrella repo for Adaptive library code
2 stars 0 forks source link

`test-with-localstack` fails on local machine #358

Open Primetalk opened 4 years ago

Primetalk commented 4 years ago 
module.user_engagement_scheduler_lambda.aws_cloudwatch_event_rule.rule[0]: Still destroying... [id=test_user_engagement_scheduler_lambda_scheduled_run, 4m50s elapsed]
module.user_profile_lambda.aws_cloudwatch_event_rule.rule[0]: Still destroying... [id=test_user_profile_every_5_min, 5m0s elapsed]
module.user_objectives_lambda.aws_cloudwatch_event_rule.rule[0]: Still destroying... [id=test_user_objectives_lambda_warmer, 5m0s elapsed]
module.community_slack_message_processor_lambda.aws_cloudwatch_event_rule.rule[0]: Still destroying... [id=test_community_slack_message_processor_lambda_warmer, 5m0s elapsed]
module.user_profile_lambda.aws_cloudwatch_event_rule.rule[0]: Still destroying... [id=test_user_profile_every_5_min, 5m10s elapsed]

Error: error deleting CloudWatch Event Rule (test_user_objectives_lambda_warmer): ValidationException: Rule can't be deleted since it has targets.
    status code: 400, request id: e44170f7-0ce6-49c7-b642-2cd35cfc4a7a

Error: error deleting CloudWatch Event Rule (test_user_profile_every_5_min): ValidationException: Rule can't be deleted since it has targets.
    status code: 400, request id: 51d5db28-7af6-4151-aeaf-e55e3e38f201

Error: error deleting CloudWatch Event Rule (test_user_engagement_scheduler_lambda_scheduled_run): ValidationException: Rule can't be deleted since it has targets.
    status code: 400, request id: bd1f1c97-c395-41bf-bd6a-f262e9f7fd5e

Error: error deleting CloudWatch Event Rule (test_community_slack_message_processor_lambda_warmer): ValidationException: Rule can't be deleted since it has targets.
    status code: 400, request id: 25cc5e13-a375-42db-9020-77b75fc99f9e

Rules are used in schedule. And terraform cannot destroy rules due to the above issue.

Primetalk commented 4 years ago

Though,

$ terraform state list
module.community_slack_message_processor_lambda.aws_cloudwatch_event_rule.rule[0]
module.user_engagement_scheduler_lambda.aws_cloudwatch_event_rule.rule[0]
module.user_objectives_lambda.aws_cloudwatch_event_rule.rule[0]
module.user_profile_lambda.aws_cloudwatch_event_rule.rule[0]

Looks like a bug in terraform

Primetalk commented 4 years ago

After switching to use localstack (It seems that we hadn't used it before...)

TestBooks 2020-05-15T12:16:17+03:00 command.go:168: module.api.aws_api_gateway_account.demo: Refreshing state... [id=api-gateway-account]
coverage: [no statements]
panic: test timed out after 30m0s
Primetalk commented 4 years ago

Hangs on

TestBooks 2020-05-15T14:11:05+03:00 command.go:168: module.api.aws_api_gateway_account.demo: Refreshing state... [id=api-gateway-account]
Primetalk commented 4 years ago 
TestBooks 2020-05-15T20:13:00+03:00 command.go:168: module.reporting_db.module.db_instance.aws_db_instance.this[0]: Creating...
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: 
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: Error: Provider produced inconsistent result after apply
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: 
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: When applying changes to aws_dynamodb_table.user_communities, provider
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: "registry.terraform.io/-/aws" produced an unexpected new value for was
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: present, but now absent.
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: 
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: This is a bug in the provider, which should be reported in the provider's own
TestBooks 2020-05-15T20:13:04+03:00 command.go:168: issue tracker.

It tries to create RDS tables in the real us-west-2 region. Localstack doesn't support creating RDS instances locally.

Per Localstack Pro pricing there is a support for RDS in Pro version.

This means that we might have to either upgrade to Localstack Pro, or get rid of localstack altogether.

Primetalk commented 4 years ago

Discussion:

Primetalk commented 4 years ago

Localstack Pro docs/RDS:

Relational Database Service (RDS)

LocalStack supports a basic version of RDS for testing. Currently, it is possible to spin up PostgreSQL databases on the local machine; support for MySQL and other DB engines is under development and coming soon.

@ctcreel, @sunnylbk We might want to change our RDS to PostgreSQL

Primetalk commented 4 years ago 
TestBooks 2020-05-20T12:00:14+03:00 command.go:168: Error: error deleting CloudWatch Event Rule (test_user_profile_every_5_min): ValidationException: Rule can't be deleted since it has targets.
...
TestBooks 2020-05-20T12:00:14+03:00 command.go:168: Error: Error deleting IAM Role (test_strategy-slack-message-processor-lambda-go): DeleteConflict: Cannot delete entity, must detach all policies first.
...

Looks like terraform has some issues and does not know a correct order of resource destruction. Terraform previously had similar issue: https://github.com/hashicorp/terraform/issues/2761

Primetalk commented 4 years ago

Manually deleted the resources that terraform could not destroy.

Primetalk commented 4 years ago 
Error creating EIP: AddressLimitExceeded: The maximum number of addresses has been reached
Primetalk commented 4 years ago

After removing old IP addresses it's finally completed. We still have some old Elastic IPs: https://us-west-2.console.aws.amazon.com/vpc/home?region=us-west-2#Addresses:

Primetalk commented 4 years ago 
Plan: 0 to add, 0 to change, 370 to destroy.
Primetalk commented 4 years ago 
Destroy complete! Resources: 370 destroyed.
Plan: 370 to add, 0 to change, 0 to destroy.
Error creating EIP: AddressLimitExceeded

Looks like it does not remove Elastic IPs automatically.

Primetalk commented 4 years ago

terraform apply/destroy works on local machine and deploy to real AWS. This process does not look very reliable. If it failed for some reason, the infrastructure is left in uncertain intermediate state and requires manual cleanup. (I guess, it's because terraform state is not transactional. See an interesting discussion here.)

With Localstack there is another issue - Elastic IP doesn't seem to be supported. This means that even if we create most of our infrastructure in localstack, some of it will be deployed to the real AWS. And probably elastic IP won't work/fail because AWS wouldn't know about the other parts of infrastructure.

This makes me think that end-2-end automatic test of our infrastructure is not possible at the moment.

What we could think instead is split our infrastructure into a few modules that we can deploy/destroy independently and then create a few tests that will be testing some of the modules. And Elastic IP will be left untested.