search

MitocGroup / deepify

software development tool-chain that abstracts package management of cloud native web applications
https://www.deep.mg
Mozilla Public License 2.0
3 stars 0 forks source link

deepify undeploy still doesn't remove all resources #369

Closed eistrati closed 7 years ago

eistrati commented 7 years ago

I had to clean up our AWS account earlier today because of some CloudWatch resources that haven't been removed by deepify undeploy. Here below is my log:

MitocGroup:~ eugene$ deepify list
09:37:38 GMT-0500 (CDT)  To get more details, run deepify list --depth=2 or deepify list --depth=3

#1. ATM Api (ebf6f3ce) | using 11 cloud services 
    -------------------------------------------- 

#2. My Custom Web App 59a59b0775a22b6374595c8ebf525f1c (959f2268) | using 11 cloud services 
    --------------------------------------------------------------------------------------- 

#3. My Custom Web App cb5193fee11d7111ff6de88cd12d62c7 (770efe66) | using 11 cloud services 
    --------------------------------------------------------------------------------------- 

#4. deep-todo SW enabled green (d4a064e7) | using 11 cloud services 
    --------------------------------------------------------------- 

#5. dev (3aa6f1d2) | using 1 cloud services 
    --------------------------------------- 

#6. dev (4f52a980) | using 1 cloud services 
    --------------------------------------- 

#7. dev (5347f79a) | using 3 cloud services 
    --------------------------------------- 

#8. dev (81a08190) | using 1 cloud services 
    --------------------------------------- 

#9. dev (d1363c23) | using 1 cloud services 
    --------------------------------------- 

#10. dev (e6c23ab8) | using 1 cloud services 
    --------------------------------------- 

#11. dev (ebf6f3ce) | using 1 cloud services 
    --------------------------------------- 

#12. hello.deep.mg (370b9c50) | using 10 cloud services 
    -------------------------------------------------- 

#13. todo.deep.mg (55182282) | using 11 cloud services 
    ------------------------------------------------- 

#14. unknown application | using 2 cloud services 
    -------------------------------------------- 

#15. www-dev.adtechmedia.io (430f1fff) | using 12 cloud services 
    ----------------------------------------------------------- 

#16. www.adtechmedia.io (57ad5dd8) | using 12 cloud services 
    ------------------------------------------------------- 

#17. www.deep.mg (41ae5689) | using 9 cloud services 
    ----------------------------------------------- 

#18. www2_serverlessConf (7bb794bc) | using 11 cloud services 
    -------------------------------------------------------- 

MitocGroup:~ eugene$ deepify list -r 3aa6f1d2 -d 3
09:39:49 GMT-0500 (CDT)  

#1. dev (3aa6f1d2) | using 1 cloud services 
    --------------------------------------- 

    1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevDdbEventualConsistencyListenQueues01213aa6f1d2

MitocGroup:~ eugene$ deepify undeploy -r 3aa6f1d2
09:40:32 GMT-0500 (CDT)  Starting undeploy for APIGateway 
09:40:32 GMT-0500 (CDT)  No resources to remove for APIGateway 
09:40:32 GMT-0500 (CDT)  Starting undeploy for APIGatewayPlan 
09:40:32 GMT-0500 (CDT)  No resources to remove for APIGatewayPlan 
09:40:32 GMT-0500 (CDT)  Starting undeploy for APIGatewayKey 
09:40:32 GMT-0500 (CDT)  No resources to remove for APIGatewayKey 
09:40:32 GMT-0500 (CDT)  Starting undeploy for IAM 
09:40:32 GMT-0500 (CDT)  No resources to remove for IAM 
09:40:32 GMT-0500 (CDT)  Starting undeploy for CognitoIdentity 
09:40:32 GMT-0500 (CDT)  No resources to remove for CognitoIdentity 
09:40:32 GMT-0500 (CDT)  Starting undeploy for Lambda 
09:40:32 GMT-0500 (CDT)  No resources to remove for Lambda 
09:40:32 GMT-0500 (CDT)  Starting undeploy for CloudFront 
09:40:32 GMT-0500 (CDT)  No resources to remove for CloudFront 
09:40:32 GMT-0500 (CDT)  Starting undeploy for DynamoDB 
09:40:32 GMT-0500 (CDT)  No resources to remove for DynamoDB 
09:40:32 GMT-0500 (CDT)  Starting undeploy for S3 
09:40:32 GMT-0500 (CDT)  No resources to remove for S3 
09:40:32 GMT-0500 (CDT)  Starting undeploy for CloudWatchLogs 
09:40:32 GMT-0500 (CDT)  Starting undeploy for SQS 
09:40:32 GMT-0500 (CDT)  No resources to remove for SQS 
09:40:32 GMT-0500 (CDT)  Starting undeploy for ElastiCache 
09:40:32 GMT-0500 (CDT)  No resources to remove for ElastiCache 
09:40:32 GMT-0500 (CDT)  Starting undeploy for ES 
09:40:32 GMT-0500 (CDT)  No resources to remove for ES 
09:40:32 GMT-0500 (CDT)  Starting undeploy for CloudWatchEvents 
09:40:32 GMT-0500 (CDT)  No resources to remove for CloudWatchEvents 
09:40:32 GMT-0500 (CDT)  Starting undeploy for CognitoIdentityProvider 
09:40:32 GMT-0500 (CDT)  No resources to remove for CognitoIdentityProvider 
09:40:34 GMT-0500 (CDT)  Removing resource #/aws/lambda/DeepDevDdbEventualConsistencyListenQueues01213aa6f1d2 from CloudWatchLogs 
09:40:34 GMT-0500 (CDT)  There are 1 resources removed from CloudWatchLogs 
MitocGroup:~ eugene$ deepify list -r 4f52a980 -d 3
09:40:54 GMT-0500 (CDT)  

#1. dev (4f52a980) | using 1 cloud services 
    --------------------------------------- 

    1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevDdbEventualConsistencyListenQueues01214f52a980

MitocGroup:~ eugene$ deepify undeploy -r 4f52a980
09:41:09 GMT-0500 (CDT)  Starting undeploy for APIGateway 
09:41:09 GMT-0500 (CDT)  No resources to remove for APIGateway 
09:41:09 GMT-0500 (CDT)  Starting undeploy for APIGatewayPlan 
09:41:09 GMT-0500 (CDT)  No resources to remove for APIGatewayPlan 
09:41:09 GMT-0500 (CDT)  Starting undeploy for APIGatewayKey 
09:41:09 GMT-0500 (CDT)  No resources to remove for APIGatewayKey 
09:41:09 GMT-0500 (CDT)  Starting undeploy for IAM 
09:41:09 GMT-0500 (CDT)  No resources to remove for IAM 
09:41:09 GMT-0500 (CDT)  Starting undeploy for CognitoIdentity 
09:41:09 GMT-0500 (CDT)  No resources to remove for CognitoIdentity 
09:41:09 GMT-0500 (CDT)  Starting undeploy for Lambda 
09:41:09 GMT-0500 (CDT)  No resources to remove for Lambda 
09:41:09 GMT-0500 (CDT)  Starting undeploy for CloudFront 
09:41:09 GMT-0500 (CDT)  No resources to remove for CloudFront 
09:41:09 GMT-0500 (CDT)  Starting undeploy for DynamoDB 
09:41:09 GMT-0500 (CDT)  No resources to remove for DynamoDB 
09:41:09 GMT-0500 (CDT)  Starting undeploy for S3 
09:41:09 GMT-0500 (CDT)  No resources to remove for S3 
09:41:09 GMT-0500 (CDT)  Starting undeploy for CloudWatchLogs 
09:41:09 GMT-0500 (CDT)  Starting undeploy for SQS 
09:41:09 GMT-0500 (CDT)  No resources to remove for SQS 
09:41:09 GMT-0500 (CDT)  Starting undeploy for ElastiCache 
09:41:09 GMT-0500 (CDT)  No resources to remove for ElastiCache 
09:41:09 GMT-0500 (CDT)  Starting undeploy for ES 
09:41:09 GMT-0500 (CDT)  No resources to remove for ES 
09:41:09 GMT-0500 (CDT)  Starting undeploy for CloudWatchEvents 
09:41:09 GMT-0500 (CDT)  No resources to remove for CloudWatchEvents 
09:41:09 GMT-0500 (CDT)  Starting undeploy for CognitoIdentityProvider 
09:41:09 GMT-0500 (CDT)  No resources to remove for CognitoIdentityProvider 
09:41:11 GMT-0500 (CDT)  Removing resource #/aws/lambda/DeepDevDdbEventualConsistencyListenQueues01214f52a980 from CloudWatchLogs 
09:41:11 GMT-0500 (CDT)  There are 1 resources removed from CloudWatchLogs 
MitocGroup:~ eugene$ deepify list -r 81a08190 -d 3
09:41:41 GMT-0500 (CDT)  

#1. dev (81a08190) | using 1 cloud services 
    --------------------------------------- 

    1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevAsyncConfigDump012181a08190

MitocGroup:~ eugene$ deepify undeploy -r 81a08190
09:42:00 GMT-0500 (CDT)  Starting undeploy for APIGateway 
09:42:00 GMT-0500 (CDT)  No resources to remove for APIGateway 
09:42:00 GMT-0500 (CDT)  Starting undeploy for APIGatewayPlan 
09:42:00 GMT-0500 (CDT)  No resources to remove for APIGatewayPlan 
09:42:00 GMT-0500 (CDT)  Starting undeploy for APIGatewayKey 
09:42:00 GMT-0500 (CDT)  No resources to remove for APIGatewayKey 
09:42:00 GMT-0500 (CDT)  Starting undeploy for IAM 
09:42:00 GMT-0500 (CDT)  No resources to remove for IAM 
09:42:00 GMT-0500 (CDT)  Starting undeploy for CognitoIdentity 
09:42:00 GMT-0500 (CDT)  No resources to remove for CognitoIdentity 
09:42:00 GMT-0500 (CDT)  Starting undeploy for Lambda 
09:42:00 GMT-0500 (CDT)  No resources to remove for Lambda 
09:42:00 GMT-0500 (CDT)  Starting undeploy for CloudFront 
09:42:00 GMT-0500 (CDT)  No resources to remove for CloudFront 
09:42:00 GMT-0500 (CDT)  Starting undeploy for DynamoDB 
09:42:00 GMT-0500 (CDT)  No resources to remove for DynamoDB 
09:42:00 GMT-0500 (CDT)  Starting undeploy for S3 
09:42:00 GMT-0500 (CDT)  No resources to remove for S3 
09:42:00 GMT-0500 (CDT)  Starting undeploy for CloudWatchLogs 
09:42:00 GMT-0500 (CDT)  Starting undeploy for SQS 
09:42:00 GMT-0500 (CDT)  No resources to remove for SQS 
09:42:00 GMT-0500 (CDT)  Starting undeploy for ElastiCache 
09:42:00 GMT-0500 (CDT)  No resources to remove for ElastiCache 
09:42:00 GMT-0500 (CDT)  Starting undeploy for ES 
09:42:00 GMT-0500 (CDT)  No resources to remove for ES 
09:42:00 GMT-0500 (CDT)  Starting undeploy for CloudWatchEvents 
09:42:00 GMT-0500 (CDT)  No resources to remove for CloudWatchEvents 
09:42:00 GMT-0500 (CDT)  Starting undeploy for CognitoIdentityProvider 
09:42:00 GMT-0500 (CDT)  No resources to remove for CognitoIdentityProvider 
09:42:02 GMT-0500 (CDT)  Removing resource #/aws/lambda/DeepDevAsyncConfigDump012181a08190 from CloudWatchLogs 
09:42:02 GMT-0500 (CDT)  There are 1 resources removed from CloudWatchLogs 
MitocGroup:~ eugene$ deepify list -r d1363c23 -d 3
09:42:20 GMT-0500 (CDT)  

#1. dev (d1363c23) | using 1 cloud services 
    --------------------------------------- 

    1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevDdbEventualConsistencyListenQueues0121d1363c23

MitocGroup:~ eugene$ deepify undeploy -r d1363c23
09:42:42 GMT-0500 (CDT)  Starting undeploy for APIGateway 
09:42:42 GMT-0500 (CDT)  No resources to remove for APIGateway 
09:42:42 GMT-0500 (CDT)  Starting undeploy for APIGatewayPlan 
09:42:42 GMT-0500 (CDT)  No resources to remove for APIGatewayPlan 
09:42:42 GMT-0500 (CDT)  Starting undeploy for APIGatewayKey 
09:42:42 GMT-0500 (CDT)  No resources to remove for APIGatewayKey 
09:42:42 GMT-0500 (CDT)  Starting undeploy for IAM 
09:42:42 GMT-0500 (CDT)  No resources to remove for IAM 
09:42:42 GMT-0500 (CDT)  Starting undeploy for CognitoIdentity 
09:42:42 GMT-0500 (CDT)  No resources to remove for CognitoIdentity 
09:42:42 GMT-0500 (CDT)  Starting undeploy for Lambda 
09:42:42 GMT-0500 (CDT)  No resources to remove for Lambda 
09:42:42 GMT-0500 (CDT)  Starting undeploy for CloudFront 
09:42:42 GMT-0500 (CDT)  No resources to remove for CloudFront 
09:42:42 GMT-0500 (CDT)  Starting undeploy for DynamoDB 
09:42:42 GMT-0500 (CDT)  No resources to remove for DynamoDB 
09:42:42 GMT-0500 (CDT)  Starting undeploy for S3 
09:42:42 GMT-0500 (CDT)  No resources to remove for S3 
09:42:42 GMT-0500 (CDT)  Starting undeploy for CloudWatchLogs 
09:42:42 GMT-0500 (CDT)  Starting undeploy for SQS 
09:42:42 GMT-0500 (CDT)  No resources to remove for SQS 
09:42:42 GMT-0500 (CDT)  Starting undeploy for ElastiCache 
09:42:42 GMT-0500 (CDT)  No resources to remove for ElastiCache 
09:42:42 GMT-0500 (CDT)  Starting undeploy for ES 
09:42:42 GMT-0500 (CDT)  No resources to remove for ES 
09:42:42 GMT-0500 (CDT)  Starting undeploy for CloudWatchEvents 
09:42:42 GMT-0500 (CDT)  No resources to remove for CloudWatchEvents 
09:42:42 GMT-0500 (CDT)  Starting undeploy for CognitoIdentityProvider 
09:42:42 GMT-0500 (CDT)  No resources to remove for CognitoIdentityProvider 
09:42:43 GMT-0500 (CDT)  Removing resource #/aws/lambda/DeepDevDdbEventualConsistencyListenQueues0121d1363c23 from CloudWatchLogs 
09:42:43 GMT-0500 (CDT)  There are 1 resources removed from CloudWatchLogs 
MitocGroup:~ eugene$ deepify list -r e6c23ab8 -d 3
09:43:09 GMT-0500 (CDT)  

#1. dev (e6c23ab8) | using 1 cloud services 
    --------------------------------------- 

    1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevReplicationStream0145e6c23ab8

MitocGroup:~ eugene$ deepify undeploy -r e6c23ab8
09:43:25 GMT-0500 (CDT)  Starting undeploy for APIGateway 
09:43:25 GMT-0500 (CDT)  No resources to remove for APIGateway 
09:43:25 GMT-0500 (CDT)  Starting undeploy for APIGatewayPlan 
09:43:25 GMT-0500 (CDT)  No resources to remove for APIGatewayPlan 
09:43:25 GMT-0500 (CDT)  Starting undeploy for APIGatewayKey 
09:43:25 GMT-0500 (CDT)  No resources to remove for APIGatewayKey 
09:43:25 GMT-0500 (CDT)  Starting undeploy for IAM 
09:43:25 GMT-0500 (CDT)  No resources to remove for IAM 
09:43:25 GMT-0500 (CDT)  Starting undeploy for CognitoIdentity 
09:43:25 GMT-0500 (CDT)  No resources to remove for CognitoIdentity 
09:43:25 GMT-0500 (CDT)  Starting undeploy for Lambda 
09:43:25 GMT-0500 (CDT)  No resources to remove for Lambda 
09:43:25 GMT-0500 (CDT)  Starting undeploy for CloudFront 
09:43:25 GMT-0500 (CDT)  No resources to remove for CloudFront 
09:43:25 GMT-0500 (CDT)  Starting undeploy for DynamoDB 
09:43:25 GMT-0500 (CDT)  No resources to remove for DynamoDB 
09:43:25 GMT-0500 (CDT)  Starting undeploy for S3 
09:43:25 GMT-0500 (CDT)  No resources to remove for S3 
09:43:25 GMT-0500 (CDT)  Starting undeploy for CloudWatchLogs 
09:43:25 GMT-0500 (CDT)  Starting undeploy for SQS 
09:43:25 GMT-0500 (CDT)  No resources to remove for SQS 
09:43:25 GMT-0500 (CDT)  Starting undeploy for ElastiCache 
09:43:25 GMT-0500 (CDT)  No resources to remove for ElastiCache 
09:43:25 GMT-0500 (CDT)  Starting undeploy for ES 
09:43:25 GMT-0500 (CDT)  No resources to remove for ES 
09:43:25 GMT-0500 (CDT)  Starting undeploy for CloudWatchEvents 
09:43:25 GMT-0500 (CDT)  No resources to remove for CloudWatchEvents 
09:43:25 GMT-0500 (CDT)  Starting undeploy for CognitoIdentityProvider 
09:43:25 GMT-0500 (CDT)  No resources to remove for CognitoIdentityProvider 
09:43:26 GMT-0500 (CDT)  Removing resource #/aws/lambda/DeepDevReplicationStream0145e6c23ab8 from CloudWatchLogs 
09:43:26 GMT-0500 (CDT)  There are 1 resources removed from CloudWatchLogs
eistrati commented 7 years ago

Here is another one:

MitocGroup:~ eugene$ deepify list
14:24:19 GMT-0400 (EDT)  To get more details, run deepify list --depth=2 or deepify list --depth=3

#1. ATM Api (ebf6f3ce) | using 11 cloud services 
    -------------------------------------------- 

#2. ATM Consolidated (3e4186ab) | using 11 cloud services 
    ----------------------------------------------------- 

#3. My Custom Web App 02db236e7d78a427e226469b921c09e1 (9d71cad5) | using 11 cloud services 
    --------------------------------------------------------------------------------------- 

#4. My Custom Web App 29ed7db20fed59212082cefe53ae629e (1558a2a9) | using 11 cloud services 
    --------------------------------------------------------------------------------------- 

#5. dev (430f1fff) | using 1 cloud services 
    --------------------------------------- 

#6. hello.deep.mg (370b9c50) | using 10 cloud services 
    -------------------------------------------------- 

#7. todo.deep.mg (55182282) | using 11 cloud services 
    ------------------------------------------------- 

#8. unknown application | using 2 cloud services 
    -------------------------------------------- 

#9. www.adtechmedia.io (57ad5dd8) | using 12 cloud services 
    ------------------------------------------------------- 

#10. www.deep.mg (41ae5689) | using 9 cloud services 
    ----------------------------------------------- 

#11. www2_serverlessConf (7bb794bc) | using 11 cloud services 
    -------------------------------------------------------- 

MitocGroup:~ eugene$ deepify list -r 430f1fff -d 3
14:24:46 GMT-0400 (EDT)  

#1. dev (430f1fff) | using 1 cloud services 
    --------------------------------------- 

    1. Security Tier / Amazon IAM | using 33 cloud resources: 
      1.1. DeepDevAdmin430f1fff
      1.2. DeepDevAdminRole1d3d2efc430f1fff
      1.3. DeepDevAdminRole3242d1c0430f1fff
      1.4. DeepDevAdminRole33266741430f1fff
      1.5. DeepDevAdminRole498cabb1430f1fff
      1.6. DeepDevAdminRole8125e59f430f1fff
      1.7. DeepDevAdminRole8bb0a8d8430f1fff
      1.8. DeepDevAdminRole997475c6430f1fff
      1.9. DeepDevAdminRolea0f89290430f1fff
      1.10. DeepDevAdminRoledbbaf089430f1fff
      1.11. DeepDevAdminRoledd4228b1430f1fff
      1.12. DeepDevAdminRoleeb84d5da430f1fff
      1.13. DeepDevAdminRolefe5785b7430f1fff
      1.14. DeepDevBackendRole1d3d2efc430f1fff
      1.15. DeepDevBackendRole3242d1c0430f1fff
      1.16. DeepDevBackendRole33266741430f1fff
      1.17. DeepDevBackendRole498cabb1430f1fff
      1.18. DeepDevBackendRole8125e59f430f1fff
      1.19. DeepDevBackendRole8bb0a8d8430f1fff
      1.20. DeepDevBackendRole997475c6430f1fff
      1.21. DeepDevBackendRolea0f89290430f1fff
      1.22. DeepDevBackendRoledbbaf089430f1fff
      1.23. DeepDevBackendRoledd4228b1430f1fff
      1.24. DeepDevBackendRoleeb84d5da430f1fff
      1.25. DeepDevBackendRolefae68c1c430f1fff
      1.26. DeepDevBackendRolefe5785b7430f1fff
      1.27. DeepDevFfhgfhfghfgh430f1fff
      1.28. DeepDevFfhgfhfghfghghgfhfghgfh430f1fff
      1.29. DeepDevTestDuplicate430f1fff
      1.30. DeepDevTesting_role430f1fff
      1.31. DeepDevTestRole123456789430f1fff
      1.32. DeepDevTestRole430f1fff
      1.33. DeepDevTest_mobile430f1fff
avozicov commented 7 years ago

Test Passed Acceptance Criteria:

avozicov commented 7 years ago

Status Update: I have found an use case in which bug is still reproduced, please refer to the steps bellow:

BUG: There are still resources left after undeploy process is completed 

1. Data Tier / Amazon CloudWatchLogs | using 1 cloud resources: 
      1.1. /aws/lambda/DeepDevDdbEventualConsistencyListenQueues357e03be8a

Note:

Please refer to attachments for more details:

screen shot 2017-06-07 at 19 01 38 screen shot 2017-06-07 at 19 02 13 screen shot 2017-06-07 at 19 02 38
avozicov commented 7 years ago

Please refer bellow for the "deepify undeploy" related issue https://github.com/MitocGroup/deepify/issues/388

avozicov commented 7 years ago

Test Failed

Please refer to attachments for more details:

1.Deployed test application
1.Listed provisioned services for deployed application

screen shot 2017-06-26 at 19 32 57

_2.Performed "deepify undeploy by [apphash] option" 2.1 Listed provisioned services after undeploy

screen shot 2017-06-26 at 19 33 07

3. Reviewed "undeploy log" 3.1 Checked "CloudWatch logs" from aws console

screen shot 2017-06-27 at 14 07 43 screen shot 2017-06-27 at 12 11 42

4. Listed provisioned services once again - Cloudwatch logs are listed as provisioned services

screen shot 2017-06-27 at 14 07 55

Note:

mgoria commented 7 years ago

This issue with CloudWatchLogs seems to be generated only for cloud watch logs related to scheduled lambdas, that's why on undeploying it tells logs are removed successfully but on successive deepify list some log groups are still listed, these log groups sometimes are created again by scheduled lambdas that weren't removed yet. Looking for a solution ...

avozicov commented 7 years ago

Test Passed

screen shot 2017-06-30 at 19 11 38 screen shot 2017-06-30 at 19 06 41