Stuck in CREATE_IN_PROGRESS / DELETE_IN_PROGRESS

[bs-thomas]

Hello there,

First of all I'm quite excited to see there is a custom resource plugin for cloudformation to create MySQL DB users. Thanks for all the work and open sourcing this great stuff!

While the documentation seems thorough, I'm experiencing an issue while trying out, and wondering if I'm missing out on some key steps.

I'm trying to deploy the demo, and noticed that it is stuck in CREATE_IN_PROGRESS when creating KongReaderUser and KongUser.

2019-09-08 22:36:15 UTC+0800 | KongReaderUser | CREATE_IN_PROGRESS | - 2019-09-08 22:36:15 UTC+0800 | KongUser | CREATE_IN_PROGRESS

I have installed both the user-provider and secret-provider (otherwise I wouldn't be able to reach the above).

To minimize dependencies, I have also tried to remove secret-provider in the demo template, and hard-coded the passwords in. However, it was still stuck in the same place.

Then, when I gave up, and tried to delete the cloudformation stack, the status was stuck in DELETE_IN_PROGRESS for the above users as well, and then finally failed after an hour.

2019-09-09 00:20:56 UTC+0800	KongReaderUser	DELETE_FAILED	Custom Resource failed to stabilize in expected time
2019-09-09 00:20:53 UTC+0800	KongUser	DELETE_FAILED	Custom Resource failed to stabilize in expected time
2019-09-08 23:20:40 UTC+0800	KongUser	DELETE_IN_PROGRESS	-
2019-09-08 23:20:40 UTC+0800	KongReaderUser	DELETE_IN_PROGRESS

I tried to look in CloudWatch, but it doesn't show any errors. I'm not an expert in Python, so what I can understand is limited. But I did notice a few functions have try and finally, but no "except".

Could it be hanging in some place, with no exceptions? Could anyone give me a hand and shed me some light, as I'm quite lost at the moment.

Thank you very much guys in advance!

[mvanholsteijn]

I suspect that provider (lambda) cannot access your database. Are your database and the lambda deployed into the same VPC? Does the security group of the lambda have network access on the database?

[bs-thomas]

@mvanholsteijn Thanks for your kind and prompt reply to my issue, and I apologize for my delay response on this as I was on another task and haven't had a chance to revisit this.

To respond to your question. The database and lambda are supposed to be deployed on the same VPC, as they were all instantiated from the demo template provided in the repository.

The security group was also a part of the template.

You can check out the template here (slightly modified version to remove the secret), and minor tweaks that shouldn't matter. So I was wondering if what could still be missing. Could it be related to user roles?

demo-stack-edited.yaml.txt

[mvanholsteijn]

@bs-thomas, if there is no network response the most likely cause is that the lambda cannot access the database via de network, eg a security group and/or NACL problem.

Can you check whether you have associated the 'SecurityGroup' parameter is associated with the Lambda?

You can set the LOG_LEVEL to DEBUG on the lambda to get a bit more information on what is going on in the lambda.

[mvanholsteijn]

I guess you solved it...

[bs-thomas]

@mvanholsteijn No I actually haven't. But because my response is slow, it's okay to close it. I do really appreciate your help on this. I'll look into this once I get a chance, and if you don't mind me, to ping you again if there is any issues. Thank you once again!!

[mvanholsteijn]

@bs-thomas When I deploy the provider and the demo it all works fine.

Normally, when the lambda is stuck it is due to lack of network connectivity. Check whether

1. the security group associated with your lambda is granted access to the MySQL database and
2. there is a route from the lambda subnets to the db subnets of the MySQL database