Closed djolereject closed 11 months ago
djolereject
commented
11 months ago Just to expand, since I noticed I didn't mention it, but this is in my config/application.rb:
config.managed_iam_policy = %w[AWSLambda_FullAccess AmazonSQSFullAccess]as I understand it should be enough to allow creating everything needed?
tongueroo
commented
11 months ago Unsure. Wondering about this:
05:15:28AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:... was not successfully created: The following resource(s) failed to create: [Waitlist]. Can you see what the error on the nested List stack is. Here are some tips https://docs.rubyonjets.com/docs/debug/cloudformation/
Note, I also put together a demo test project https://github.com/tongueroo/demo-jets-sqs-queue It deployed successfully on a brand new cloud9 machine.
Maybe try deploying the test project to debug also. Thinking the CloudFormation error in the List stack will be most helpful though.
Screenshot of CloudFormation stacks:
Shared SQS List screenshot:
Here's are also the lambda functions.
I also did some manual test by sending a message to the generated SQS queue.
$ export QUEUE_URL=https://sqs.us-west-2.amazonaws.com/112233445566/demo-dev-List-NDBP8IFB1M05-Waitlist-StxATDNj3X78
$ aws sqs send-message --queue-url $QUEUE_URL --message-body '{"test": "hello world"}'
{
"MD5OfMessageBody": "3d635e69eb93fd184b47a31d460ca2b6",
"MessageId": "3a9a99e1-a866-4e68-ac31-e413fb495988"
}
$ You can log the hard_job-fix in another terminal and see the event coming in
$ jets logs -f -n hard_job-fix
Tailing logs for /aws/lambda/demo-dev-hard_job-fix
Waiting for log group /aws/lambda/demo-dev-hard_job-fix to exist. Waiting 5 seconds.
2023-12-15 18:17:55 UTC INIT_START Runtime Version: ruby:3.2.v9 Runtime Version ARN: arn:aws:lambda:us-west-2::runtime:b96ddb9b1905c3979339d7706a5f7cfda1d851593b1255eb0f15ff573c17fd28
2023-12-15 18:17:58 UTC START RequestId: be392533-83a5-5aae-b263-78c5192df684 Version: $LATEST
2023-12-15 18:17:58 UTC fix {"Records":[{"messageId":"3a9a99e1-a866-4e68-ac31-e413fb495988","receiptHandle":"AQEBzmNbOceOGqBkoYPEfF7ZUQ2MoXkrY1cHFADrUjDiw3t4ecP2ZOlhAOicrPONYzmC9EPkRRchxaGDH4mN/G5WfzyCmbUo/BQBFuvqpVhBrXpZJ8pNJPKkDfXSZbtpvMLdCofTHuAg6IhDU3K9l/EEQzlF9WSyOFnGfrsLY15yeGyJ4pfJ6R3+X6dt4uCy3fpU1hD2zwJoVrZVYz499Sr9b5vh+0zyR570idC2EQww+r6XkNR1YA2+rALsashA1XI/868ZPA2CGeaj+qAlPOoQUv0JBW22ulILUi0i8jzVu9sTcJAoyzz4JzAjq0uaSIa2WPShMKA7XN0ArM7bs3MJ6ngCnqHzOKR857i6dyN5BeVENH+2mQzHZwwUX0ImanOIWi1bxXEztzEr8SNo2l88r9gIk1Tw5QGmhGL1nw3lwg3nHwjH3hDZE9YED4YVqVJZ","body":"{\"test\": \"hello world\"}","attributes":{"ApproximateReceiveCount":"1","SentTimestamp":"1702664274758","SenderId":"AIDAJTCD6O457Q7BMTLYM","ApproximateFirstReceiveTimestamp":"1702664274764"},"messageAttributes":{},"md5OfBody":"3d635e69eb93fd184b47a31d460ca2b6","eventSource":"aws:sqs","eventSourceARN":"arn:aws:sqs:us-west-2:536766270177:demo-dev-List-NDBP8IFB1M05-Waitlist-StxATDNj3X78","awsRegion":"us-west-2"}]}
2023-12-15 18:17:58 UTC END RequestId: be392533-83a5-5aae-b263-78c5192df684
2023-12-15 18:17:58 UTC REPORT RequestId: be392533-83a5-5aae-b263-78c5192df684 Duration: 4.91 ms Billed Duration: 5 ms Memory Size: 1536 MB Max Memory Used: 168 MB Init Duration: 3035.12 msAlso note: I tested both eu-central-1 and us-west-2 to eliminate AWS_REGION as an issue.
djolereject
commented
11 months ago I have to say that I'm novice with CloudFormation so maybe I'm missing something, but here is the screenshot of all Stacks that are related to this project:
I'll look into the documentation you provided so maybe I'll get some better understanding of CloudFormation which will allow me to show results in more details and ask better question.
tongueroo
commented
11 months ago All good. Can you click on the mailroom-dev-JetsController-* stack and see what that error is when you get a chance? It usually has details that explains why it rolled back. Thanks!
djolereject
commented
11 months ago Here it is:
tongueroo
commented
11 months ago Interesting. In this case, there's no useful info. Bummer. Can you create project that reproduces the issue and put it on github when you get a chance?
Also, can you try deploying the demo project https://github.com/tongueroo/demo-jets-sqs-queue Wondering what’s different.
Also, just released Jets 5.0.3 to fix #690 Unsure if it's related. Maybe give that a shot too.
cd project
bundle update
bundle info jets # confirm jets 5.0.3
bundle exec jets deployTook another look, was looking too fast last time. In the first error logs. This stands out.
05:15:28AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:... was not successfully created: The following resource(s) failed to create: [Waitlist].
...
05:15:39AM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack mailroom-dev The following resource(s) failed to create: [PostmanJob, List]. The following resource(s) failed to update: [JetsController]. The JetsController stack is only rolling back because of the Waitlist stack error. So believe the root error is in the Waitlist stack.
When you get a chance, can you click on the Waitlist stack and see what that error says. It should provide more details.
Thanks!
djolereject
commented
11 months ago 1) I just looked at the CloudFormation and there is no Waitlist stack on its own. You can see the first image in this thread, those are my only stacks. I tried to find the first red line and it might be this one:
2) Checked Jets 5.0.3 and received the same issue.
3) I tried your demo and got the same situation:
05:15:08AM UPDATE_IN_PROGRESS AWS::CloudFormation::Stack demo-dev User Initiated
05:15:11AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
05:15:11AM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer
05:15:11AM CREATE_IN_PROGRESS AWS::IAM::Role IamRole
05:15:12AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway Resource creation Initiated
05:15:13AM CREATE_IN_PROGRESS AWS::IAM::Role IamRole Resource creation Initiated
05:15:20AM CREATE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer Resource creation Initiated
05:15:21AM CREATE_COMPLETE AWS::Lambda::LayerVersion GemLayer
05:15:23AM CREATE_COMPLETE AWS::CloudFormation::Stack ApiGateway
05:15:24AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1
05:15:25AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1 Resource creation Initiated
05:15:29AM CREATE_COMPLETE AWS::IAM::Role IamRole
05:15:30AM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController
05:15:30AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob
05:15:31AM CREATE_IN_PROGRESS AWS::IAM::Policy IamPolicy Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack JetsController Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob Resource creation Initiated
05:15:31AM CREATE_IN_PROGRESS AWS::CloudFormation::Stack List Resource creation Initiated
05:15:35AM CREATE_COMPLETE AWS::CloudFormation::Stack ApiResources1
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack List Embedded stack arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87 was not successfully created: The following resource(s) failed to create: [Waitlist].
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack JetsController Resource creation cancelled
05:15:42AM CREATE_FAILED AWS::CloudFormation::Stack PostmanJob Resource creation cancelled
05:15:42AM CREATE_FAILED AWS::IAM::Policy IamPolicy Resource creation cancelled
05:15:43AM UPDATE_ROLLBACK_IN_PROGRESS AWS::CloudFormation::Stack demo-dev The following resource(s) failed to create: [PostmanJob, JetsController, IamPolicy, List].
05:15:45AM UPDATE_ROLLBACK_COMPLETE_CLEANUP_IN_PROGRESS AWS::CloudFormation::Stack demo-dev
05:15:46AM DELETE_IN_PROGRESS AWS::IAM::Policy IamPolicy
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiResources1
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack List
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack PostmanJob
05:15:46AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack JetsController
05:15:46AM DELETE_COMPLETE AWS::IAM::Policy IamPolicy
05:15:56AM DELETE_COMPLETE AWS::CloudFormation::Stack ApiResources1
05:16:07AM DELETE_COMPLETE AWS::CloudFormation::Stack PostmanJob
d05:18:00AM DELETE_COMPLETE AWS::CloudFormation::Stack List
05:18:24AM DELETE_COMPLETE AWS::CloudFormation::Stack JetsController
05:18:24AM DELETE_IN_PROGRESS AWS::Lambda::LayerVersion GemLayer
05:18:24AM DELETE_IN_PROGRESS AWS::IAM::Role IamRole
05:18:24AM DELETE_IN_PROGRESS AWS::CloudFormation::Stack ApiGateway
05:18:25AM DELETE_COMPLETE AWS::Lambda::LayerVersion GemLayer
05:18:35AM DELETE_COMPLETE AWS::CloudFormation::Stack ApiGateway
05:18:37AM DELETE_COMPLETE AWS::IAM::Role IamRole
05:18:37AM UPDATE_ROLLBACK_COMPLETE AWS::CloudFormation::Stack demo-dev
Stack rolled back: UPDATE_ROLLBACK_COMPLETEdemo problem seems to me to be the same, here is the only log from demo:
tongueroo
commented
11 months ago Opps should had mentioned. So the WaitList stack will "disappear". This is because when a nested stack fails to create for the first time, it'll rollback and delete itself. So you won't see it if you refresh too late.
CloudFormation behaves this way so you don't have orphan resources and tries to keep things tidy. It makes debugging trickier.
There are a few ways to get to the failed deleted stack:
arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87. You can then go to the "deleted stack" url directly. Build it up to this: https://eu-central-1.console.aws.amazon.com/cloudformation/home?region=eu-central-1#/stacks/events?filteringText=&filteringStatus=active&viewNested=true&stackId=arn:aws:cloudformation:eu-central-1:228342851134:stack/demo-dev-List-2AXDJ1D221D1/eac76b40-9c92-11ee-94a4-02719741ea87 You have to replace the stackId query string parameter with the failed deleted stack arn value. Don't worry about url escaping the value, modern browsers do this automatically.I'm wondering:
WaitList stack's event log would be useful.djolereject
commented
11 months ago And you guessed it correctly! I went to the failed stack ARN with your helpful link and saw it there:
Resource handler returned message: "User: arn:aws:iam::228342851134:user/{my user} is not authorized to perform: sqs:createqueue on resource: arn:aws:sqs:eu-central-1:228342851134:demo-dev-List-2AXDJ1D221D1-Waitlist-QzhRPuni2qNn because no identity-based policy allows the sqs:createqueue action ...
I added AmazonSQSFullAccess to the user and all went smoothly. Thanks a lot! I don't think I would've found this on my own. Something about all those users/permissions/services is just too much to keep in working memory for me.
Thank you once more!
tongueroo
commented
11 months ago Made some improvements in #696 This should help for future users.
Checklist
jets upgradecommand that makes this a simple task. There's also an Upgrading Guide: http://rubyonjets.com/docs/upgrading/My Environment
Expected Behaviour
Deployment with
jets deployshould create SQS queue and two Lambdas.Current Behavior
Deployment finishes with an error.
Step-by-step reproduction instructions
First I created a project with newest possible version of the Jets gem (5.0.2) and Ruby 3.2.0 to see if everything works as expected, and it does. I deploy the app with one controller action and I can see one Lambda created. Then I add literal example from the documentation: https://docs.rubyonjets.com/docs/events/sqs/ (Generated Shared SQS Queue). This is my change in regard to working project:
I made sure to copy Jobs and resource as they are used in documentation to make sure I'm not causing the problem with something specific to my setup. This is the output of
jets deployafter I do this:I tried to do some variations on this, but couldn't find any way of generating SQS queue. Please advise if I should do something different or is this a bug.
Thanks!