Jets Deploy is not successful.

NacchaF commented 5 months ago

Checklist

[x] Upgrade Jets: Are you using the latest version of Jets? This allows Jets to fix issues fast.
[x] Reproducibility: Are you reporting a bug others will be able to reproduce and not asking a question. If you're unsure or want to ask a question, do so on https://community.boltops.com
[x] Code sample: Have you put together a code sample to reproduce the issue and make it available? Code samples help speed up fixes dramatically. If it's an easily reproducible issue, then code samples are not needed. If you're unsure, please include a code sample.

My Environment

Software	Version
Operating System	MacOS Ventura 13.4.1
Jets	6.0.3
Ruby	3.3.1
Rails	7.1.3.3

Expected Behaviour

Successful deployment.

Current Behavior

Error after building and pushing container image.

Step-by-step reproduction instructions

The following is an excerpt from the error log at that time.

113c74938a2f: Layer already exists
c13df57104fa: Layer already exists
2bd1a2222589: Layer already exists
a0446a1a49e6: Pushed
b7922889b265: Pushed
9c7852261ccf: Pushed
87cfc986bc55: Pushed
v1-2024-06-01T07-21-29Z: digest: sha256:143aafd303be09243b4e3be02371f36da6c8f2b266374c9d831f30113df21622 size: 3669
Building CloudFormation templates
bundler: failed to load command: jets-remote (/codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bin/jets-remote)
/codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:66:in `all_actions_colon_star': undefined method `include?' for nil:NilClass (NoMethodError)

      action.include?(":") ? action : "#{action}:*"
            ^^^^^^^^^
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:88:in `statement_from_hash'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:59:in `block in statement_from_array'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:55:in `map'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:55:in `statement_from_array'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-76ce3d3b3501/lib/jets/cfn/iam/policy.rb:37:in `standardize'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/application_role.rb:108:in `custom_policy'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/application_role.rb:59:in `computed_policies'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/class_role.rb:19:in `computed_policies'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/application_role.rb:50:in `policies'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/separate_policy.rb:50:in `policy_document'
        from /codebuild/output/src2987860223/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7acc981690be/lib/jets/remote/cfn/resource/iam/separate_policy.rb:38:in `definition'

Code Sample

just "new"ed Rails App with API mode.

Solution Suggestion

nothing.

tongueroo commented 5 months ago

Bummer.

Killing 2 birds with one stone. Created a video demo with an Rails API built from scratch and deployed it to AWS Lambda with Jets: https://learn.boltops.com/courses/ruby-on-jets-6-0-guide/lessons/rails-api-on-aws-lambda-with-jets

Interested in what's going on here. Will follow up.

NacchaF commented 5 months ago

@tongueroo

Thanks for making the explanatory video.

I'll check it out later and let you know how it goes.

tongueroo commented 5 months ago

@NacchaF Np. When you get a chance, check your email for my followup 😄

NacchaF commented 5 months ago

I tried again, following the video you created. This time the type of error that occurs seems to have changed. Below is the error log from that time.

$ AWS_PROFILE=******* AWS_REGION=ap-northeast-1 bundle exec jets deploy

f913b9a38ea5: Pushed
a5de6de934c2: Pushed
3c6ec9c9f94a: Pushed
v1-2024-06-03T18-11-40Z: digest: sha256:863ff69410cef6df7b8d748f5fda81fdbd948cc7ec68c7cc5a47f3f770fc9386 size: 3460
Building CloudFormation templates
Built CloudFormation templates at /tmp/jets/******-web-api-dev/templates
#<Thread:0x0000ffff9d525d58 /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7550aa744771/lib/jets/remote/package/assets/upload.rb:25 run> terminated with exception (report_on_exception is true):
/codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Access Denied (Aws::S3::Errors::AccessDenied)
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/request.rb:72:in `send_request'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/client.rb:15616:in `put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:66:in `block in put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `block in open_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `open'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `open_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:65:in `put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:46:in `block in upload'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/user_agent.rb:29:in `feature'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:40:in `upload'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/customizations/object.rb:477:in `block in upload_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/user_agent.rb:29:in `feature'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/customizations/object.rb:476:in `upload_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7550aa744771/lib/jets/remote/package/assets/upload.rb:40:in `upload_to_s3'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7550aa744771/lib/jets/remote/package/assets/upload.rb:26:in `block (3 levels) in upload_folder'
bundler: failed to load command: jets-remote (/codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bin/jets-remote)
/codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/raise_response_errors.rb:17:in `call': Access Denied (Aws::S3::Errors::AccessDenied)
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/sse_cpk.rb:24:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/dualstack.rb:21:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/plugins/accelerate.rb:43:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/checksum_algorithm.rb:111:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/jsonvalue_converter.rb:16:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/invocation_id.rb:16:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/idempotency_token.rb:19:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/param_converter.rb:26:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/request_callback.rb:89:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/response_paging.rb:12:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/plugins/response_target.rb:24:in `call'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/seahorse/client/request.rb:72:in `send_request'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/client.rb:15616:in `put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:66:in `block in put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `block in open_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `open'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:55:in `open_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:65:in `put_object'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:46:in `block in upload'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/user_agent.rb:29:in `feature'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/file_uploader.rb:40:in `upload'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/customizations/object.rb:477:in `block in upload_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-core-3.196.1/lib/aws-sdk-core/plugins/user_agent.rb:29:in `feature'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/gems/aws-sdk-s3-1.151.0/lib/aws-sdk-s3/customizations/object.rb:476:in `upload_file'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7550aa744771/lib/jets/remote/package/assets/upload.rb:40:in `upload_to_s3'
        from /codebuild/output/src3576152674/src/wrapper/vendor/bundle/ruby/3.2.0/bundler/gems/jets-remote-7550aa744771/lib/jets/remote/package/assets/upload.rb:26:in `block (3 levels) in upload_folder'
Error running command: exit status 1

[Container] 2024/06/03 18:11:51.991075 Command did not exit successfully ./jets-go deploy exit status 1

There was no difference between executing the command inside the container and executing the command directly on the host.

And, I have granted AdministratorAccess to my AWS user.

I wish I could provide more information, but I really don't have anything else.

tongueroo commented 5 months ago

RE: I wish I could provide more information, but I really don't have anything else.

All good. It does help a little bit.

Thanks for sending over the config/jets files. Well, it looks like the vanilla-generated files. So that's not it.
I also deployed to AWS_REGION=ap-northeast-1, the same region as you. Of course, it deployed successfully. 🥺

RE: I have granted AdministratorAccess to my AWS user.

There's a difference between the AWS IAM user you use locally vs the AWS Codebuild Remote Runner IAM Role. Docs: https://docs.rubyonjets.com/docs/iam/deploy/vs/ Jets creates and manages the IAM Role associated with the CodeBuild Remote Runner, and your config/jets settings did not override them. So believe that looks good.

This seems to be an issue specific to your AWS account.
This is a guess and a shot-in-the-dark, but your AWS account might have an AWS Service Control Policy at the Organization level.
Handling multiple birds with one stone again. Added these docs: https://docs.rubyonjets.com/docs/iam/deploy/remote/#aws-service-control-policies

I'm guessing it's an AWS SCP that's resulting in the Access Denied (Aws::S3::Errors::AccessDenied) error. I was kind of able to mimic the error by creating an SCP for one of my accounts. I created a deny policy that denies all s3. It mimic the error, albeit it errors earlier in the process. It's might be because my mimic SCPs is not like yours and even more restrictive. Again, it's a guess.

Maybe check if your AWS account has SCPs.

NacchaF commented 5 months ago

Maybe check if your AWS account has SCPs.

I checked my AWS account and found that SCPs were set up and the permissions to change them did not exist.

Is there any way to change to Local（on my own?) Deploy instead of Remote Deploy?

I will also try to see if I can get the permissions here in parallel.

NacchaF commented 5 months ago

I tested it in a clean environment with no SCPs, and the deploy was successful. I will let you know if I find any other problems again.

Thanks for your help.

rubyonjets / jets