More helpful logs when service "Create Failed"

[kdhillon]

Tell us about your request When creating a new service, create fails without any errors in deployment logs. The last visible logs are

02-14-2022 10:18:10 PM [Build] Successfully tagged application-image:latest
02-14-2022 10:18:10 PM [Build] Successfully built 2a53f3e899cb

Which suggests that build succeeded. Seems like there should always be an error logged in deployment log anytime a Service Create fails.

Describe alternatives you've considered Tried looking through forums and past posts, all of them suggest checking logs, but event logs and/or deployment logs show no errors explaining why it failed.

Additional context ARN: arn:aws:apprunner:us-west-2:520917683465:service/arpeggi-web-beta2/600285cea79a48fbb137786c78d31b52 Another ARN: arn:aws:apprunner:us-west-2:520917683465:service/arpeggi-web-beta/ea870ffb4d0a44979811a77ac1bf96f5

Current Service Quota: 10. Existing Services: 3.

Repro steps (reproducible 5/5 tries):

• "Create Service"
• "Source code repository"
• Connect github repo + branch (we have another Service correctly set up and running from the exact same github branch)
• Deployment settings: Manual or Automatic (I've tried both)
• Configure settings to be exact same as existing Service
• Default settings (AWS managed key, etc)
• Wait for "Create service in progress..." to complete, always get "Create Failed"
• Event log is always:

  02-14-2022 10:08:30 PM [AppRunner] Starting source code build.
  02-14-2022 10:08:29 PM [AppRunner] Successfully pulled source code.
  02-14-2022 10:07:41 PM [AppRunner] Service status is set to OPERATION_IN_PROGRESS.
  02-14-2022 10:07:41 PM [AppRunner] Service creation started.

Deployment Log for "Create Service" contains no errors: https://us-west-2.console.aws.amazon.com/cloudwatch/home?region=us-west-2#logsV2:log-groups/log-group/$252Faws$252Fapprunner$252Farpeggi-web-beta2$252F600285cea79a48fbb137786c78d31b52$252Fservice/log-events/deployment$252F1ecf04a9c05c4bf1a7410d61a652aff6

[tsheaff]

Yeah this is crazy. I've run into the same issue. I have absolutely no way to debug why it's failing!

[tsheaff]

I'm sure it's an easily solvable issue for why the creation is failing, but with no way to inspect what went wrong, all I can do is poke around in the dark for hours 😠

All I see is something like this. None of the other tabs or CloudWatch has any extra details:

[tsheaff]

cc @akshayram-wolverine

[tsheaff]

In your official guide it says this, which is completely unhelpful because there are no logs

[McLeroy]

I couldn't use App Runner because of this :(

[abba-logico-srl]

Same problem here :(

[michael-newsrx]

This morning I can't resume or create any services. :-(

Nothing logged in either case as to why.

[vladnicula]

same problem here...

[laurenprete]

I faced this issue as well. Failed service creation and deployment both produce empty/useless logs.

I ended up giving up and deploying my app with ECS on Fargate instead. This got Cloudwatch to log an actual error, so I fixed the problem, then redeployed to app runner and it worked. It would have been helpful if App Runner had logged the underlying error in the first place.

[snnles]

Hi everyone,

We are working on providing more actionable logs for create service workflow. You will see incremental improvements in the 'Deployment Logs' and 'Event Logs' over the time.

Meanwhile, If you have any idea to share around how we can make logging and troubleshooting in App Runner better and what information you would like to see in 'Deployment Logs' and 'Event Logs', please share your ideas/feedback here.

Thanks!

[simllll]

Experience something simliar right now, "[AppRunner] Failed to deploy your application source code." but no reason why, is this also part of the improvements around logs?

[hariohmprasath]

Hi @simllll Thanks for reaching out, we are currently working on different options to improve the log experience. Meanwhile can you let us know the service ARN so we can provide you with some details around the service creation failure.

[simllll]

@hariohmprasath thanks for your quick response, e.g. arn:aws:apprunner:eu-west-1:770785616967:service/app-runner-git-deploy-service/aa6ff0e1cbc9414b84b9b104e0373d83 or arn:aws:apprunner:eu-west-1:770785616967:service/website-app-test/c017277194e04c08b05977db5dd7b11a

I guess both have the same issue though. Thanks a lot!

[hariohmprasath]

Hi @simllll, I had a quick look at these services looks like both of them failed at build stage when running the build command. These build logs are available under Deployment logs section in the console. If you are fine you can share us a sample project without any proprietary code so we can try recreating this locally and help you fix this.

[simllll]

The deployment logs just show: not really helpful.

Is there any possibility to get the build output?

[simllll]

Interesting, tried again just now, and now I got some output :-) I added to the npm install command the --ignore-scripts tag, not sure if this is related or some update just got rolled out. anyways, this makes things a lot easier (y)

[simllll]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue.

build-command: npm install --ignore-scripts && patch-package && npm run build

I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

[hariohmprasath]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue.

build-command: npm install --ignore-scripts && patch-package && npm run build

I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

Hi @simllll, Yes based on our internal analysis its related to the build command, will you be able to share a sample project that you are using to run these tests? We would love to look at it and understand this better and improve the experience on our end. Thanks

[hariohmprasath]

It really seems to be related to the "build" command, I changed it agian, and now I'm back with the "no output" issue. build-command: npm install --ignore-scripts && patch-package && npm run build I guess patch-package is failing, but somehow this leads to the behaviour that there is no output logged at all. I will try if it improves if I run a script instead.

Hi @simllll , Thanks for your patience. We are able to recreate this issue locally, looks like any errors related to build commands doesn't get transmitted to customer environment. We will work on a fix and update this thread in the upcoming weeks. Meanwhile you can workaround this problem by updating the build command to the pattern mentioned below and you will start seeing any command execution errors in the Deployment logs.

(buildCommand) 2>&1

I was able to give it a try with an invalid command and was able to identify the error using the Deployment logs (refer below):

Hopefully this helps, let us know if you have any questions.

[simllll]

Thanks @hariohmprasath I succeeded with building now, at least according to the log (arn:aws:apprunner:eu-west-1:770785616967:service/website-app-deploy-test-envs/9b8c8ad1fa4a419f9960bac46f58b75a), last few log lines:

	2022-11-30T09:28:02.824+01:00	[Build] Removing intermediate container 65911fcd9654
	2022-11-30T09:28:02.824+01:00	[Build] ---> 46860de98696
	2022-11-30T09:28:02.824+01:00	[Build] Step 5/5 : EXPOSE 18000
	2022-11-30T09:28:02.824+01:00	[Build] ---> Running in a425b9b34785
	2022-11-30T09:28:02.824+01:00	[Build] Removing intermediate container a425b9b34785
	2022-11-30T09:28:02.824+01:00	[Build] ---> a7eb9cc657cf
	2022-11-30T09:28:02.824+01:00	[Build] Successfully built a7eb9cc657cf
	2022-11-30T09:28:02.824+01:00	[Build] Successfully tagged application-image:latest

looks actually good, but "Create service" is in state failed?

Am I missing something?

I found earlier in the log:

2022-11-30T09:27:38.261+01:00	[AppRunner] Failed to deploy your application source code.

and it started at 2022-11-30T09:07:11.816+01:00	[AppRunner] Starting to build your application source code.

is it maybe related to some timeout?

[hariohmprasath]

Hi @simllll, I have moved this discussion to repost (https://repost.aws/questions/QUHRZ_kd9hR--kIUfAfrDdCg/app-runner-service-creation-failed) and also updated my observation to this service ARN

[asmfadholi]

Hi @hariohmprasath , I'm facing the same issue when deploying to app runner, and the error just said: [AppRunner] Failed to deploy your application source code.

there is no detail provided. I just added @sentry/nextjs library in my nextjs project. Do you have insight about it?

Thank you.

[hariohmprasath]

Hi @asmfadholi, I have moved this discussion to https://repost.aws/questions/QUYEjXJekeQ3O5Xz_YU5W1qw/app-runner-failed-to-deploy. Can you provide the serviceARN, part of your response? We can look into this and share some insights. Thanks

[asmfadholi]

@hariohmprasath I added arn to comment in aws discussion. Thank you.

[hariohmprasath]

Hi, @asmfadholi,

Looks like you have run into a service bug because of a conflict in the NodeJS versions. We will prioritize this and work on a fix soon and keep you posted on the progress. As a temporary workaround you can install nodeJS 14 part of the pre-build phase and use that as a default runtime for building your code. This can be done by creating a configuration based service (using apprunner.yaml) instead of API based one.

Here is a sample apprunner.yaml file that you can use with the suggested fix:

version: 1.0
runtime: nodejs14
build:
  commands:
    pre-build:
      - n 14.18.3
    build:      
      - <<build_command>>
run: 
  command: <<start_command>>
  network:
    port: <<port>>
    env: APP_PORT

Give it a try and let us know if you are still blocked on this. Thanks

[asmfadholi]

I created new app runner service, and config as you suggested above. It works well now. Thanks for your help. @hariohmprasath

[RichardGrant-93]

Hi everyone,

We are working on providing more actionable logs for create service workflow. You will see incremental improvements in the 'Deployment Logs' and 'Event Logs' over the time.

Meanwhile, If you have any idea to share around how we can make logging and troubleshooting in App Runner better and what information you would like to see in 'Deployment Logs' and 'Event Logs', please share your ideas/feedback here.

Thanks!

Well this didn't age well did it.

[januszm]

Not to mention that logs are flipped upside down, newest at the top, which completely breaks most of the multiline messages

[hariohmprasath]

Not to mention that logs are flipped upside down, newest at the top, which completely breaks most of the multiline messages

Hi @januszm A follow up question: Did the service creation fail in your case? If so can you please create a post in repost.aws with the 'serviceARN', screenshot of your logs and share us the post URL? We will be happy to look into this and help you out. Thanks.

[januszm]

oh yeah it did fail. I'm actually still unable to launch a Rails application on AppRunner using "Source code repository".

[hariohmprasath]

oh yeah it did fail. I'm actually still unable to launch a Rails application on AppRunner using "Source code repository".

Can you share the serviceARN and if possible a sample ruby on rails app that you used in this test so we can look into this and get back to you. Thanks

[januszm]

I'll share it with our aws support representatives

[BahodiRajabov]

Me too arn:aws:apprunner:eu-central-1:121334784369:service/tridmo_front/832792c9c49d4e10a9f27d368e498e09

[NachoMoneyNow]

Hey - I got this issue too. everything was working - (deployments from a push worked, build scripts worked). I also test the nextjs site before pushing for deployment using npm run build.

I made some updates to the code, did my npm run build - it came out clean, then I committed and pushed. (nothing else was changed - build scripts are still the same).

the issue starts after "[AppRunner] Starting source code build" then I get "[AppRunner] Deployment failed. For details, see service logs."

So, I have have no idea where the "service logs" are.... (I also looked in Cloudwatch- it just shows the same error message)

Here is our Service ARN: arn:aws:apprunner:us-east-1:125560917055:service/vulcanPortalDev2/663fff4ff8184a12a6dd7f069a34b835

here is the date and time that it happened: 05-04-2023 02:58:34 PM (GMT +8 hours)

Can a helpful AWS App Runner team member look up the Service log for this error and give me more error details?

Thanks

[mstoyanovv]

More than a year and still no decent way to check logs about why a deploy is failing...

[tnurmi82]

Yeah, me too. I'm new to App Runner. I have a .NET Core app which works fine locally, also works fine if ran with Docker locally, but I can't get it to work in App Runner. "Create Failed".

06-02-2023 02:02:31 PM [AppRunner] Your application stopped or failed to start. See logs for more information.  Container exit code: 1
06-02-2023 02:00:25 PM [AppRunner] Performing health check on path '/' and port '443'.
06-02-2023 02:00:15 PM [AppRunner] Provisioning instances and deploying image for publicly accessible service.
06-02-2023 02:00:03 PM [AppRunner] Successfully built source code.

The "Create Service" log shows nothing is wrong, until it ends with line:

[AppRunner] Failed to deploy your application source code.

I cannot find further logs anywhere. EDIT: Finally I realized that Application log appears with delay for some reason. I was too quick to just delete the app. The Application log provided at least some hints to point me to next direction.

[runlaiyang]

Hi @NachoMoneyNow, thanks for reaching out!

I had a quick look at this service, while the latest deployments are successful. The failed ones were at build stage when running the build command.

Build logs are available in CloudWatch Log group > /aws/apprunner/service-name/service-id/service Log steam > deployment/operation-id Within deployment/operation-id, each log message starts with prefix [Build] is the output of your own build scripts.

Hope this would help on your analysis on build phases during deployment.

[vnedelchev]

The issue I encountered was related to building a Docker image from a Mac with an M1 chip. To overcome this problem, I used the following command: "docker build --platform linux/amd64 -t .". This command specifically targets the Linux/amd64 architecture, ensuring compatibility with AWS App Runner. I hope this helps.

[TimSon777]

Hi! No updates here? I am using .net 6, asp.net core. The application stopped deploying, the aws configuration was not changed, a similar aws configuration for a different environment (more strict) works. Old versions also do not work, everything works on a different environment and locally

Logs (nothing about health):

Service status is set to RUNNING.
Service update failed. For details, see service logs.
Provisioning instances and deploying image.
Successfully copied the image from ECR.
Service status is set to OPERATION_IN_PROGRESS.
Service update started.

arn:aws:apprunner:us-east-1:347648581624:service/tilly-staging/130e2119d25749399a9cbe369d6a19da

Upd. An absolutely identical copy next to it, with all the same settings, roles, etc. (only the log group was different) started up (the port has changed)

Upd2. After creating a new apprunner, the deployment of the old one was fixed

[Frans06]

Same here, deploys were working amazing until yesterday no changes on infra. this is my service. arn:aws:apprunner:us-east-2:986761589464:service/platform-staging-platform/13ee82df0d1948368d9f769b30f3d935

once app runner try to provision instances immediately rollback. deploy messages jus said:

[AppRunner] Failed to deploy your application image.

same logs from here. @hariohmprasath Thanks in advance.

Hi! No updates here? I am using .net 6, asp.net core. The application stopped deploying, the aws configuration was not changed, a similar aws configuration for a different environment (more strict) works. Old versions also do not work, everything works on a different environment and locally

Logs (nothing about health):

Service status is set to RUNNING.
Service update failed. For details, see service logs.
Provisioning instances and deploying image.
Successfully copied the image from ECR.
Service status is set to OPERATION_IN_PROGRESS.
Service update started.

[Frans06]

For all this people including @TimSon777 , I solve my problem just pausing the app runner service, wait a time to let AWS kill the current Node and then run service again. After that your service will run fine. (I think It is what happens behind the wires).

[enriquinho-catalan]

I received this error through Pulumi due to an IAM problem. Try to look through Cloudtrail to see if anything is out-of-sorts. I got equally unuseful logs.

[TimSon777]

On the same day, the price of apprunner skyrocketed, almost 5 times. Why is that? @Frans06, do you also?

[TimSon777]

@enriquinho-catalan, I studied everything there, because I thought that someone had broken it. I didn't see anything out of the ordinary.

[timxyz]

In the past I've had my services silently fail because I had a .python-version file committed to the source repository, which AppRunner didn't like. Sharing that in case it helps someone in the same situation.

Unfortunately that is not the reason why I'm currently seeing the issue. My service runs fine when I manually specify the build configuration, but not when I use an equivalent apprunner.yaml file, and I have no idea why because there are no relevant logs.

[szandara]

I chose AWS App Runner because it was supposed to help me quickly deploy my webapp. I waste hours trying to understand what's going on without luck. Would have been easier to just use EC2.

This is all I have

2023-11-01T14:22:47.639+01:00 | [AppRunner] Starting to build your application source code.

[tnurmi82]

@szandara I was stuck on that same error a few times, but eventually I got through it. Now I'm creating my App Runner services using CloudFormation to avoid any manual mistakes and it's working nicely. I'm using the option where my code is in GitHub, and my repository root contains an apprunner.yaml file for instructing App Runner. My 2 cents:

1. If that error appears, give it a minute, then refresh to see if any Application Logs have appeared. For some reason there is a significant delay sometimes, until the App Log appears. In my case I had defined HTTP as the method for App Runner to monitor if my app is alive (root path, port 80). But my app has auth, and root path was returning 401 Unauthorized (silly mistake). Now I ensure that HTTP GET / always returns 200 OK.

2. On another occasion, my apprunner.yaml was incorrect. This time I got no Application Log (because App Runner couldn't build the app), and the vague Deployment Log didn't indicate where the issue was. I looked at my past commits, and realized I had recently made an indentation error in my YAML file.

3. When I later switched from using CLI only to using CloudFormation, I once again got the same error. This time the issue was in my CloudFormation template itself. I was providing incorrect set of parameters for App Runner, and it was causing App Runner not to read my apprunner.yaml file. Again, same vague error on CloudFormation side. Eventually I figured out my mistake in the CF template.

But after all the hurdles, I'm currently very happy. I think that overall App Runner is saving me time and therefore money. I'm now able to sping up new App Runner services very quickly. Let's see if my opinion changes during following months, after I've used App Runner in production for a while.

But I still agree AWS could improve the docs, error messages and logging overall. I must have wasted more than 20 hours on these issues. Better logs would have saved a lot of time.

[runlaiyang]

I chose AWS App Runner because it was supposed to help me quickly deploy my webapp. I waste hours trying to understand what's going on without luck. Would have been easier to just use EC2.

This is all I have

2023-11-01T14:22:47.639+01:00 | [AppRunner] Starting to build your application source code.

Hi @szandara Thanks for reaching out, can you let us know the service ARN so we can provide you with some details around the service creation failure?

[szandara]

@runlaiyang I ended up using AWS Amplify, which seems to do very similar things. There I could find the log that I needed. In fact it was a simple missing dependency.

[neidiom]

I had similar problems and when I switched from HTTP to TCP health checks the deployment passed.