Improve speed UpdateService

[kamitath]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do * not help prioritize the request If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request Now our updating an AWS App Runner service process takes 5m on average. When deployment fails , it takes over 15m for rollback. I appreciate if AWS improve those speed.

Describe alternatives you've considered No alternative available.

Additional context

Attachments

[nerdeveloper]

Please, AWS needs to increase the speed of deployment for Create service and UpdateService. I just tried out the Go source-based feature based on this blog post, and it takes 15-20 mins to re-deploy. We want to use this, but this is too slow compared to modern deployment speed.

[MarcoGlauser]

We have another angle for this. We use App Runner for gitlab review apps. We often push a final commit for rebasing or something similar and then merge the PR without requiring to check the review app again. Once we merge the PR, gitlab will automatically run a script to clean up the review app. However, the cleanup of the review apps often fails because the environment is still updating from the last commit. All we update is the image tag. everything else stays the same.

[hariohmprasath]

Hi @nerdeveloper and @MarcoGlauser, Thanks for reaching out, I have created a repost question regarding this - https://repost.aws/questions/QUT5meInAbSYG-iFMtwkvKTw/app-runner-update-service-taking-15-to-20-mins-to-complete. Can you please share the service ARN's associated with these behaviors, so we can take a look to understand the delay.

[MarcoGlauser]

We don't have specific ARN's that we can share, since we quickly create and delete the services for our pull requests. Most services only exist for a couple of hours and get automatically deleted after a certain time. The images that we deploy with App Runner are roughly 20MB but the update process stays in pulling image for 5+ minutes. Although we have observed 15 minutes in the past, in most cases we have observed closer to 5 minutes to update services. However, it's still inconvenient to have to wait for 5-15 minutes to quickly update one or two lines in a PR.

A quick workaround that would help us would also be a cli flag that blocks until the create/update/delete-service operation is completed. That way our CI knows that the environment is still busy before trying to update or delete the service

[cemo]

@hariohmprasath we have around 5 mins deployments. Is this expected? Can we speed up this process?

[hariohmprasath]

Hi @cemo, the team is actively looking into improving the speed of an image based service (to start with), we will keep this community posted regarding the progress going forward. thanks

[ollnixon]

Hi @hariohmprasath , do you have an update on this? 6 months on from when this issue was opened, and we are still experiencing 5 minute deployment times which is unacceptable, particularly given alternative hosts have proved that it can be done within a fraction of the time. Thanks

[hariohmprasath]

Hi @ollnixon, The team is actively working on improving the deployment speed for image-based service, so stay tuned for the upcoming announcements (in a few weeks). :) Thanks

[epomatti]

hi @hariohmprasath we too are facing slow deployments and extremely slow rollbacks (25+ minutes) with image-based.

I can see looking at the roadmap that this issue is still in the 'Researching' phase. I will assume that this will take at least 3 months for an initial patch to be available.

Is the recommendation to use code-based deployments in the meanwhile?

Additionally, if there's anything I can share or do to help, let me know.

[hariohmprasath]

Hi @epomatti, Just to understand the usecase better: Do you see 25+ minutes delay in rollback everytime in case of bad deployment on a image based service?

[epomatti]

@hariohmprasath now that you mentioned, it depends on what triggered the rollback.

For example, if the application code fails to startup, it takes 20/25+ minutes to rollback (as I shared in the previous image).

Rollbacks due to service configuration errors are faster, but still take around 5 minutes. Example: error when loading a Secret Manager secret when setting up the instance as a secret parameter to App Runner.

[hariohmprasath]

Thanks for the quick response, we will look into this and get back to you.

[ThomasFahrner-Amazon]

Hi all,

Thank you for your comments and reaching out.

We have just released an improvement to fast-rollback any code or image based deployments that fail. This brings the end-to-end failed deployment speed from the previous 25 minutes to under 5 minutes.

As @hariohmprasath mentioned, we are additionally working on improvements for the overall deployment speed of image-based services and will be in touch soon.

[epomatti]

@ThomasFahrner-Amazon thanks, I can confirm the entire deployment+rollback tests with < 2min (no image change, only config) and < 3min (with image pull deployment). Dev team really appreciated the good news. Can't wait now for the upcoming improvements for overall deployment speed.

[scuw19]

Share this good news that we just release the performance improvement for image-based related operations(create/update/deploy). Customer will noticed an approximately 30 to 40% reduction in deployment time, depending on container image size.

see the details in this blog post: https://aws.amazon.com/blogs/containers/aws-app-runner-improves-performance-for-image-based-deployments/

[neidiom]

Is this task completed?