`sam deploy` is slow to upload build artifacts to S3

[zyrif]

Description:

When running sam deploy command, it seems to upload build artifacts to S3 at a fixed rate of ~230 KB/s, while the upload speed of my internet connection is capped at 2 MB/s. Although the stack region (US-EAST-1) is geographically quite far from my physical location, normal S3 upload speed doesn't suffer as much compared to SAM. I'm working on a Golang project with several lambda functions and the upload takes a while to complete, delaying the process already slowed down by cfn.

Interestingly, when using sam sync --watch command, this slow upload only occurs when initially uploading the build artifacts. On subsequent changes to the code and resyncing, the upload speed is comparable to normal S3 upload speed.

Also, this is not a regression afaik. I've been facing this issue ever since I started working with sam (version 1.29). I thought this behavior was normal until I started using sam sync --watch.

Steps to reproduce:

1. Run sam deploy for a Go project.
2. Observe upload speed using windows task manager or any internet speed monitoring software.

Observed result:

Upload speed when uploading build artifacts:

Upload speed when directly uploading to S3 from browser:

Upload speed when running Amazon S3 Transfer Acceleration Speed Comparison tool:

Expected result:

Upload speed of build artifacts should be comparable to normal S3 upload speed.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Windows 10
2. sam --version: 1.58.0
3. AWS region: US-EAST-1

[sriram-mv]

Thanks for the issue! what are the sizes of the artifacts that you are uploading?

[zyrif]

@sriram-mv Thanks for the quick response! These are about 20MB for each function.

[lucashuy]

Hi, for both sam deploy and sam sync, is the same bucket being used in both instances? SAM CLI will use a default bucket if none is provided, so if deploy was using a custom bucket while sync was using the default bucket (or vice versa) there may be differences.

[zyrif]

@lucashuy Yes, both are using the same bucket, I haven't specified any custom ones. Moreover, manual uploads to the default bucket itself from S3 dashboard are not subjected to this limit.

Also, as I've said before, this behavior is also present in sam sync --watch, but only for the initial deployment which uses Cloudformation. This does not happen on subsequent resync on code changes.

On that note, I've discovered afterwards that sam sync --code is also not affected, which doesn't seem to use CFN and the uploads are much faster. It looks to me, upload speed to S3 is slow for some reason whenever CFN is involved.

Now, I haven't looked into why the initial deployment uses CFN when the point of sam sync is to avoid it (as far as I understand). I assumed it's because I have 'non-serverless' resources defined in the templates file but I haven't tested if this assumption is correct.

[mndeveci]

If you are using sam sync --watch command, then it uses a feature called Auto Dependency Layer (which creates a layer in your stack, and moves your function's dependencies into that layer, so that your actual function zip will only contain your function code). For that reason, using sam sync might be faster than sam deploy.

If you try with sam sync --watch --no-dependency-layer, you might still see some slowness, since this disable that feature and will need to upload the full zip of the function, including your dependencies.

[zyrif]

Hi @mndeveci, the issue I'm primarily facing is slow S3 upload speed of build artifacts with some, but not all commands. Going by the maintainer responses, I don't think I was able to properly convey the issue I'm facing, so I'll try to clarify.

Commands with slow S3 upload speed:

• sam deploy
• sam sync

Commands with normal S3 upload speed:

• sam sync --code

S3 Upload speed comparison between the two commands:

Notes:

• All commands upload to the same bucket.
• --watch switch is not needed to reproduce this issue. When I initially created this issue, I simply didn't know about --code switch.
• --no-dependency-layer switch doesn't appear to have any effect regarding this issue.

Please let me know if you want me to clarify the issue further / perform any actions for further diagnosis.

[mndeveci]

Thanks for your detailed answer here.

If you could provide an example application (in a GH repo for example), that would be great for us since we can try to reproduce the issue on our side.

[zyrif]

If you could provide an example application (in a GH repo for example), that would be great for us since we can try to reproduce the issue on our side.

For me, this issue is reproducible with a Go hello-world project generated using the sam init command. Should I still provide an example repo?

[kaihendry]

sam deploy/sync is outrageously slow compared to apex up. sam sync --watch is the best way to iterate, but it's a shame it requires that setup! DX 🤮

We need a sam function to upload straight to the lambda without touching cloud formation.

[mndeveci]

@kaihendry sad to see your frustration that you have with sam sync --watch, can you elaborate a little bit more where you had struggles while using it?

[kaihendry]

24s for a sam sync on my example here: https://github.com/kaihendry/helloworld

Is that considered reasonable?

[zyrif]

sam sync in my project consistently takes > 5 minutes 😅

[mattgillard]

sam sync --watch for me (after initial setup) takes < 5s on each change for a single tiny python function. I think @kaihendry is concerned about the CFN updates - which do take a while but not what the OP was talking about. I cannot replicate the OP concerns about slow S3 uploads either (but I am not using large artefacts).

[zyrif]

@mattgillard My functions are written in Go, which can have varying artifact sizes (mine around around 15-20MB). When there are several of them and the fact them sam sync uploads (exports) all of them (regardless of they were changed or not) to S3 in a single threaded manner, the process can take a lot of time. The existence of sam sync --watch and sam sync --code makes it bearable, but that's until I have to make a configuration change and run the full sam sync command.

[mattgillard]

Yes - this issue I have experienced as well. So I 100% agree @zyrif - I think this is the root "symptom" and needs to be resolved somehow.

[mndeveci]

Let me first deep dive into different modes of sam sync and then provide feedback about the example repo that @kaihendry provided above.

sam sync has different flags and options, which will change how it executes the sync flow behind;

• If you just run sam sync it will run regular CFN deployment, it is very close to running sam deploy
• You can also run sam sync --code --resource-id {LogicalId} which will build specific function provided with LogicalId and call update_function_code API to update your function (I am giving function as example, but this is also supported for layers, SFN & APIGW resources [if they have their definition as external file]).
• If you want to do continuous synchronization, you can also run sam sync --watch which will listen to file changes to your local files (template, function/layer codes, SFN & APIGW definition files). If template file changes it will trigger a CFN deployment. If you function/layer code changes, it will update these resources directly via API calls. Same goes with SFN & APIGW definition files, if they are changed locally, SAM CLI will update them through API calls.
• If you want to do continuous synchronization by discarding template changes, you can also run sam sync --watch --code, this will discard any CFN deployment which is usually run at the beginning of your sam sync --watch session as well as any change to template file during the same session.

For this example (https://github.com/kaihendry/helloworld), I've made following changes;

• I've moved all function related files (go.mod, go.sum, logger.go, main.go, main_test.go) into another folder called function
• I've created Makefile in new function folder which holds its build definition (basically copied build definition from the Makefile in the root folder)
• I've updated CodeUri for the function in template.yaml file to point to this new function folder

The reason I made this changes is because is because SAM CLI listens to CodeUri folder for code changes. Keeping it as the root folder of the application causes any change in any file or folder will trigger a sync flow execution. After these changes, I am running sam sync --watch and made a change to main.go file, you can see from the video below, it takes around 6 seconds to update the function.

https://user-images.githubusercontent.com/5735811/218819469-9f4ea1c0-21f1-487e-8662-ecd523e2896d.mov

Please let us know if you have further questions.

[kaihendry]

hi @mndeveci sam sync --watch also takes around 6 seconds (wish your logger could print timestamps), without needing to re-organise the code.

This issue is not that sam sync --watch doesn't work. It does work. However it has a huge setup time.

Apex up can upload straight to the lambda runtime (without messing with slow Cloudformation I believe) and afford ~5s iterations without the setup, with one simple command up. I am asking for a similar developer experience.

I also feel that once we have a straightforward way of uploading code changes (without touching Cloudformation), it will be far easier to debug slow uploads.

[mcblair]

👍 to investing in improving this.

We're an AWS Lambda Golang shop as well, and the developer experience is heinous.

We've even broken up our application into 8 different SAM template/applications and parallelized each of their sam build && sam package calls , but that doesn't solve the glaring issue of painfully slow uploads.

I'm thinking it might be worth spending a week or two getting entirely off of SAM and Cloudformation - the temporary disruption to our developers will be nothing compared to how much time they'll save once they have a proper devex.

[GeeWee]

Also encountering this, sam sync takes a couple of seconds to deploy my very minimal setup, while sam deploy can take multiple minutes.