Open jfuss opened 5 years ago
To me this only happens using Ruby once I've added a gem into the equation, with my functions having Gemfile
s in their respective directories. Then I'm forced to build using --use-container
parameter, which takes a lot of time; simply running sam build
doesn't work because the app doesn't find the gems.
If I don't use gems at all, then I have live reloading; but once I need gems, everything only works by rebuilding inside a container. And since that takes about a minute, it makes SAM pretty much unusable for me. I can't build every time I make a small change, it just takes too long.
Is there any progress on this? Having to build every time for the most minor of code changes is not ideal.
I'd like to see a slightly different take on this proposal. Rather than having a continual build function through --watch
option, only build functions that have changed since the last build. I often find myself having to rebuild functions that haven't been modified, or just making changes to the template file, making the feedback loop unnecessarily long.
To me this only happens using Ruby once I've added a gem into the equation, with my functions having
Gemfile
s in their respective directories. Then I'm forced to build using--use-container
parameter, which takes a lot of time; simply runningsam build
doesn't work because the app doesn't find the gems.If I don't use gems at all, then I have live reloading; but once I need gems, everything only works by rebuilding inside a container. And since that takes about a minute, it makes SAM pretty much unusable for me. I can't build every time I make a small change, it just takes too long.
Hi @alxx,
So what is your workaround for that? Or can you share your experience on that?
Thank you,
I don't like to compare but Chalice as this feature if I'm not mistaken ;)
Assuming this is hard to build the watch option since this has been open for the better part of a year. It seems an easier option would be to use a docker volume to share your code rather than requiring a build, this way you do not have to have an actual watch.
For a workaround, you're able to simply delete the .aws-sam
folder at the root of your project and get back invoking your function.
ref: rm -rf .aws-sam/*
at root.
Lacking of a watch
mechanism is eating much of my time (and laptop battery). Would love to have this feature. Especially we are so fully-heartedly dedicate ourselves to serverless thus creating lots of APIs and functions that could really use some watch
ing.
For a workaround, you're able to simply delete the
.aws-sam
folder at the root of your project and get back invoking your function.ref:
rm -rf .aws-sam/*
at root.
That doesn't work for me. Running in python3.8 now and it doesn't pick up the packages in requirements.txt
.
If you're using a Node runtime you may want to consider using Webpack to build bundles for each function instead of sam build
. Then you can use webpack -w
to have it watch for changes which sam local
picks up.
I followed this very excellent write-up: https://dev.to/elthrasher/managing-multiple-functions-with-aws-sam-and-webpack-1581 I found https://www.npmjs.com/package/aws-sam-webpack-plugin also but preferred the simplicity of the former setup.
The relevant config:
module.exports = {
entry: {
functionA: path.join(__dirname, 'src/functionA.js'),
functionB: path.join(__dirname, 'src/functionB.js'),
},
output: {
libraryTarget: 'commonjs2',
path: `${__dirname}/build`,
filename: '[name].js',
},
mode: process.env.NODE_ENV === 'dev' ? 'development' : 'production',
target: "node", // Let webpack know to generate a Node.js bundle
}
PingFunction:
Type: AWS::Serverless::Function
Properties:
CodeUri: build/
Handler: functionA.lambdaHandler
Runtime: nodejs12.x
npx webpack -w
sam local start-api
I just used nodemon for this. Going to put this here in case it would make anyone else's life easier:
npm i -g nodemon
nodemon --exec sam build
Run this in the root folder in one terminal and run sam local start-api
in another. nodemon should build the application automatically on any code changes.
Hey guys,
So, we created a small npm package based in nodemon (samwatch):
https://www.npmjs.com/package/samwatch
https://github.com/mxitgo/samwatch
The way it works is, it copies your js/json files as you save them from the source folder to the corresponding .aws-sam/build folder (as long as the names are the same for your lambda and the code uri in template.yaml).
That is useful to see your changes reflected on the fly if you are running sam local start-api, so you don't have to run sam build after every change.
On the other hand, if the corresponding file copy is not within the .aws-sam/build folder already, the package will trigger a sam build (unless you use n parameter).
The package has helped us get a hot reload feeling for our local lambda function development with sam.
I hope anyone can find it useful.
Thanks!
My usage is also with ruby code so i am using the live reloading as follows currently without actually using sam build. Not sure though if this is recommend or bad practice so would like to get feedback on it.
Let say i have the following folder structure with 2 functions:
lambda_example/
|--hello
||--app.rb
||--Gemfile
|--hello_other
||--app.rb
||--Gemfile
|--template.yaml
instead of using sam build i just do: (i do this from the root directory lambda_example)
# To install new gems for hello (only need to run this when you update the Gemfile or when you don't have a Gemfile.lock yet)
bundle install --gemfile=hello_world/Gemfile
# Now the following installs the gems into the vendor bundle inside the function directory
# similar what sam build does actually if you would look in the .aws-sam folder
bundle install --gemfile=hello_world/Gemfile --deployment --path vendor/bundle
Same thing then for the other function
bundle install --gemfile=hello_other/Gemfile
bundle install --gemfile=hello_other/Gemfile --deployment --path vendor/bundle
Now if you then issue sam local start-api and you make changes to the code then the live reloading stuff should be working without actually needing to build each time. Ofcourse this only probably works if you don't need to build native extensions for gems but esentially you only need to call the bundle install commands above if you change gem dependencies which should not happen that often.
The samwatch thing is cool but does not work for ruby cause it only monitors js or json files it seems. So maybe add an extra option in your cli @eamarce to specify which files to watch?
Now that warm containers has been delivered under #2383 . Is there anyway this can be prioritized? It would help make that dev experience so much better.
Its such a painful process running sam build even for a small change such as a console.log(). Any good workaround? The solutions above (webpack and samwatch) force a folder structure which I am not willing to do.
@anubhavmalik only thing is to modify the code in the build folder.
such a pain... maybe instead of a "watch" option, which will still need to build on every change and slow down development, we could mount the source folder in the container?
In the case of lambda container images it looks like a sam build
is always required as the build output is an image rather than code being pushed to the .aws-sam
directory. I have a working docker compose setup that spins up two services; one serving the API with sam local start-lambda
and the other that rebuilds the images with sam build
on code changes using nodemon
. The feedback loop could be quicker but it works fairly well.
In the case of lambda container images it looks like a
sam build
is always required as the build output is an image rather than code being pushed to the.aws-sam
directory. I have a working docker compose setup that spins up two services; one serving the API withsam local start-lambda
and the other that rebuilds the images withsam build
on code changes usingnodemon
. The feedback loop could be quicker but it works fairly well.
I've been actually doing this and I am curious if this will produce many local images as it builds again. Is it going to do that? Seems like that's what's happening to mine.
I remember using a previous version of the SAM cli and being able to just run the sam local start-api
and so long as I didn't need to add a new endpoint to the template.yml
file the code was reflected automatically, no need for sam build
every single time. I even have a repository setup with a nodemon script set explicitly to watch only for changes in the template.yml
file so that I wouldn't forget to restart it if I modified the template file.
I'm wondering why this behavior was changed? Now I'm honestly considering not using the SAM cli because this workflow is rather annoying.
Maybe a simple solution to that: make a symbolic-link (ln -s) from the source code to the build folder?
Something like:
sam build ...
ln -s src/* .aws-sam/build/FunctionName
@jhechtf That behavior still holds true the difference is when you use build, SAM CLI places the artifacts into another directory with an updated template. When running sam local start-api
, it will read the "built" template. So you can get the same behavior through sam build
in a different terminal or not using sam build
at all and therefore the "source" template is always read.
As a note, we do prioritize off of reactions. So more reactions do help us in deciding priority for outstanding feature requests from the community.
@jfuss I've seen that mentioned before, about simply not using "sam build", but when I remove the .aws-sam
directory and attempt to run sam local start-api
I receive errors stating that it can't find any node modules, e.g. I use the @aws-sdk/client-dynamodb
and without running sam build
I get the error of Error: Cannot find module '@aws-sdk/client-dynamodb'
@jhechtf Is the node modules within the CodeUri of the Function? If not, you need to install your dependencies into the CodeUri location. SAM CLI mounts the CodeUri into the container, which means all the code needs to be under that directory. This is what sam build
does for you, it just does not mutate the source and instead places everything under .aws-sam/
@jfuss the template that I am using is the nodejs14.x generated by the sam cli, which does not include any CodeUri properties. I'll try to add that in, but considering the code gets generated in the <project root>/src/*
and the node_modules are at <project root>
I'm not entirely sure that's going to work out for me.
sam --version
SAM CLI, version 1.27.2
@jhechtf Happy to help but let's spin this off into a different issue. This will distract from the original intent of this issue/request.
Check it.
CommonLayer:
Type: AWS::Serverless::LayerVersion
Properties:
LayerName: common-dependencies
Description: Common dependencies
ContentUri: layers/common/
CompatibleRuntimes:
- python3.8
RetentionPolicy: Retain
HelloWorldFunction:
Type: AWS::Serverless::Function # More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Properties:
CodeUri: functions/consult/
Handler: app.lambda_handler
Runtime: python3.8
Layers:
- !Ref CommonLayer
$ cd layers/commons/
$ pip install -r ../상대경로/requirements.txt -t ./python
$ sam local start-api -t template.yaml --skip-pull-image
I am 100% NOT a fan of this watch idea. Editing code and hitting refresh would still take a long time. Perhaps the solution to the problem is rethinking what is needed. Perhaps... can we have a start-api
option that does a simple API Gateway => Lambda simulation similar to Zip which would do the following so we can have code reloading for containers?
sam build
to run.I do not know if this is a fair ask of the SAM team. Maybe I need the work y'all have done inside of SAM to be a smaller components to assemble. Maybe we have this already and I have not found a good way to do just that... for my needs?
For both Ruby, Node, etc, we use the SAM build image as a Docker in Docker development image. A good example of this in practice for Ruby/Rails is here. https://lamby.custominktech.com/docs/quick_start. When doing this, we can assume the local "host" (docker dev image) has everything we need, system dependencies, dev code which can be edited, etc. All we need is a simple API Gateway => Lambda emulator.
For Rails, we completely side step this issue by using the built-in ./bin/rails server
and the need for API Gateway => Lambda handler integration because that is tested in abstract at another layer. See here: https://github.com/customink/lamby
Today I was coming back to an old Node zip function I wrote a few years ago and wanted to move it to our latest patterns which includes SAM dev containers, production package type image, etc and was super frustrated that SAM does not have a way to support start-api with container images. I tried a few hacks. First, I tried a little bit of @mims92 recommendation by leaning into the need to do sam build
(which technically I do not need because I an assume the host has everything) to trick start-api
into thinking it had what it needed. Along with forcing the host because Docker in Docker with compose, etc. Again, see Lamby Rails cookiecutter. Still, I could not edit my code and hit refresh in the browser.
if [ ! -d "./.aws-sam" ]; then
sam build \
--parameter-overrides "StageEnv=development"
mkdir ./.aws-sam/build/Lambda
pushd ./.aws-sam/build/Lambda
ln -s ../../../src src
popd
fi
sam local start-api \
--host '0.0.0.0' \
--port 3031 \
--container-host 'host.docker.internal'
I then had the crazy idea maybe I could use the RIE (https://docs.aws.amazon.com/lambda/latest/dg/images-test.html) built into my image. So I tried adding this to my Dockerfile-build
(dev & build image) along with a change to my local ./bin/server
script. But I quickly learned this is too LOW LEVEL and not going to work for me. I need the API Gateway integration.
RUN curl -Lo /usr/local/bin/aws-lambda-rie https://github.com/aws/aws-lambda-runtime-interface-emulator/releases/latest/download/aws-lambda-rie \
&& chmod +x /usr/local/bin/aws-lambda-rie
docker-compose run \
--entrypoint "/usr/local/bin/aws-lambda-rie --log-level debug" \
-p 9001:3031 \
node-container-test
For this hack I tried to see if I could have a development only version of my (template-dev.yaml
) to use just with start-api
to see if I could get some simple Zip behavior in my dev container. The idea being this would only be used for the local development server, not used during package & deploy. The start-api
command would pass this template.
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Resources:
DevLambda:
Type: AWS::Serverless::Function
Properties:
CodeUri: .
Events:
HttpApiProxy:
Type: HttpApi
Properties:
ApiId: !Ref DevHttpApi
Runtime: nodejs14.x
Handler: src/index.handler
MemorySize: 512
Timeout: 30
DevHttpApi:
Type: AWS::Serverless::HttpApi
Properties:
StageName: development
sam local start-api \
--template 'template-dev.yaml' \
--container-host 'host.docker.internal' \
--docker-volume-basedir $DIDPWD \
--host '0.0.0.0' \
--port 3031
Where $DIDPWD
would be $PWD
passed down from the host. This used to be needed during a really old SAM version but I can NOT confirm it works (it should) because...
💥💣💥 SAM HAS ANOTHER BUG THAT PREVENTS THIS. SEE HERE: https://github.com/aws/aws-sam-cli/issues/2837
But even if that did work, it would not be a good workaround for a container function that might have done a system dependency via yum install on both the production and development Docker files.
I GET IT! I get why this is hard. The sam build
historic ways of working make it difficult to solve everything. But I do think we have been pained into a corner and exposing an interface like a watcher here is adding to the list. It would be prone to being slow. Maybe there is a better way forward?
Is there a way to use SAM's API Gateway => Lambda integration in some way to make local development work?
Using @sbhvt's comment here https://github.com/aws/aws-sam-cli/issues/921#issuecomment-907859353 I was able to change my last hack above the 💥💣💥 in the following ways. First in my docker-compose.yml
file. Then in my script used to start the server in the SAM dev container.
volumes:
- ${PWD}:${PWD}
working_dir: $PWD
sam local start-api \
--template 'template-dev.yaml' \
--container-host 'host.docker.internal' \
--host '0.0.0.0' \
--port 3031
But this only works on Mac! I can not get it to work with GitHub Codespaces yet. I tried following this advice (https://dev.to/natterstefan/docker-tip-how-to-get-host-s-ip-address-inside-a-docker-container-5anh) and used 172.17.0.1
but that did not help.
So I'm blocked even on the hacks of hacks :(
GOT IT TO WORK ON CODESPACES. Just needed to clear out Docker images and start from scratch. I'll be using COMPOSE_FILE
to add another compose layer for Codespaces that adds this.
extra_hosts:
- host.docker.internal:172.17.0.1
Which allows the bin/server wrapper to do the following, the key here is was adding --container-host-interface '0.0.0.0'
which is needed for Linux and does not hurt the Mac side of things. So final bin/server is something like this.
sam local start-api \
--template 'template-dev.yaml' \
--container-host 'host.docker.internal' \
--container-host-interface '0.0.0.0' \
--host '0.0.0.0' \
--port 3031
AGAIN! This is hacks on top of hacks. IMO we do not need a watch command but better tooling to expose a simple API Gateway => Lambda server option that works with both Docker in Docker and Lambda Containers where it can be assumed that the Docker dev env has everything setup (built) where sam build
is not needed.
Hey guys,
So, we created a small npm package based in nodemon (samwatch):
https://www.npmjs.com/package/samwatch
https://github.com/mxitgo/samwatch
The way it works is, it copies your js/json files as you save them from the source folder to the corresponding .aws-sam/build folder (as long as the names are the same for your lambda and the code uri in template.yaml).
That is useful to see your changes reflected on the fly if you are running sam local start-api, so you don't have to run sam build after every change.
On the other hand, if the corresponding file copy is not within the .aws-sam/build folder already, the package will trigger a sam build (unless you use n parameter).
The package has helped us get a hot reload feeling for our local lambda function development with sam.
I hope anyone can find it useful.
Thanks!
Hi!
Works with PackageType: Image?
Thks!
Workaround All of the sam local command allow 'reloading' of the function code. This means for a given template, if you update the contents of a functions Code/CodeUri, SAM CLI will mount this updated content. So the current option for making interactions easier is to have two terminal windows open, one that builds and one that you local invoke with.
Is there a workaround for projects created with aws-cdk
. Do I need to run cdk watch
or npm run build
(ts => js) in the second terminal?
Just having a sensible way to have a mounted volume inside the docker container would be awesome for my use case
AWS, What does it take to solve that significant UX problem?
Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project
# Running Locally
1. Start lambda build loop (will watch for changes and rebuild)
nodemon --exec "sam build --template template.yaml --use-container" -e py
* install nodemon globally if you don't have it already `npm install -g nodemon`
2. Start local api server on port 5000
sam local start-api -p 5000
AWS, What does it take to solve that significant UX problem?
Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project
# Running Locally 1. Start lambda build loop (will watch for changes and rebuild) nodemon --exec "sam build --template template.yaml --use-container" -e py * install nodemon globally if you don't have it already `npm install -g nodemon` 2. Start local api server on port 5000 sam local start-api -p 5000
This appears to be a JavaScript example
But Python would need to be handled also
AWS, What does it take to solve that significant UX problem? Thanks to everyone who contributed with their solution; I'll re-iterate on what worked for me for Python project
# Running Locally 1. Start lambda build loop (will watch for changes and rebuild) nodemon --exec "sam build --template template.yaml --use-container" -e py * install nodemon globally if you don't have it already `npm install -g nodemon` 2. Start local api server on port 5000 sam local start-api -p 5000
This appears to be a JavaScript example
But Python would need to be handled also
Nope, this works in python, it watches the files with the .py extension - -e py
After playing with it for some time, I've added a delay to avoid re-building too quickly.
nodemon --exec "sam build --template template.yaml --use-container" -e py --delay 2.5
almost 4 years of this issue. Any news?
Here's the command I ended up with for now:
sam build && concurrently "nodemon --on-change-only --ext ts --exec sam build" "sam local start-api"
Requires npm i -g nodemon concurrently
It runs nodemon and sam local start-api
in the background simultaneously. The trick is to have the build available before creating docker image, so it's done beforehand and nodemon does it only after changes (using --on-change-only
flag).
+1 for this feature
Describe your idea/feature/enhancement
When using
sam build
to build functions that will later be invoked throughsam local [invoke|start-lambda|start-api]
, you need to runsam build
before each invoke. The reason for this requirement is because the build command places artifacts (code + dependencies + updated template) into a build folder. Thesam local [invoke|start-lambda|start-api]
use these built templates (this has updated locations to the built function code) by default but there is no way to update the built code without runningsam build
again.Proposal
For the initial implementation of
sam build
, the--watch
option was out of scope. The proposal is to add support for this option to all customers to enable building when source code changes.An alternative could be to make the
sam local
suite of commands build the function during invoke. The main concern with this is the speed at which you can now invoke (other build enhancements might be a requirement for this, speed, incremental builds, etc).Workaround
All of the
sam local
command allow 'reloading' of the function code. This means for a given template, if you update the contents of a functions Code/CodeUri, SAM CLI will mount this updated content. So the current option for making interactions easier is to have two terminal windows open, one that builds and one that you local invoke with.