Closed tommedema closed 6 years ago
Also, I tried this with aws-sdk
being both a devDependency
and a normal dependency. It's included in both cases.
These are my deps:
"devDependencies": {
"babel-core": "^6.26.0",
"babel-loader": "^7.1.2",
"babel-plugin-source-map-support": "^1.0.0",
"babel-plugin-transform-runtime": "^6.23.0",
"babel-preset-es2015": "^6.24.1",
"babel-preset-stage-3": "^6.24.1",
"serverless-scriptable-plugin": "^0.6.0",
"serverless-webpack": "^4.0.0",
"standard": "^10.0.3",
"uuid": "^3.1.0",
"webpack": "^3.8.1",
"webpack-node-externals": "^1.6.0",
"shelljs": "^0.7.8",
"aws-sdk": "^2.162.0"
},
"dependencies": {
"babel-runtime": "^6.26.0",
"cfn-response": "^1.0.1",
"greenlock": "^2.1.18",
"le-challenge-s3": "^1.0.1",
"le-store-s3": "^1.0.1",
"source-map-support": "^0.5.0"
}
Note that le-challenge-s3
and le-store-s3
have aws-sdk
defined as a dependency. Of course, it should still be excluded.
Even setting this in webpack.config.js did not help:
module.exports = {
...
externals: [nodeExternals(), /aws-sdk/],
...
}
I just confirmed that this is due to indirect dependencies on aws-sdk
. E.g. if I change:
import s3StoreFactory from 'le-store-s3'
import s3ChallengeFactory from 'le-challenge-s3'
to:
import s3StoreFactory from './no-op-challenge' // 'le-store-s3'
import s3ChallengeFactory from './no-op-challenge' // 'le-challenge-s3'
The package size will go from 10MB to 1MB.
Of course, indirect dependencies on aws-sdk
should be excluded too. Is there any way to force this or should this be fixed at a lower level in serverless-webpack
?
It seems that other build tools have "solved" this with a deepExclude
, see https://github.com/nfour/serverless-build-plugin/issues/37
Hi @tommedema , thanks for reporting. Yes, this is, because forceExclude excludes first level dependencies. As your 2 dependencies themselves introduce the aws-sdk it is not possible to exclude the aws-sdk installed from there. I'll explain why, below.
In general, this declaration is the correct one to exclude dependencies (from your tries above):
custom:
# allows for the exclusion of external modules to webpack
webpackIncludeModules:
forceExclude:
- aws-sdk
But you can do a workaround, to transform the aws-sdk into a first level dependency.
If you bundle the two le-*
dependencies, they will become first level dependencies of the deployment package and the aws-sdk exclusion definition mentioned above will effectively exclude it.
Just declare:
externals: [nodeExternals( { whitelist: [ 'le-challenge-s3', 'le-store-s3' ] ) } ) ]
Then the modules will be bundled and aws-sdk is now a first level dependency. The exclude declaration now will lead to a proper exclusion.
I had a look at the deepExclude
and the build module (source code) you mentioned above. It does not work that way. I'm pretty sure that it will eliminate just the aws-sdk folder and leaves all dependencies of the aws-sdk itself untouched.
The reason is, because NPM5 does a module flattening and optimization after install, that means that all dependencies of the aws-sdk are strayed beneith the /node_modules
directory, and do not reside under node_modules/le-*.s3/node_modules
anymore.
Even if you now delete the aws-sdk directory, you'll miss all dependencies of the aws-sdk which makes up nearly all of the sdk's size.
With NPM 5 (and other modern optimizing module packagers), you just cannot predict anymore, what exactly is part of a dependency just by looking at the directory structure. (See #239 for a discussion about that). So the directory structure must be treated as opaque and cannot be used to make any decisions.
You can verify that just by doing a npm install aws-sdk
with an empty package.json
. Then inspect the node_modules folder:
$ ls node_modules/
aws-sdk/ crypto-browserify/ isarray/ punycode/ url/ xmlbuilder/
base64-js/ events/ jmespath/ querystring/ uuid/
buffer/ ieee754/ lodash/ sax/ xml2js/
Of course, indirect dependencies on aws-sdk should be excluded too. Is there any way to force this or should this be fixed at a lower level in serverless-webpack?
So this cannot be solved in a correct way easily, if ever. Only NPM could solve that by having a switch/option to exclude packages - regardless where they appear.
The most correct way is, that modules that require the aws-sdk define them as peerDependencies. So the le-*
dependencies should have aws-sdk as peerDependency, which in turn requires the using module (your service) to have it as either "devDependecy" or "dependency" to fulfill the requirement.
Then the forceExclude will work as expected.
This is the same as with serverless-webpack and the webpack dependency.
As long as the modules have the sdk in their production dependencies it could additionally come to version conflicts and even duplications of the aws-sdk.
Much appreciated, thanks again @HyperBrain.
I understand the situation and why it's hard to resolve. I will for now use externals: [nodeExternals( { whitelist: [ 'le-challenge-s3', 'le-store-s3' ] ) } ) ]
and will consider automating this process in the future.
Not sure if this is very maintainable though, because it means that all these whitelisted packages will be bundled inside the webpack bundle and processed etc. I guess this could cause issues in the future, e.g. if they use binaries.
I found a workaround described here: https://github.com/serverless-heaven/serverless-webpack/issues/306#issuecomment-420426295 for anyone struggling with this that can't change the library that requires aws-sdk
Just adding a comment for those who may come to this late like me and was also stuck with aws-sdk
being included.
It took me quite a few different attempts (due to my ignorance) and I didn't get it working until I read the source code of this plugin and found the correct syntax that works for me is:
custom:
webpack:
includeModules:
forceExclude:
- aws-sdk
Just thought it was worth a post in case anyone else got stuck (I had forceExclude
directly under webpack
🙄 )
Ran into this recently and no the config in serverless.yml did not work. I had to update the webpack config by adding aws-sdk in the externals.
This is a Bug Report
Description
For bug reports:
What went wrong? aws-sdk is a built-in dependency on Lambda, therefore it should not be added to the bundle of functions that import
aws-sdk
, yet this does happen. Even when excludingaws-sdk
using theforceExclude
option. This results in each function being at least 10MB (the package itself is 25MB).What did you expect should have happened? For
aws-sdk
to not be in the artifact zip filesnode_modules
folder.What was the config you used? I tried several variations. This is my webpack.config.js:
And these are the variations of my serverless.yml that I tried; all failed to exclude aws-sdk:
I also tried:
I also tried:
And finally:
Additional Data