Closed jvgeee closed 5 years ago
Same issue here
@wowi42 Can you post your dev/dependencies from your package.json as well as your webpack config?
Update: I've just cleared a bunch of memory on my laptop and run this again. I had 11gb free, the packaging service (52 functions) got me down to 185mb during sls package
. Then when zipping it all up it freed up space so the zips only take up ~2gb.
Here's the output from webpack:
Serverless: Invoke webpack:package
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, zone.js@^0.8.18, fast-xml-parser@^2.7.4, request@^2.85.0, bluebird@^3.4.6, joi@^10.6.0, uuid@^3.1.0, dynogels@^8.0.1, firebase-admin@^5.12.1, @slack/client@^4.3.1, sharp@^0.20.2, swagger-ui-dist@^3.17.1, haversine@^1.1.0
Serverless: Packaging service...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
When I look in my .zip files, each zip is about 50mb, and I can see that each function has a node_modules folder that unzips to 183mb, with the following packages inside:
@firebase colour gauge isobject optjs sntp
@google-cloud combined-stream gcp-metadata isstream os-homedir sort-keys
@mrmlnc component-emitter gcs-resumable-upload isurl os-locale source-map
@nodelib compressible get-stream items os-tmpdir source-map-resolve
@protobufjs concat-map get-value jmespath osenv source-map-url
@sindresorhus concat-stream getpass joi p-cancelable split-array-stream
@slack configstore github-from-package jsbn p-defer split-string
@types console-control-strings glob json-buffer p-finally sshpk
abbrev copy-descriptor glob-parent json-schema p-is-promise static-extend
acorn core-js glob-to-regexp json-schema-traverse p-queue stream-events
acorn-es7-plugin core-util-is globby json-stringify-safe p-retry stream-shift
ajv create-error-class google-auth-library jsonwebtoken p-timeout strict-uri-encode
ansi-regex cryptiles google-auto-auth jsprim pascalcase string-format-obj
aproba crypto-browserify google-gax jwa path-dirname string-width
are-we-there-yet crypto-random-string google-p12-pem jws path-is-absolute string_decoder
arr-diff dashdash google-proto-files keyv path-type stringifier
arr-flatten debug got kind-of performance-now stringstream
arr-union decamelize graceful-fs lcid pify strip-ansi
array-filter decode-uri-component grpc lodash posix-character-classes strip-json-comments
array-union decompress-response gtoken lodash.includes power-assert stubs
array-uniq deep-equal har-schema lodash.isboolean power-assert-context-formatter tar
array-unique deep-extend har-validator lodash.isinteger power-assert-context-reducer-ast tar-fs
arrify define-properties has lodash.isnumber power-assert-context-traversal tar-stream
ascli define-property has-symbol-support-x lodash.isplainobject power-assert-formatter through2
asn1 delay has-to-string-tag-x lodash.isstring power-assert-renderer-assertion timed-out
assert-plus delayed-stream has-unicode lodash.once power-assert-renderer-base to-buffer
assign-symbols delegates has-value log-driver power-assert-renderer-comparison to-object-path
async detect-libc has-values loglevel power-assert-renderer-diagram to-regex
async-limiter diff-match-patch hash-stream-validation long power-assert-renderer-file to-regex-range
asynckit dir-glob hawk lowercase-keys power-assert-util-string-width topo
atob dot-prop hoek lru-cache prebuild-install tough-cookie
aws-sdk duplexer3 http-cache-semantics make-dir prepend-http traverse
aws-sign2 duplexify http-parser-js map-cache process-nextick-args tslib
aws4 dynogels http-signature map-visit protobufjs tunnel-agent
axios eastasianwidth iconv-lite merge2 pseudomap tweetnacl
babel-runtime ecc-jsbn ieee754 methmeth pump type-name
balanced-match ecdsa-sig-formatter ignore micromatch pumpify typedarray
base empower ignore-walk mime punycode union-value
base64-js empower-core imurmurhash mime-db qs unique-string
base64url end-of-stream indexof mime-types query-string universal-deep-strict-equal
bcrypt-pbkdf ent inflight mimic-response querystring unset-value
bl es-abstract inherits minimatch rc urix
bluebird es-to-primitive ini minimist readable-stream url
boom espurify into-stream minipass regenerator-runtime url-join
brace-expansion estraverse invert-kv minizlib regex-not url-parse-lax
braces eventemitter3 is mixin-deep repeat-element url-to-options
buffer expand-brackets is-accessor-descriptor mkdirp repeat-string use
buffer-alloc expand-template is-arrayish modelo request util-deprecate
buffer-alloc-unsafe extend is-buffer ms resolve-url uuid
buffer-equal-constant-time extend-shallow is-callable nan responselike verror
buffer-fill extglob is-data-descriptor nanomatch ret websocket-driver
buffer-from extsprintf is-date-object needle retry websocket-extensions
bun fast-deep-equal is-descriptor node-abi retry-axios which-pm-runs
bytebuffer fast-glob is-extendable node-forge retry-request wide-align
cache-base fast-json-stable-stringify is-extglob node-pre-gyp rimraf window-size
cacheable-request faye-websocket is-fullwidth-code-point noop-logger safe-buffer wrap-ansi
call-me-maybe fill-range is-glob nopt safe-regex wrappy
call-signature finity is-number normalize-url safer-buffer write-file-atomic
camelcase firebase-admin is-obj npm-bundled sax ws
capture-stack-trace follow-redirects is-object npm-packlist semver xdg-basedir
caseless for-in is-odd npmlog set-blocking xml2js
chownr foreach is-plain-obj number-is-nan set-value xmlbuilder
class-utils forever-agent is-plain-object oauth-sign sharp xtend
cliui form-data is-regex object-assign signal-exit y18n
clone-response fragment-cache is-retry-allowed object-copy simple-concat yallist
co from2 is-stream object-keys simple-get yargs
code-point-at fs-constants is-stream-ended object-visit simple-swizzle zone.js
collection-visit fs-copy-file-sync is-symbol object.entries slash
color fs-minipass is-typedarray object.getownpropertydescriptors snakeize
color-convert fs.realpath is-windows object.pick snapdragon
color-name function-bind isarray object.values snapdragon-node
color-string functional-red-black-tree isemail once snapdragon-util
The biggest dependancies in those node modules are:
So 133mb is just these 10 dependancies
Any ideas @HyperBrain ? I'm at a bit of a loss here. I've also reverted to using the webpack config from the readme (with the addition of the nodeExternals() external)
// Version with promises
// webpack.config.js
const webpack = require('webpack');
const slsw = require('serverless-webpack');
const BbPromise = require('bluebird');
const nodeExternals = require('webpack-node-externals');
module.exports = BbPromise.try(() => {
return slsw.lib.serverless.providers.aws.getAccountId()
.then(accountId => ({
entry: slsw.lib.entries,
plugins: [
new webpack.DefinePlugin({
AWS_ACCOUNT_ID: `${accountId}`,
}),
],
devtool: 'source-map',
// webpack-node-externals
// don't bundle built in modules like 'fs', 'path' etc.
target: 'node',
// webpack-node-externals
// Define 'externals' which won't be included in the bundle that webpack generates.
// We're excluding everything in node_modules.
externals: [nodeExternals()],
// Run babel on all .js files and skip those in node_modules
module: {
rules: [{
test: /\.js$/,
loader: 'babel-loader',
include: __dirname,
exclude: /node_modules/
}]
}
}));
});
I'll check and reply tomorrow. Sorry for the delay
jascination notifications@github.com schrieb am Di., 4. Sep. 2018, 2:51 nachm.:
Update: I've just cleared a bunch of memory on my laptop and run this again. I had 11gb free, the packaging service (52 functions) got me down to 185mb during sls package. Then when zipping it all up it freed up space so the zips only take up ~2gb.
Here's the output from webpack:
Serverless: Invoke webpack:package Serverless: Package lock found - Using locked versions Serverless: Packing external modules: babel-runtime@^6.26.0, zone.js@^0.8.18, fast-xml-parser@^2.7.4, request@^2.85.0, bluebird@^3.4.6, joi@^10.6.0, uuid@^3.1.0, dynogels@^8.0.1, firebase-admin@^5.12.1, @slack/client@^4.3.1, sharp@^0.20.2, swagger-ui-dist@^3.17.1, haversine@^1.1.0 Serverless: Packaging service... Serverless: Invoke aws:package:finalize Serverless: Invoke aws:common:moveArtifactsToPackage
When I look in my .zip files, each zip is about 50mb, and I can see that each function has a node_modules folder that unzips to 183mb, with the following packages inside:
@firebase colour gauge isobject optjs sntp @google-cloud combined-stream gcp-metadata isstream os-homedir sort-keys @mrmlnc component-emitter gcs-resumable-upload isurl os-locale source-map @nodelib compressible get-stream items os-tmpdir source-map-resolve @protobufjs concat-map get-value jmespath osenv source-map-url @sindresorhus concat-stream getpass joi p-cancelable split-array-stream @slack configstore github-from-package jsbn p-defer split-string @types console-control-strings glob json-buffer p-finally sshpk abbrev copy-descriptor glob-parent json-schema p-is-promise static-extend acorn core-js glob-to-regexp json-schema-traverse p-queue stream-events acorn-es7-plugin core-util-is globby json-stringify-safe p-retry stream-shift ajv create-error-class google-auth-library jsonwebtoken p-timeout strict-uri-encode ansi-regex cryptiles google-auto-auth jsprim pascalcase string-format-obj aproba crypto-browserify google-gax jwa path-dirname string-width are-we-there-yet crypto-random-string google-p12-pem jws path-is-absolute string_decoder arr-diff dashdash google-proto-files keyv path-type stringifier arr-flatten debug got kind-of performance-now stringstream arr-union decamelize graceful-fs lcid pify strip-ansi array-filter decode-uri-component grpc lodash posix-character-classes strip-json-comments array-union decompress-response gtoken lodash.includes power-assert stubs array-uniq deep-equal har-schema lodash.isboolean power-assert-context-formatter tar array-unique deep-extend har-validator lodash.isinteger power-assert-context-reducer-ast tar-fs arrify define-properties has lodash.isnumber power-assert-context-traversal tar-stream ascli define-property has-symbol-support-x lodash.isplainobject power-assert-formatter through2 asn1 delay has-to-string-tag-x lodash.isstring power-assert-renderer-assertion timed-out assert-plus delayed-stream has-unicode lodash.once power-assert-renderer-base to-buffer assign-symbols delegates has-value log-driver power-assert-renderer-comparison to-object-path async detect-libc has-values loglevel power-assert-renderer-diagram to-regex async-limiter diff-match-patch hash-stream-validation long power-assert-renderer-file to-regex-range asynckit dir-glob hawk lowercase-keys power-assert-util-string-width topo atob dot-prop hoek lru-cache prebuild-install tough-cookie aws-sdk duplexer3 http-cache-semantics make-dir prepend-http traverse aws-sign2 duplexify http-parser-js map-cache process-nextick-args tslib aws4 dynogels http-signature map-visit protobufjs tunnel-agent axios eastasianwidth iconv-lite merge2 pseudomap tweetnacl babel-runtime ecc-jsbn ieee754 methmeth pump type-name balanced-match ecdsa-sig-formatter ignore micromatch pumpify typedarray base empower ignore-walk mime punycode union-value base64-js empower-core imurmurhash mime-db qs unique-string base64url end-of-stream indexof mime-types query-string universal-deep-strict-equal bcrypt-pbkdf ent inflight mimic-response querystring unset-value bl es-abstract inherits minimatch rc urix bluebird es-to-primitive ini minimist readable-stream url boom espurify into-stream minipass regenerator-runtime url-join brace-expansion estraverse invert-kv minizlib regex-not url-parse-lax braces eventemitter3 is mixin-deep repeat-element url-to-options buffer expand-brackets is-accessor-descriptor mkdirp repeat-string use buffer-alloc expand-template is-arrayish modelo request util-deprecate buffer-alloc-unsafe extend is-buffer ms resolve-url uuid buffer-equal-constant-time extend-shallow is-callable nan responselike verror buffer-fill extglob is-data-descriptor nanomatch ret websocket-driver buffer-from extsprintf is-date-object needle retry websocket-extensions bun fast-deep-equal is-descriptor node-abi retry-axios which-pm-runs bytebuffer fast-glob is-extendable node-forge retry-request wide-align cache-base fast-json-stable-stringify is-extglob node-pre-gyp rimraf window-size cacheable-request faye-websocket is-fullwidth-code-point noop-logger safe-buffer wrap-ansi call-me-maybe fill-range is-glob nopt safe-regex wrappy call-signature finity is-number normalize-url safer-buffer write-file-atomic camelcase firebase-admin is-obj npm-bundled sax ws capture-stack-trace follow-redirects is-object npm-packlist semver xdg-basedir caseless for-in is-odd npmlog set-blocking xml2js chownr foreach is-plain-obj number-is-nan set-value xmlbuilder class-utils forever-agent is-plain-object oauth-sign sharp xtend cliui form-data is-regex object-assign signal-exit y18n clone-response fragment-cache is-retry-allowed object-copy simple-concat yallist co from2 is-stream object-keys simple-get yargs code-point-at fs-constants is-stream-ended object-visit simple-swizzle zone.js collection-visit fs-copy-file-sync is-symbol object.entries slash color fs-minipass is-typedarray object.getownpropertydescriptors snakeize color-convert fs.realpath is-windows object.pick snapdragon color-name function-bind isarray object.values snapdragon-node color-string functional-red-black-tree isemail once snapdragon-util
The biggest dependancies in those node modules are:
- aws-sdk (23M)
- babel-runtime (8.2M)
- core-js (7.3M)
- dynogels (11M)
- firebase (4.2M)
- google-p12-pem (1.8M)
- google-proto-files (2.9M)
- grpc (46M)
- lodash (4.8M)
- sharp (24M)
Any ideas @HyperBrain https://github.com/HyperBrain ? I'm at a bit of a loss here. I've also reverted to using the webpack config from the readme (with the addition of the nodeExternals() external)
// Version with promises // webpack.config.js
const webpack = require('webpack'); const slsw = require('serverless-webpack'); const BbPromise = require('bluebird'); const nodeExternals = require('webpack-node-externals');
module.exports = BbPromise.try(() => { return slsw.lib.serverless.providers.aws.getAccountId() .then(accountId => ({ entry: slsw.lib.entries, plugins: [ new webpack.DefinePlugin({ AWS_ACCOUNT_ID:
${accountId}
, }), ], devtool: 'source-map',// webpack-node-externals // don't bundle built in modules like 'fs', 'path' etc. target: 'node', // webpack-node-externals // Define 'externals' which won't be included in the bundle that webpack generates. // We're excluding everything in node_modules. externals: [nodeExternals()], // Run babel on all .js files and skip those in node_modules module: { rules: [{ test: /\.js$/, loader: 'babel-loader', include: __dirname, exclude: /node_modules/ }] }
})); });
— You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub https://github.com/serverless-heaven/serverless-webpack/issues/441#issuecomment-418354973, or mute the thread https://github.com/notifications/unsubscribe-auth/AFRM3n5jWV5v-y2OAC1J3Oy_HfERYLuYks5uXndigaJpZM4WT5dU .
Did you sort this out @jascination?
I ran into this same issue with dynogels. It appears all of dynogels deps are being included regardless of having peers at the top level.
Here's what worked for me: externals:[nodeExternals( { whitelist: [ /^dynogels/ ] } )]
This forces the dependencies to the top per a comment made by @HyperBrain in a different thread.
https://github.com/serverless-heaven/serverless-webpack/issues/247#issuecomment-388389152
Are there any other workarounds for this? I'm considering having a seperate serverless application for each of my functions as that currently seems to be the only reliable way of using webpack with node externals whilst avoiding huge package sizes
I have a similar issue without webpack. 10 times faster to not package them individually ...
I originally had this problem with a separate project, but now working on a new one it doesn't seem to happen anymore. I think I might have update my serverless in the mean time (currently running 1.38.0)
Relevant config as follows:
custom:
webpack:
webpackConfig: ./webpack.config.js
includeModules:
forceExclude:
- aws-sdk
packager: 'yarn'
package:
individually: true
functions:
foo:
handler: handlers/foo.handle
bar:
handler: handlers/bar.handle
webpack.config.js
const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');
module.exports = {
entry: slsw.lib.entries,
target: 'node',
mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
optimization: {
minimize: false
},
devtool: 'nosources-source-map',
externals: [nodeExternals()],
module: {
rules: [
{
test: /\.js$/,
exclude: /node_modules/,
use: [
{
loader: 'babel-loader'
}
],
}
]
},
output: {
libraryTarget: 'commonjs2',
path: path.join(__dirname, '.webpack'),
filename: '[name].js',
sourceMapFilename: '[file].map'
}
};
It might be worth noting that each function has a separate file for its handler. I would imagine that if you had two handlers in one file for two functions (handler.foo, handler.bar as an example) then webpack wouldn't know how to separate out the deps and shake those out.
I'm also skeptical whether this will tree shake a dependency that isn't used in a required file. e.g.
lib/mylib.js
const _ = require('lodash');
const foo = () => {
return _.min(1,2);
}
const bar = () => {
return 1;
}
export { foo, bar }
If you were to require this above file and only use the bar function as an example, then it might still include lodash in your bundle, but I haven't tested this personally.
Hi @nzchicken, thanks for updating us, going to close the issue since it seems resolved 👍
if you had two handlers in one file for two functions
I'm pretty sure Webpack handles that fine, but perhaps @HyperBrain can confirm? 😄
it might still include lodash in your bundle
IIRC this whole situation can be avoided by either installing specific Lodash functions i.e. (lodash.min
) or by using the ES-module version of Lodash (lodash-es
).
This is a Bug Report
Description
Trying to do
sls package
with yourpackage: individually: true
flag set. When I tried this, it took about 30mins (for around 30 functions within 8 services) and, and ate up about 5gb of my memory.I looked inside the
.webpack/
folder, and within each function thenode_modules
folder was about 500mb.Additional Data
``
None available, although here is my webpack config:
And here are my dependancies:
What's the best way to debug this? I can see that
aws-cli
is getting included in all of these lambdas, but this is only about 20mb. I've seen the threads on force-excluding AWScli too, but I think my issue is a little more sever than that.Any ideas what I'm doing wrong?