Optimize Package

[mataide]

This is a Feature Proposal

Description

For feature proposals:

My package have 69mb when I try to package using webpack. But it is way less when I use serverless-plugin-optimize, but I think that both plugins are not compatible.

There is a way to optimize the node_modules? Maybe using any plugin from webpack?

Additional Data

• Serverless-Webpack Version you're using: 5.1.1
• Webpack version you're using: 4.6.0
• Serverless Framework Version you're using: 1.25.0
• Operating System:
• Stack Trace (if available):

^[[A^[[BServerless: Bundling with Webpack...
Time: 1264ms
Built at: 2018-04-26 13:33:16
                                                          Asset       Size  Chunks             Chunk Names
                                            lib/gameOptions.yml  693 bytes          [emitted]  
                                     api/teste/match/handler.js   10.8 KiB       0  [emitted]  api/teste/match/handler
                                                   lib/match.js   4.63 KiB          [emitted]  
                                                lib/response.js   2.34 KiB          [emitted]  
                                                    lib/util.js   4.54 KiB          [emitted]  
                                                     lib/acl.js   1.15 KiB          [emitted]  
                                                  lib/config.js  666 bytes          [emitted]  
                                                 lib/crypter.js   1.35 KiB          [emitted]  
                                                    lib/game.js  855 bytes          [emitted]  
                                 api/teste/match/handler.js.map   7.33 KiB       0  [emitted]  api/teste/match/handler
                                                  lib/logger.js  112 bytes          [emitted]  
                                          lib/name_generator.js   1.97 KiB          [emitted]  
         ibm/dev/admin@proak-hyperledger-network/connection.yml   27.1 KiB          [emitted]  
 ibm/dev/admin@proak-hyperledger-network/credentials/privateKey  241 bytes          [emitted]  
                   ibm/dev/admin@proak-hyperledger-network.card   28.4 KiB          [emitted]  
        ibm/dev/admin@proak-hyperledger-network/connection.json   28.8 KiB          [emitted]  
ibm/dev/admin@proak-hyperledger-network/credentials/certificate  733 bytes          [emitted]  
          ibm/dev/admin@proak-hyperledger-network/metadata.json   78 bytes          [emitted]  
Entrypoint api/teste/match/handler = api/teste/match/handler.js api/teste/match/handler.js.map
[0] external "source-map-support/register" 42 bytes {0} [built]
[1] external "babel-polyfill" 42 bytes {0} [built]
[2] ./api/lib/util.js 1.57 KiB {0} [built]
[3] external "composer-client" 42 bytes {0} [built]
[4] ./api/lib/match.js 2.31 KiB {0} [built]
[5] ./api/lib/response.js 2.27 KiB {0} [built]
[6] ./api/teste/match/post/action.js 573 bytes {0} [built]
[7] ./api/teste/match/handler.js 727 bytes {0} [built]
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: source-map-support@^0.5.4, babel-polyfill@^6.26.0, composer-client@^0.18.1
Serverless: Packaging function: testeMatch...
Serverless: Uploading function: testeMatch (67.37 MB)...

  Serverless Error ---------------------------------------

  Request must be smaller than 69905067 bytes for the UpdateFunctionCode operation

[HyperBrain]

Hi @mataide , yes, the plugins are not compatible.

The serverless-webpack plugin normally should do the optimal optimization of your modules, because it only packages modules that are actually referenced from your code. 69MB sounds too big, and I assume that there's something wrong with the configuration.

According to your log, it only packages 3 modules: Serverless: Packing external modules: source-map-support@^0.5.4, babel-polyfill@^6.26.0, composer-client@^0.18.1 plus the compiled sources.

Can you check if, after a serverless package the generated zip files in .serverless only contain these modules and the sources and not any additional stuff? If they do, then it would be great if you could post your webpack configuration (and the configuration from serverless.yml).

[HyperBrain]

One additional improvement would be that you bundle the composer-client through webpack so that it strips all unused code and only integrates the 2nd level modules. This can be done by configuring node-externals with nodeExternals({ whitelist: [ 'composer-client' ] }). But I'd keep that option open until it is clear that it's not some other problem.

[mataide]

Yes, .serverless have additional modules at the root of node_modules. I think that all dependencies modules from 'composer-client' are being duplicated and added to the root of node_modules.

webpack.config.js

const path = require('path')
const slsw = require('serverless-webpack')
const nodeExternals = require('webpack-node-externals')
const CopyWebpackPlugin = require('copy-webpack-plugin')

module.exports = {
  entry: slsw.lib.entries,
  target: 'node',
  mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
  optimization: {
    // We no not want to minimize our code.
    minimize: false
  },
  performance: {
    // Turn off size warnings for entry points
    hints: false
  },
  plugins: [
    new CopyWebpackPlugin([{
      context: `./api/vendor/ibm`,
      from: '**/*',
      to: './ibm',
      force: true
    },{
      context: `./api/lib`,
      from: '**/*',
      to: './lib',
      force: true
    }], {
      copyUnmodified: true
    })
  ],
  devtool: 'nosources-source-map',
  externals: [nodeExternals( { whitelist: [ 'le-challenge-s3', 'le-store-s3'] } ) ],
  module: {
    rules: [
      {
        test: /\.js$/,
        exclude: /node_modules/,
        use: [
          {
            loader: 'babel-loader'
          }
        ],
      }
    ]
  },
  output: {
    libraryTarget: 'commonjs2',
    path: path.join(__dirname, '.webpack'),
    filename: '[name].js',
    sourceMapFilename: '[file].map'
  }
}

Serverless.yml

 webpack:
    webpackConfig: 'webpack.config.js'   # Name of webpack configuration file
    includeModules:
      forceExclude:
        - aws-sdk
    packager: 'npm'   # Packager that will be used to package your external modules
    packagerOptions:
      scripts:
        - npm rebuild grpc --target=6.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc

package:
  excludeDevDependencies: true
  individually: ${self:custom.individually.${self:provider.environment.PROFILE}}
  exclude:
    - .gitignore
    - .git/**

.babelrc

{
  "comments": false,
  "presets": [
    ["env", {
      "targets": {
        "node": "8.10"
      }
    }],
    "stage-2"
  ],
  "plugins": [
    "source-map-support"
  ]
}

I also tested with nodeExternals({ whitelist: [ 'composer-client' ] }) It reduces the code to 29mb, but gives me an error: Unable to import module 'api/teste/match/handler': Error

[HyperBrain]

I think that all dependencies modules from 'composer-client' are being duplicated and added to the root of node_modules.

That should be the default behavior of npm. It optimizes the packages in node_modules on install and flattens the folder tree. So that should be ok.

Unable to import module 'api/teste/match/handler': Error

The reason for the error is most likely that the composer client does dynamic imports/requires and webpack is not able to detect all dependencies. So the load of your api/teste/match/handler fails. What you could do is, only build the zips with serverless package. Then extract the zip file somewhere, enter the root of the extracted files and call your function handler with node from the command line. That should reveal the exact error message and show which module is not found that is required by your match/handler file. In AWS Lambda, the reason cannot be shown, because you cannot see the logs of the AWS Lambda bootstrapper itself. After you know which one is missing, you can add it to dependencies and use includeModules: forceInclude: ... to make it included.

Is there any chance that you could provide a small sample project that allows to reproduce or analyze the problem?

[mataide]

I just put the demo project on: https://github.com/mataide/serverless-hyperledger.

Yes, I tried a lot of difference configs but nothing seems to work. Invoke Locally the project works perfect.

[marckaraujo]

@HyperBrain, its possible to force include modules only to specific functions?

I reduced the code a little, now it have 53MB, but now AWS allow to upload.

[HyperBrain]

@marckaraujo Currently forceInclude is a global setting of the webpack configuration but I agree that having the configuration on a function level too, would simplify things and optimize the packages in a better way. Feel free to create a feature request for setting forceInclude and forceExclude on a function level.

[mataide]

@HyperBrain, ok good, let me have some free time and could help you with that.

One other thing,

Is the optimization of serverless deploy is exactly the same as for serverless deploy function?

I ask this because if I do a serverless deploy function I got this error:

Serverless Error ---------------------------------------

  Request must be smaller than 69905067 bytes for the UpdateFunctionCode operation

But with serverless deploy it works well.

You can try with the same demo project.

[HyperBrain]

Yes, deploy function should do exactly the same as deploy, only with the compile restricted to one function. It looks like the Serverless native packaging has been invoked somehow - maybe there was a change in Serverless in the latest version that triggers the behavior. Previously, it did work and I do a lot of debugging with function deploy with our projects to deploy a debug version. I never had that issue in the past.

[mataide]

Do you think it is a serverless bug? I saw this topic: https://github.com/serverless/serverless/issues/2494 They said that isnt a bug, but I dont agree.

As you said both commands should do exactly the same.

[HyperBrain]

Do you think it is a serverless bug?

I'm not quite sure if it is relates to the issue you posted. Maybe yes, but then it would be a different bug/change in Serverless, because it looks like Serverless' native packaging was invoked somehow. This should not be, because serverless-webpack should do the whole packaging and Serverless should just use the generated zip file instead of creating its own one.

A good start to debug what's exactly happening here is to run SLS_DEBUG="*" serverless package --verbose. This will output more information and we should see if Serverless disrupts the generated package or uses it correctly. Depending on the output we can decide where to look further.

[mataide]

Marcuss-MacBook-Pro:serverless-hyperledger marcusataide$ serverless package --verbose
Serverless: Using configuration:
{
  "webpackConfig": "webpack.config.js",
  "includeModules": {
    "forceInclude": [
      "composer-wallet-filesystem",
      "composer-connector-hlfv1"
    ],
    "forceExclude": [
      "aws-sdk",
      "node-mailer"
    ]
  },
  "packager": "npm",
  "packagerOptions": {
    "scripts": [
      "npm rebuild grpc --target=8.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc"
    ]
  }
}
Serverless: Removing /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack
Serverless: Using multi-compile (individual packaging)
Serverless: Bundling with Webpack...
Time: 1644ms
Built at: 2018-05-15 23:46:50
                                                                                                                      Asset       Size  Chunks             Chunk Names
                                                                .composer/client-data/admin@proak-hyperledger-network/admin  959 bytes          [emitted]  
                                                                                           api/hyperledger/teste/handler.js    104 KiB       0  [emitted]  api/hyperledger/teste/handler
                                                                    .composer/cards/admin@Alps-Network---bf/connection.json   27.9 KiB          [emitted]  
                                                                      .composer/cards/admin@Alps-Network---bf/metadata.json  975 bytes          [emitted]  
                                                            .composer/cards/admin@proak-hyperledger-network/connection.json   26.8 KiB          [emitted]  
                                                    .composer/cards/admin@proak-hyperledger-network/credentials/certificate  733 bytes          [emitted]  
                                                     .composer/cards/admin@proak-hyperledger-network/credentials/privateKey  246 bytes          [emitted]  
                                                              .composer/cards/admin@proak-hyperledger-network/metadata.json   78 bytes          [emitted]  
                                                                                  .composer/cards/adminCard/connection.json   26.8 KiB          [emitted]  
                                                                          .composer/cards/adminCard/credentials/certificate  733 bytes          [emitted]  
                                                                           .composer/cards/adminCard/credentials/privateKey  246 bytes          [emitted]  
                                                                                    .composer/cards/adminCard/metadata.json   69 bytes          [emitted]  
                                                                                         .composer/cards/ca/connection.json   26.8 KiB          [emitted]  
                                                                                           .composer/cards/ca/metadata.json   64 bytes          [emitted]  
                                                                  .composer/client-data/admin@proak-hyperledger-network.zip   2.09 KiB          [emitted]  
.composer/client-data/admin@proak-hyperledger-network/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-priv  246 bytes          [emitted]  
 .composer/client-data/admin@proak-hyperledger-network/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-pub  182 bytes          [emitted]  
                                                                                       api/hyperledger/teste/handler.js.map   21.2 KiB       0  [emitted]  api/hyperledger/teste/handler
                      .composer/client-data/adminCard/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-priv  246 bytes          [emitted]  
                       .composer/client-data/adminCard/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-pub  182 bytes          [emitted]  
                                                                                      .composer/client-data/adminCard/admin  959 bytes          [emitted]  
                .composer/client-data/PeerAdmin@hlfv1/114aab0e76bf0c78308f89efc4b8c9423e31568da0c340ca187a9b17aa9a4457-priv  246 bytes          [emitted]  
                 .composer/client-data/PeerAdmin@hlfv1/114aab0e76bf0c78308f89efc4b8c9423e31568da0c340ca187a9b17aa9a4457-pub  182 bytes          [emitted]  
                                                                            .composer/client-data/PeerAdmin@hlfv1/PeerAdmin      1 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180403.log  523 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180411.log  523 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180412.log  845 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180414.log  684 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180420.log  558 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180503.log  558 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180508.log   4.85 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180514.log   25.9 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180515.log   5.75 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180517.log   1.88 KiB          [emitted]  
Entrypoint api/hyperledger/teste/handler = api/hyperledger/teste/handler.js api/hyperledger/teste/handler.js.map
 [0] external "composer-common" 42 bytes {0} [built]
 [2] external "source-map-support/register" 42 bytes {0} [built]
 [3] ./node_modules/composer-client/lib/transactionregistry.js 9.92 KiB {0} [built]
 [4] ./node_modules/composer-client/lib/participantregistry.js 7.85 KiB {0} [built]
 [5] ./node_modules/composer-client/lib/identityregistry.js 5.13 KiB {0} [built]
 [6] ./node_modules/composer-client/lib/historian.js 5.63 KiB {0} [built]
 [7] ./node_modules/composer-client/lib/assetregistry.js 7.6 KiB {0} [built]
 [8] ./node_modules/composer-client/package.json 4.2 KiB {0} [built]
[11] ./node_modules/composer-client/lib/businessnetworkconnection.js 36.1 KiB {0} [built]
[12] (webpack)/buildin/module.js 497 bytes {0} [built]
[13] ./node_modules/composer-client/index.js 1.95 KiB {0} [built]
[14] ./model/match.js 2.9 KiB {0} [built]
[15] ./lib/response.js 2.27 KiB {0} [built]
[16] ./api/hyperledger/teste/post/action.js 531 bytes {0} [built]
[17] ./api/hyperledger/teste/handler.js 727 bytes {0} [built]
    + 3 hidden modules
Time: 1604ms
Built at: 2018-05-15 23:46:50
                                                                                                                      Asset       Size  Chunks             Chunk Names
                                                                .composer/client-data/admin@proak-hyperledger-network/admin  959 bytes          [emitted]  
                                                                                                api/network/ping/handler.js    104 KiB       0  [emitted]  api/network/ping/handler
                                                                    .composer/cards/admin@Alps-Network---bf/connection.json   27.9 KiB          [emitted]  
                                                                      .composer/cards/admin@Alps-Network---bf/metadata.json  975 bytes          [emitted]  
                                                            .composer/cards/admin@proak-hyperledger-network/connection.json   26.8 KiB          [emitted]  
                                                    .composer/cards/admin@proak-hyperledger-network/credentials/certificate  733 bytes          [emitted]  
                                                     .composer/cards/admin@proak-hyperledger-network/credentials/privateKey  246 bytes          [emitted]  
                                                              .composer/cards/admin@proak-hyperledger-network/metadata.json   78 bytes          [emitted]  
                                                                                  .composer/cards/adminCard/connection.json   26.8 KiB          [emitted]  
                                                                          .composer/cards/adminCard/credentials/certificate  733 bytes          [emitted]  
                                                                           .composer/cards/adminCard/credentials/privateKey  246 bytes          [emitted]  
                                                                                    .composer/cards/adminCard/metadata.json   69 bytes          [emitted]  
                                                                                         .composer/cards/ca/connection.json   26.8 KiB          [emitted]  
                                                                                           .composer/cards/ca/metadata.json   64 bytes          [emitted]  
                                                                  .composer/client-data/admin@proak-hyperledger-network.zip   2.09 KiB          [emitted]  
.composer/client-data/admin@proak-hyperledger-network/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-priv  246 bytes          [emitted]  
 .composer/client-data/admin@proak-hyperledger-network/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-pub  182 bytes          [emitted]  
                                                                                            api/network/ping/handler.js.map   20.3 KiB       0  [emitted]  api/network/ping/handler
                      .composer/client-data/adminCard/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-priv  246 bytes          [emitted]  
                       .composer/client-data/adminCard/17d5fb97dec0b8d90b4793c4ddb12c04cbde6d3e80ce083d7df85d2de84d8573-pub  182 bytes          [emitted]  
                                                                                      .composer/client-data/adminCard/admin  959 bytes          [emitted]  
                .composer/client-data/PeerAdmin@hlfv1/114aab0e76bf0c78308f89efc4b8c9423e31568da0c340ca187a9b17aa9a4457-priv  246 bytes          [emitted]  
                 .composer/client-data/PeerAdmin@hlfv1/114aab0e76bf0c78308f89efc4b8c9423e31568da0c340ca187a9b17aa9a4457-pub  182 bytes          [emitted]  
                                                                            .composer/client-data/PeerAdmin@hlfv1/PeerAdmin      1 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180403.log  523 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180411.log  523 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180412.log  845 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180414.log  684 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180420.log  558 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180503.log  558 bytes          [emitted]  
                                                                                          .composer/logs/trace_20180508.log   4.85 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180514.log   25.9 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180515.log   5.75 KiB          [emitted]  
                                                                                          .composer/logs/trace_20180517.log   1.88 KiB          [emitted]  
Entrypoint api/network/ping/handler = api/network/ping/handler.js api/network/ping/handler.js.map
 [0] external "composer-common" 42 bytes {0} [built]
 [2] external "source-map-support/register" 42 bytes {0} [built]
 [3] ./node_modules/composer-client/lib/transactionregistry.js 9.92 KiB {0} [built]
 [4] ./node_modules/composer-client/lib/participantregistry.js 7.85 KiB {0} [built]
 [5] ./node_modules/composer-client/lib/identityregistry.js 5.13 KiB {0} [built]
 [6] ./node_modules/composer-client/lib/historian.js 5.63 KiB {0} [built]
 [7] ./node_modules/composer-client/lib/assetregistry.js 7.6 KiB {0} [built]
 [8] external "ncp" 42 bytes {0} [built]
[12] ./node_modules/composer-client/lib/businessnetworkconnection.js 36.1 KiB {0} [built]
[13] (webpack)/buildin/module.js 497 bytes {0} [built]
[14] ./node_modules/composer-client/index.js 1.95 KiB {0} [built]
[15] ./model/network.js 1.73 KiB {0} [built]
[16] ./lib/response.js 2.27 KiB {0} [built]
[17] ./api/network/ping/get/action.js 746 bytes {0} [built]
[18] ./api/network/ping/handler.js 723 bytes {0} [built]
    + 4 hidden modules
Serverless: Fetch dependency graph from /Users/marcusataide/Documents/Workspace/serverless-hyperledger/package.json
Serverless: Ignoring 6 NPM errors:
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-connector-hlfv1@0.19.5
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-connector-hlfv1@0.19.5
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-wallet-filesystem@0.19.5
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-wallet-inmemory@0.19.5
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-wallet-filesystem@0.19.5
Serverless: => peer dep missing: composer-common@0.19.5, required by composer-wallet-inmemory@0.19.5
Serverless: WARNING: Could not determine version of module composer-wallet-filesystem
Serverless: Adding explicit peers for dependency composer-connector-hlfv1
Serverless: WARNING: Could not determine version of module composer-wallet-filesystem
Serverless: Adding explicit peers for dependency composer-connector-hlfv1
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: composer-common@^0.19.4, source-map-support@^0.5.5, composer-wallet-filesystem, composer-connector-hlfv1@^0.19.5, ncp@^2.0.0
Serverless: Package took [42973 ms]
Serverless: WARNING: Could not determine version of module composer-wallet-filesystem
Serverless: Adding explicit peers for dependency composer-connector-hlfv1
Serverless: Copy modules: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/hyperledgerTeste [6936 ms]
Serverless: Prune: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/hyperledgerTeste [3047 ms]
Serverless: Run scripts: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/hyperledgerTeste [3448 ms]
Serverless: WARNING: Could not determine version of module composer-wallet-filesystem
Serverless: Adding explicit peers for dependency composer-connector-hlfv1
Serverless: Copy modules: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/networkPing [6744 ms]
Serverless: Prune: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/networkPing [3182 ms]
Serverless: Run scripts: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/networkPing [3220 ms]
Serverless: Zip function: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/hyperledgerTeste [10924 ms]
Serverless: Zip function: /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack/networkPing [11037 ms]
Serverless: Packaging service...
Serverless: Remove /Users/marcusataide/Documents/Workspace/serverless-hyperledger/.webpack
Serverless: [serverless-plugin-split-stacks]: Summary: 9 resources migrated in to 3 nested stacks
Serverless: [serverless-plugin-split-stacks]:    Resources per stack:
Serverless: [serverless-plugin-split-stacks]:    - (root): 17
Serverless: [serverless-plugin-split-stacks]:    - APINestedStack: 5
Serverless: [serverless-plugin-split-stacks]:    - PermissionsNestedStack: 2
Serverless: [serverless-plugin-split-stacks]:    - VersionsNestedStack: 2

[mataide]

@HyperBrain This debug helps you to find the problem? I couldnt find why serverless deploy and serverless deploy function isnt the same package.