Hangs on additional asset processing

[damianobarbati]

Works fine without babili-webpack-plugin

Repro:

git clone https://github.com/damianobarbati/mdi/tree/v1.0.14
yarn install
yarn build:prod

[damianobarbati]

Can anybody help here? I did set up a very simple repro: all is needed to investigate is cloning, installing and running yarn build:prod https://github.com/damianobarbati/mdi/tree/v1.0.20

😞

[wasd171]

I have the same problem :(

[damianobarbati]

can you help us @d3viant0ne?

[alexander-akait]

@damianobarbati what is exactly hangs in additional asset? Just freeze? Can your remove some plugins, code and etc to understand why this is happening?

[damianobarbati]

I mean the webpack step additional asset processing Here the full output for reference. Thanks for helping @evilebottnawi :)

imac:mdi damz$ yarn build:prod
yarn build:prod v0.24.6
$ NODE_ENV=production webpack --config ./config/webpack.config.js --progress 
clean-webpack-plugin: /Users/damz/Desktop/mdi/build has been removed.
clean-webpack-plugin: /Users/damz/Desktop/mdi/dist has been removed.
 10% building modules 0/1 modules 1 active ...!/Users/damz/Desktop/mdi/src/index.jskeyword select requires $data option
babel-preset-env: `DEBUG` option

Using targets:
{
  "chrome": "59"
}

Modules transform: false

Using plugins:

Using polyfills:
  web.timers {"chrome":"59"}
  web.immediate {"chrome":"59"}
  web.dom.iterable {"chrome":"59"}
[BABEL] Note: The code generator has deoptimised the styling of "/Users/damz/Desktop/mdi/src/index.js" as it exceeds the max of "500KB".
 91% additional asset processing          

[alexander-akait]

@damianobarbati very strange, wip

[alexander-akait]

@damianobarbati Which command to run to reproduce problem?

[damianobarbati]

git clone https://github.com/damianobarbati/mdi
cd mdi
yarn install
yarn build #here I'm downloading some stuff to build the src/index.js and svg files
yarn build:prod #here webpack is fired

[alexander-akait]

@damianobarbati in next please describe this steps in readme, thanks

[damianobarbati]

Got it, sorry for the current broken readme: fixing in next push

[alexander-akait]

@damianobarbati using your steps, my output:

yarn build:prod v0.23.2
$ NODE_ENV=production webpack --config ./config/webpack.config.js --progress 
clean-webpack-plugin: /home/evilebottnawi/IdeaProjects/mdi/build has been removed.
clean-webpack-plugin: /home/evilebottnawi/IdeaProjects/mdi/dist has been removed.
 10% building modules 0/1 modules 1 active ...ottnawi/IdeaProjects/mdi/src/index.jskeyword select requires $data option
babel-preset-env: `DEBUG` option

Using targets:
{
  "chrome": "59"
}

Modules transform: false

Using plugins:

Using polyfills:
  web.timers {"chrome":"59"}
  web.immediate {"chrome":"59"}
  web.dom.iterable {"chrome":"59"}                                           Hash: 38ead94a9b51b36ad883                                                         
Version: webpack 3.4.1
Time: 897ms
               Asset       Size  Chunks             Chunk Names
   dist/index.min.js  735 bytes       0  [emitted]  main
dist/index.min.js.gz  378 bytes          [emitted]  
   [0] ./src/index.js 0 bytes {0} [built]
Done in 1.91s.

[alexander-akait]

@damianobarbati Can your provide your nodejs version?

[damianobarbati]

Sure thing.

imac:mdi damz$ node -v
v8.2.1
imac:mdi damz$ cat package.json 
{
    "name": "@damianobarbati/mdi",
    "version": "1.0.20",
    "license": "MIT",
    "author": "Damiano Barbati <damiano.barbati@gmail.com> (http://github.com/damianobarbati)",
    "main": "src/index.js",
    "module": "src/index.js",
    "jsnext:main": "src/index.js",
    "scripts": {
        "jslint": "eslint **/*.js --ignore-path .gitignore --ignore-pattern **/*.min.js .",
        "build:dev": "NODE_ENV=development webpack --config ./config/webpack.config.js --progress --watch",
        "build:stg": "NODE_ENV=staging webpack --config ./config/webpack.config.js --progress",
        "build:prod": "NODE_ENV=production webpack --config ./config/webpack.config.js --progress",
        "build": "sh build.sh",
        "build2": "sh build2.sh"
    },
    "browserlist": [
        "chrome 59"
    ],
    "eslintConfig": {
        "parser": "babel-eslint",
        "parserOptions": {
            "ecmaVersion": 2017,
            "sourceType": "module",
            "impliedStrict": true,
            "ecmaFeatures": {
                "jsx": true,
                "impliedStrict": true,
                "globalReturn": false,
                "experimentalObjectRestSpread": true
            }
        },
        "env": {
            "browser": true,
            "node": true,
            "jest": true,
            "es6": true
        },
        "plugins": [
            "import",
            "react",
            "jest"
        ],
        "extends": [
            "eslint:recommended",
            "plugin:react/recommended",
            "plugin:jest/recommended"
        ],
        "rules": {
            "no-console": "off",
            "no-unused-vars": "off",
            "no-unsafe-finally": "off",
            "no-unreachable": "off",
            "react/no-unescaped-entities": "off",
            "react/no-children-prop": "off"
        }
    },
    "babel": {
        "presets": [
            [
                "env",
                {
                    "targets": {
                        "browsers": [
                            "chrome 59"
                        ]
                    },
                    "modules": false,
                    "useBuiltIns": true,
                    "debug": true
                }
            ],
            "stage-0",
            "react"
        ],
        "plugins": [
            "transform-decorators-legacy",
            "transform-class-properties",
            [
                "react-css-modules",
                {
                    "filetypes": {
                        ".scss": {
                            "syntax": "postcss-scss"
                        }
                    }
                }
            ]
        ]
    },
    "peerDependencies": {
        "material-ui": "1.0.0-beta.2",
        "react": "^15.6.1"
    },
    "dependencies": {
        "babel-eslint": "^7.2.3",
        "babel-loader": "^7.0.0",
        "babel-plugin-react-css-modules": "^3.0.0",
        "babel-plugin-transform-class-properties": "^6.24.1",
        "babel-plugin-transform-decorators-legacy": "^1.3.4",
        "babel-preset-env": "^1.5.1",
        "babel-preset-latest": "^6.24.1",
        "babel-preset-react": "^6.24.1",
        "babel-preset-stage-0": "^6.24.1",
        "babili-webpack-plugin": "^0.1.1",
        "case-sensitive-paths-webpack-plugin": "^2.1.1",
        "clean-webpack-plugin": "^0.1.16",
        "compression-webpack-plugin": "^1.0.0",
        "eslint": "^4.1.1",
        "eslint-loader": "^1.7.1",
        "eslint-plugin-import": "^2.3.0",
        "eslint-plugin-jest": "^20.0.3",
        "eslint-plugin-react": "^7.0.1",
        "file-loader": "^0.11.1",
        "glob": "^7.1.2",
        "lodash": "^4.17.4",
        "prop-types": "^15.5.10",
        "string-replace-loader": "^1.3.0",
        "webpack": "^3.0.0"
    }
}

But @evilebottnawi I'm checking the commands because this is impossible: 3mb of svg files emitted in ~800bytes?

Did you run yarn build before yarn build:prod? :|

[damianobarbati]

My current output with the steps => 🤔

imac:Desktop damz$ git clone https://github.com/damianobarbati/mdi
Cloning into 'mdi'...
remote: Counting objects: 11549, done.
remote: Total 11549 (delta 0), reused 0 (delta 0), pack-reused 11549
Receiving objects: 100% (11549/11549), 6.67 MiB | 2.71 MiB/s, done.
Resolving deltas: 100% (8608/8608), done.
imac:Desktop damz$ cd mdi/
imac:mdi damz$ yarn install
yarn install v0.24.6
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 4.64s.
imac:mdi damz$ yarn build
yarn build v0.24.6
$ sh build.sh 
Cloning into 'material-design-icons'...
remote: Counting objects: 42051, done.
remote: Compressing objects: 100% (32101/32101), done.
remote: Total 42051 (delta 10321), reused 40199 (delta 9947), pack-reused 0
Receiving objects: 100% (42051/42051), 15.54 MiB | 3.46 MiB/s, done.
Resolving deltas: 100% (10321/10321), done.
Checking out files: 100% (90943/90943), done.
Cloning into 'MaterialDesign'...
remote: Counting objects: 4183, done.
remote: Compressing objects: 100% (3202/3202), done.
remote: Total 4183 (delta 1078), reused 4007 (delta 979), pack-reused 0
Receiving objects: 100% (4183/4183), 4.52 MiB | 2.62 MiB/s, done.
Resolving deltas: 100% (1078/1078), done.
mkdir: icons: File exists
yarn install v0.24.6
[1/4] 🔍  Resolving packages...
[2/4] 🚚  Fetching packages...
[3/4] 🔗  Linking dependencies...
[4/4] 📃  Building fresh packages...
✨  Done in 2.41s.
./material-design-icons/action/svg/production/ic_3d_rotation_24px.svg Action3DRotation
./material-design-icons/action/svg/production/ic_accessibility_24px.svg ActionAccessibility
etc etc etc
✨  Done in 35.50s.
imac:mdi damz$ yarn build:prod
yarn build:prod v0.24.6
$ NODE_ENV=production webpack --config ./config/webpack.config.js --progress 
clean-webpack-plugin: /Users/damz/Desktop/mdi/build has been removed.
clean-webpack-plugin: /Users/damz/Desktop/mdi/dist has been removed.
 10% building modules 0/1 modules 1 active ...!/Users/damz/Desktop/mdi/src/index.jskeyword select requires $data option
babel-preset-env: `DEBUG` option

Using targets:
{
  "chrome": "59"
}

Modules transform: false

Using plugins:

Using polyfills:
  web.timers {"chrome":"59"}
  web.immediate {"chrome":"59"}
  web.dom.iterable {"chrome":"59"}
 91% additional asset processing                                                            

[alexander-akait]

@damianobarbati yarn build

yarn build v0.23.2
$ sh build.sh 
Cloning into 'material-design-icons'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Cloning into 'MaterialDesign'...
Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
mkdir: cannot create directory ‘icons’: File exists
yarn install v0.23.2
[1/4] Resolving packages...
warning Integrity check: Lock files don't match
[2/4] Fetching packages...
warning fsevents@1.1.2: The platform "linux" is incompatible with this module.
info "fsevents@1.1.2" is an optional dependency and failed compatibility check. Excluding it from installation.
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 2.85s.
failed []
Done in 6.20s.

[damianobarbati]

I don't know what's going on with github today, anyway it's not allowing you to download the public repo with icons =>

Permission denied (publickey).
fatal: Could not read from remote repository.

And it's not letting me push new commits throwing errors about authentication.

You should have the /icons folder in place anyway and yarn build:prod should build the dist file: but it's huge, definitely can't be 800bytes!

[alexander-akait]

@damianobarbati can your change src/index.js to one line export Action3DRotation from '../icons/Action3DRotation'; and try to build?

[alexander-akait]

@damianobarbati Reproduced, looking for a problem

[damianobarbati]

Great! I did some test, with a bunch of lines it is indeed working: once they're more then ~1000 (?) it starts hanging :)

[alexander-akait]

@damianobarbati seems nothing is hands, using 1000 modules i have 195.32s. with disabled source maps. And 100% cpu in this time :smile:

[damianobarbati]

@evilebottnawi I have the sourcemaps disabled as well but it never ends the compilation. Right now it has been stuck at 91% for about 22minutes (since I tried the 1000 and then the full 3000 list).

I'm afraid it's not linear the time required (1k ~ 1min => 3k ~ 3min) but it simply hangs.

And I'm on a 2017 iMac with maxed out "3,6 GHz Intel Core i7-7700" cpu :(

[damianobarbati]

Did you try to compile the full list? And thanks again for your efforts 🤗

[alexander-akait]

@damianobarbati i have very power machine in my office Intel Xeon E5-2699 v4 @ 2.20GHz 2k icons compiling ~25 min :smile: Seems 3k have ~60-120 min, yep not linear, I think your should search other solution for your purpose, this is not acceptable :disappointed:

[damianobarbati]

oh no... 😔

[alexander-akait]

@damianobarbati in other hand your can search way to fix performance in babel plugins, but it is hard way :smile:

[alexander-akait]

@damianobarbati right now my machine (Intel Xeon E5-2699 v4 @ 2.20GHz) is done build:

    + 3000 hidden modules
Done in 1181.66s.

real    19m42,102s
user    19m40,280s
sys     0m2,004s

[alexander-akait]

@damianobarbati seems babel plugins for minify have bad performance, your can try uglify-es as webpack plugins (https://github.com/webpack-contrib/uglifyjs-webpack-plugin), maybe it is be more faster. I can not do anything else here, I'm sorry.

[damianobarbati]

@evilebottnawi thanks so much for investigating this: at least I know it's just about horse power right now. I'll try to leave it the whole night compiling, I always killed it in 30-45mins supposing it was stuck. "sadly" closing this T_T

[wasd171]

I still think there is a bug somewhere. In my project webpack hangs too, but if I kill the process and check the dist folder, all files seem to be minified

[alexander-akait]

@wasd171 please provide minimum reproducible test repo? But seems problems not in babili-webpack-plugin, most likely somewhere in babel plugins. We can catch this only have test repo.

[wasd171]

I don't have any other babel plugins. The repo: https://github.com/wasd171/prettier-chrome Uncomment line https://github.com/wasd171/prettier-chrome/blob/master/webpack.config.js#L29 in webpack.config.js and run npm install && npm run build:prod

[alexander-akait]

@wasd171 i will try this in near future, thanks!

[damianobarbati]

@evilebottnawi I managed to compile (~37mins -_-")!

But now how do I exclude that module (already published compiled) from being processed again from Babili plugin in the project including it? :|

[alexander-akait]

@damianobarbati https://github.com/webpack-contrib/babili-webpack-plugin/issues/50?

[ramsunvtech]

Below settings helped me to reduce the heap memory since it will not include the sourceMap file.

devtool: false, // Webpack.

new webpack.optimize.UglifyJsPlugin({
   sourceMap: shouldUseSourceMap
}),

or below code will help for latest webpack uglify plugin.

> npm install uglifyjs-webpack-plugin --save-dev

// conf.js
const UglifyJsPlugin = require('uglifyjs-webpack-plugin');
module.exports = {
  optimization: {
    minimizer: [
      new UglifyJsPlugin({
        sourceMap: true,
      }),
    ],
  },
};

If this still not help to fix the problem. below command helps to increase heap memory and answer link to know more about heap memory is below as well.

node --max-old-space-size=16000 scripts/build.js

https://stackoverflow.com/a/57480923/1533666