search

Polymer / tools

Polymer Tools Monorepo
BSD 3-Clause "New" or "Revised" License
430 stars 200 forks source link

build error with 1.6.0 : excluded files still break build process. #2572

Open christophe-g opened 6 years ago

christophe-g commented 6 years ago

Description

Trying to build an app with newly released 1.6.0 version, using exclude.

The build was successful under #878.

I now get the error below - I suspect the individual files to be excluded (below in firebase.json, firebase-database.js,viz-lite.js) are still processed while minifying.

build error: 

info:    (default) Building...

<--- Last few GCs --->

  140560 ms: Mark-sweep 1368.3 (1419.4) -> 1368.3 (1435.4) MB, 1453.7 / 0.0 ms [allocation failure] [GC in old space requested].
  141995 ms: Mark-sweep 1368.3 (1435.4) -> 1368.3 (1435.4) MB, 1435.0 / 0.0 ms [allocation failure] [GC in old space requested].
  143449 ms: Mark-sweep 1368.3 (1435.4) -> 1372.1 (1419.4) MB, 1453.9 / 0.0 ms [last resort gc].
  144905 ms: Mark-sweep 1372.1 (1419.4) -> 1375.9 (1419.4) MB, 1455.4 / 0.0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2d82313cf781 <JS Object>
    2: checkPath [/usr/lib/node_modules/polymer-cli/node_modules/babel-traverse/lib/path/lib/virtual-types.js:~14] [pc=0x20ceca34f5bf] (this=0xa9c77312fa1 <an Object with map 0x217400f65c01>,_ref=0x2ea57e417a11 <a NodePath with map 0x1a01a06a641>,opts=0x2d8231304381 <undefined>)
    3: arguments adaptor frame: 1->2
    4: newFn [/usr/lib/node_modules/polymer-cli/node_modules/babel-traverse/lib/...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [polymer]
 2: 0x10a03cc [polymer]
 3: v8::Utils::ReportApiFailure(char const*, char const*) [polymer]
 4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [polymer]
 5: v8::internal::Factory::NewFixedArray(int, v8::internal::PretenureFlag) [polymer]
 6: v8::internal::TypeFeedbackVector::New(v8::internal::Isolate*, v8::internal::Handle<v8::internal::TypeFeedbackMetadata>) [polymer]
 7: 0xb4e120 [polymer]
 8: 0xb4e719 [polymer]
 9: 0xb50b10 [polymer]
10: 0xb581d6 [polymer]
11: v8::internal::Compiler::Compile(v8::internal::Handle<v8::internal::JSFunction>, v8::internal::Compiler::ClearExceptionFlag) [polymer]
12: v8::internal::Runtime_CompileLazy(int, v8::internal::Object**, v8::internal::Isolate*) [polymer]
13: 0x20cec6f092a7
Aborted

Versions & Environment


### P.S.
It was pretty handy to get feedback from the console (warnings) when files were being excluded from the stream.
christophe-g commented 6 years ago

@usergenic - just wanted to dig a bit more on this.

The file that is creating problem is : https://github.com/PolymerEl/polymer-vizjs/blob/master/build/viz-lite.js. It contains asm code.

It is/should be excluded ("exclude": ["*.min.js","firebase-database.js","viz-lite.js"]).

When I add some logging to the build process(see snippet below - naive debugging addition to matchesExtAndNotExcluded), the file appears to be excluded from the stream.

However, the build process breaks just after : 

...
excluding  .js :  /home/christophe/Programming/Polymer/preignition/bower_components/multi-graph/multi-graph-dot.html_script_0.js
excluding  .js :  /home/christophe/Programming/Polymer/preignition/bower_components/polymer-graphlib/build/graphlib.min.js
excluding  .js :  /home/christophe/Programming/Polymer/preignition/bower_components/polymer-vizjs/build/viz-lite.js

<--- Last few GCs --->

  148753 ms: Mark-sweep 1351.3 (1435.4) -> 1351.3 (1435.4) MB, 1595.3 / 0.0 ms [allocation failure] [GC in old space requested].
  150294 ms: Mark-sweep 1351.3 (1435.4) -> 1351.3 (1435.4) MB, 1540.9 / 0.0 ms [allocation failure] [GC in old space requested].
  151907 ms: Mark-sweep 1351.3 (1435.4) -> 1354.7 (1418.4) MB, 1611.9 / 0.0 ms [last resort gc].
  153535 ms: Mark-sweep 1354.7 (1418.4) -> 1358.3 (1418.4) MB, 1628.3 / 0.0 ms [last resort gc].

<--- JS stacktrace --->

==== JS stack trace =========================================

Security context: 0x2dcf927cf781 <JS Object>
    1: /* anonymous */(aka /* anonymous */) [/home/christophe/Programming/Polymer/polymer-cli/node_modules/babel-traverse/lib/path/family.js:152] [pc=0x2f07e4cbf92f] (this=0x2dcf92704381 <undefined>,_=0x6ee81b35d31 <a Node with map 0x35dcf5ecf741>,i=0)
    2: arguments adaptor frame: 3->2
    3: map [native array.js:~994] [pc=0x2f07e6eeed92] (this=0x1e1f36a4c6d1 <JS Array[8]>,bq=0xc7ed8dcb059 <...

FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
 1: node::Abort() [polymer]

...

Modified version of matchesExtAndNotExcluded for logging purpose.

function matchesExtAndNotExcluded(
    extension: string,
    option: boolean|{exclude?: string[]}) {
  const exclude = typeof option === 'object' && option.exclude || [];
  return (fs: vinyl) => {
    const ret = !!fs.path &&
        fs.relative.endsWith(extension) &&
        !exclude.some((pattern: string) => matcher.isMatch(fs.relative, pattern));
    if(ret && extension === '.js') {
      console.warn('excluding ', extension, ': ', fs.path);
    }        
    return ret    
  };
}
usergenic commented 6 years ago

Sorry I need to document this better. You have to put the exclude array as a property of minify like this. true is implicit when using the object form like so:

{
  "entrypoint": "index.html",
  "shell": "src/apps/preignition-app.html",
  "fragments": [ "..."],
 "sources": [
    "src/**/*",
    "..."
  ],
  "builds": [{
    "name": "bundlemin",
    "bundle": true,
    "addServiceWorker": true,
    "swPrecacheConfig": "sw-precache-config.js",
    "css": { "minify": true },
    "js": { 
        "minify": {
          "exclude": ["*.min.js","firebase-database.js","viz-lite.js"]
        }
    },
    "html": { "minify": true }
  }],
  "extraDependencies": [
    "manifest.json",
    "bower_components/webcomponentsjs/webcomponents-lite.min.js"
  ]
}
usergenic commented 6 years ago

And by "document this better" I mean I need to have some documentation.

usergenic commented 6 years ago

(exclude is an explicit sub-option you can assign independently to js's minify, compile, as well as html and css minify.

usergenic commented 6 years ago

@christophe-g let me know if this change works for you.

christophe-g commented 6 years ago

@usergenic - thanks for the rapid feedback.

Would have been nice, but nope, no change.

What is weird is that matchesExtAndNotExcluded seems to work as expected (e.g. excluding files matching exclude pattern - as the log above shows), but those files are nevertheless processed further down.

For instance, the build itself - before breaking - takes approx 1 min 30 sec, while it was completed in approx 20 sec under #878. This would be a strong indication that minification still runs files marked as excluded.

As oversights are really easy: 

>polymer --version 
>1.6.0

polymer.json

{
  "entrypoint": "index.html",
  "shell": "src/apps/preignition-app.html",
  "fragments": [
...
  ],
  "sources": [
    "src/**/*",
    "scripts/**/*",
    "images/**/*",
    "data/**/*",
    "favicon.ico",
    "bower.json",
    "manifest.json"
  ],
  "builds": [{
    "name": "default"
  },{
    "name": "bundlemin",
    "bundle": true,
    "addServiceWorker": true,
    "swPrecacheConfig": "sw-precache-config.js",
    "css": { "minify": true },
    "js": { "minify": {"exclude": ["*.min.js","firebase-database.js","viz-lite.js"]}},
    "html": { "minify": true }
  }],
  "extraDependencies": [
    "manifest.json",
    "bower_components/webcomponentsjs/webcomponents-lite.min.js",
    "scripts/redux.min.js",
    "scripts/redux-thunk.min.js"
  ]
}
usergenic commented 6 years ago

Something that might be happening is that the default of bundle: true is that it inlines external JavaScript, so the minification is probably actually taking place in terms of the html file that the js is being inlined into. You may need to add the file name(s) in question to the excludes <-note plural property of bundle...

},{
    "name": "bundlemin",
    "bundle": {
      "excludes": ["firebase-database.js", "viz-lite.js"]
    },
    "addServiceWorker": true,
    "swPrecacheConfig": "sw-precache-config.js",
    "css": { "minify": true },
    "js": { "minify": {"exclude": ["*.min.js","firebase-database.js","viz-lite.js"]}},
    "html": { "minify": true }
  }
...

Note that the excludes property of bundler assumes the file names provided are package-relative so make sure to include folder names if nested in the package, like in src/ or whatever. Also, bundler's excludes property doesn't support the matcher '*' wildcards etc yet (https://github.com/Polymer/polymer-bundler/issues/633)

usergenic commented 6 years ago

Note that when running inlined javascript through the minifier, a name like this is produced "my-file.html_01.js" where the ordinal suffix relates the ordinal position in the dom but otherwise there is no naming to help you identify the specific inlined file from these synthetic names. However if you know the html file that firebase-database.js or vis-lite.js is being inlined into you could do "js": {"minify": {"exclude": ["whatever-file.html*"]} and that might avoid passing it through the minifier.

I've filed a separate issue https://github.com/Polymer/polymer-cli/issues/958 to see if I can get the html minifier to honor the exclusions by name of javascript.

christophe-g commented 6 years ago

Hmmm, it was as simple as : 

    "js": { "minify": {"exclude": ["*.min.js","*/firebase-database.js","*/viz-lite.js"]}},

I thought I did test that but it was with previous configuration ... :

"js": { 
        "minify": true ,
        "exclude": ["*.min.js","*/firebase-database.js","*/viz-lite.js"]
    },

Sorry for that !

I keep the issue open as :

And by "document this better" I mean I need to have some documentation.

stale[bot] commented 4 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.