Universal server bundle is not working properly

chrillewoodz commented 7 years ago

Bug Report or Feature Request (mark with an `x`)

- [x] bug report -> please search issues before submitting
- [ ] feature request

Versions.

@angular/cli: 1.3.0-rc.1 node: 6.10.3 os: darwin x64

Repro steps.

Follow these steps here:

https://github.com/angular/angular-cli/wiki/stories-universal-rendering

Add https://github.com/ngx-translate/core to the app.

Build a server bundle.

Test it and you will see the error below.

The log given by the failure.

<some-excluded-path>/ng-boilerplate/node_modules/@ngx-translate/core/src/translate.store.js:1
(function (exports, require, module, __filename, __dirname) { import { EventEmitter } from "@angular/core";
                                                              ^^^^^^
SyntaxError: Unexpected token import
    at createScript (vm.js:56:10)
    at Object.runInThisContext (vm.js:97:10)
    at Module._compile (module.js:542:28)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.require (module.js:497:17)
    at require (internal/module.js:20:19)
    at Object.17 (<some-excluded-path>/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:6560)
    at e (<some-excluded-path>/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:149)
    at Object.2cGb (<some-excluded-path>/chrillewoodz/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:7952)
    at e (<some-excluded-path>/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:149)
    at Object.Zq8w (<some-excluded-path>/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:28947)
    at e (<some-excluded-path>/ng-boilerplate/dist-server/main.75d7fa7aeab5f9b24f61.bundle.js:1:149)

Desired functionality.

Basically the issue here is that node is resolving the wrong module. It's looking inside of the node_modules folder instead of in the vendor file in the server bundle.

Mention any other details that might be useful.

You can download a project with these steps already done:

https://github.com/chrillewoodz/ng-boilerplate/tree/universal

So simply run npm run universal and you will see the error.

cyrilletuzi commented 7 years ago

Same error using @angular/flex-layout

filipesilva commented 7 years ago

@FrozenPandaz @alxhub any idea what the problem is?

cyrilletuzi commented 7 years ago

Is the server bundle allowing all formats (umd, es2015, commonjs and so on) ? Because I noticed flex-layout doesn't have all the package.json entry points compared to other official Angular package, and node requires commonjs compatible librairies.

donoso-eth commented 7 years ago

@chrillewoodz I´ve cloned and wanted to have a look in your repo, nevertheless I don´t found the script universal in the package.json neither following the steps of the https://github.com/angular/angular-cli/wiki/stories-universal-rendering I´ve found the 2nd app in the angular.cli.json??

Can it be part of the problem?

joejordanbrown commented 7 years ago

@chrillewoodz it doesn't look like you have followed the wiki step by step, @ampgular is correct with what he's pointed out, but also your @angular/cli version should be the RC release in your package.json.

You have:

  "devDependencies": {
    "@angular/cli": "^1.1.3"
  }

When you should have:

  "devDependencies": {
    "@angular/cli": "1.3.0-rc.3"
  }

You should also update your global @angular/cli, uninstall using npm -g uninstall @angular/cli make sure you clear the npm cache npm cache clean then install the latest RC version npm -g install @angular/cli@1.3.0-rc.3

chrillewoodz commented 7 years ago

It looks like I didn't push the latest stuff, my bad. Anyway I know others who have gotten the same issue so it's not an issue with my setup. But I will make sure to push it once I get the opportunity.

chrillewoodz commented 7 years ago

There, I've pushed the latest changes. Now you should be able to find a "universal": "ng build --prod && ng build --prod --app 1 && ts-node server", script in the package json and also the cli at the latest rc.3 release.

cyrilletuzi commented 7 years ago

If it can help, I did a blog post with full explanations.

First it will help you about the hash (no need of it), and second the first thing I would consider is to stick to node. ts-node is a great tool, but in this case it adds more complexity to an already complex configuration, you can do the same with just node.

chrillewoodz commented 7 years ago

@cyrilletuzi Have you actually tested it out with @ngx/translate for example? Cuz that's where the issue is coming from.

cyrilletuzi commented 7 years ago

Just tried, I get the same error (the first one).

So you know, there was the exact same error with @angular/flex-layout beta 8, but with the last builds, it's gone, because they updated to the official way Angular modules are build. One main difference is that there is now a es2015 entry point in their package.json, but I don't know if it's what solved the problem, or a change in their build config.

One point which surprised me first is that the server bundle only contains the app code. So Angular packages (@angular/core and so on) and all other packages need to be installed on the server project too. And the error seems to show that the server bundle requests the packages from node_modules in the wrong format (it's es2015, while it should be commonjs).

Shouldn't the CLI produce a vendor bundle too, to be sure everything is here in the good format ?

cyrilletuzi commented 7 years ago

By the way, having to install Angular packages on the server is OK in Node, but it will be impossible with other languages Universal engines (like the one for .NET).

lukaszbachman commented 7 years ago

I'm seeing the same with ng-bootstrap. Any ideas how to fix this? I'm currently investigating using universal for our large project and the issue with integrating 3rd party libraries puts this work on hold.

feloy commented 7 years ago

I've noticed the same problem with ngx-translate: https://github.com/ngx-translate/core/issues/616 A minimal project is available on github for testing: https://github.com/feloy/bug-ngx-translate

hansl commented 7 years ago

We are working on a fix with the libraries themselves. In the meantime, you can use (this is not a supported solution, just a fix) something like https://www.npmjs.com/package/import-export. Alternatively, you can disable AOT in the meantime.

I'm keeping this issue open in the meantime for keeping the discussion.

chrillewoodz commented 7 years ago

Sounds great :) Any early estimate on when this could be resolved? Just so I can plan ahead a bit.

intellix commented 7 years ago

What's the proposed fix for each of the libraries? I'm guessing to export a variety of formats like angular-quickstart-lib? There's a lot of libraries and guess we've all got to do our part by educating and sending PRs to the wrongly exported libraries out there :)

Just now I saw a PR to change the format of angulartics2 to CommonJS which I'm guessing is a wrong fix

niravshah commented 7 years ago

Even when I do my build with the '--no-aot' flag, I still get the same error.

> ng build --prod --no-aot && ngc

Hash: 3527298f2a176234eb92                                                              
Time: 15948ms
chunk    {0} polyfills.550cf10c9aa54b8194c7.bundle.js (polyfills) 177 kB {4} [initial] [rendered]
chunk    {1} main.dc72226c09f60d9253bd.bundle.js (main) 79.7 kB {3} [initial] [rendered]
chunk    {2} styles.d41d8cd98f00b204e980.bundle.css (styles) 69 bytes {4} [initial] [rendered]
chunk    {3} vendor.68fbe47955f62acf4ee4.bundle.js (vendor) 2.56 MB [initial] [rendered]
chunk    {4} inline.42fcf4018d66c4aef552.bundle.js (inline) 0 bytes [entry] [rendered]

> ts-node src/server.ts

/Documents/code/universal-demo/node_modules/ngx-facebook/dist/esm/providers/facebook.js:1
(function (exports, require, module, __filename, __dirname) { import { Injectable } from '@angular/core';
                                                              ^^^^^^
SyntaxError: Unexpected token import

Has anyone else managed to work around this problem? I'd appreciate any help as this is currently completely blocking my plans to go live with my changes.

bliitzkrieg commented 7 years ago

I am also experiencing this issue but for ngx-meta (https://github.com/ngx-meta/core)

J2D2Development commented 7 years ago

We've had the same issue with ng2-page-scroll. In that case, they have a umd version in a 'bundles' subfolder within the node module.

We wrote a simple node script to check main.bundle.js (created when we build for the server) for references to versions of ng2-page-scroll and replace them with the umd version.

This seems to fix the "unexpected token import" error at least- though I'm not sure if it actually works for the end result, as we have many more errors even after getting past this one. Could be worth a try.

The script we used is in this gist for reference: https://gist.github.com/J2D2Development/6f520dd991a6a33c1152aabcdf346790

kirillgroshkov commented 7 years ago

I have the same issue with https://github.com/ngx-translate/core

Can someone clarify how to use import/export fix or if disabling aot helps?

kirillgroshkov commented 7 years ago

Just tried to use import-export, it passes the error place but gives another error:

/node_modules/@angular/compiler/bundles/compiler.umd.js:19601
                ("ort * as " + prefix + " = require("" + importedModuleName + "");"));
                                                     ^^^^^^^^^^^^^^^^^^^^^^^^^^

SyntaxError: Unexpected string

Toanzzz commented 7 years ago

Here is the workaround that I'm currently using in my project: recompile the server output with webpack so that other 3rd party libs got transpile to commonjs for node to read

Add webpack to devDependencies: npm i webpack --save-dev
Add minimal webpack config
```
// webpack.config.js
```

const hash = getHash(); const input = ./dist-server/main.${hash}.bundle; const output = ./dist-server/main.repack.${hash}.bundle.js;

module.exports = { target: 'node', entry: input, output: { filename: output, libraryTarget: 'commonjs' }, }

function getHash() { const fs = require('fs'); const path = require('path');

const files = fs.readdirSync(${process.cwd()}/dist-server); const mainFiles = files.filter(file => file.startsWith('main')); return mainFiles[0].split('.')[1]; }

3. Target the new compiled file

```typescript
// server.ts

//const AppServerModuleNgFactory = require(`./dist-server/main.${hash}.bundle`).AppServerModuleNgFactory;
const AppServerModuleNgFactory = require(`./dist-server/main.repack.${hash}.bundle`).AppServerModuleNgFactory;

Append webpack build step after server build
```
// package.json
```

//"universal": "ng build --prod && ng build --prod --app 1 && ts-node server" "universal": "ng build --prod && ng build --prod --app 1 && webpack -p && ts-node server"



Hope it help

kirillgroshkov commented 7 years ago

@Toanzzz I'm going to try your workaround now. One comment: by using --output-hashing=none you can avoid getHash() function, cause bundle name is always main.bundle.js.

harshes53 commented 7 years ago

if you are using webpack, just whitelist the package causing import syntax error using webpack's externals property. Let ts-loader compile the package.. not the best workaround but it works for me! I had the same issue when using ngx-cookie package. seems like these packages are using ES6 modules syntax while Node.js still uses common.js module syntax.

e.g:

const webpackNodeExternals = require('webpack-node-externals');
...
externals: [webpackNodeExternals({ whitelist: [/ngx-cookie/] })],
...

Hope it helps..

lukaszbachman commented 7 years ago

I tried workaround mentioned by @Toanzzz and quickly ran into this exception: reflect-metadata shim is required when using class decorators I tried importing the reflect-metadata package which seemed to help with this issue, but I quickly ran into another one: Cannot read property 'subscribe' of undefined. I also tried the solution supplied by @harshes53, but this one didn't work at all to me and once again I got syntaxt error on the import token. At this point I called it quits - this issue is quite frustrating and I was even considering dropping use of angular-cli for the sake of some seed project which had universal support built-in. I'm still evaluating Angular 4.x and CLI, so I'm not working under the time pressure, but having such issues with integration of 3rd party libraries is a major concern to me at this point. Anyone has some other ideas how to resolve this?

mrwogu commented 7 years ago

Just tryied @Toanzzz and @harshes53 solutions and I finally received this exception:

$. /node_modules/.bin/ts-node server

<some-excluded-path>\node_modules\zone.js\dist\zone-node.js:365
                callback.apply(applyThis, applyArgs);
                         ^
TypeError: Cannot read property 'create' of undefined
    at <some-excluded-path>\packages\core\src\application_ref.ts:365:1
    at ZoneDelegate.invoke (<some-excluded-path>\node_modules\zone.js\dist\zone-node.js:365:26)
    at Object.onInvoke (<some-excluded-path>\node_modules\@angular\core\bundles\core.umd.js:3922:33)
    at ZoneDelegate.invoke (<some-excluded-path>\node_modules\zone.js\dist\zone-node.js:364:32)
    at Zone.run (<some-excluded-path>\node_modules\zone.js\dist\zone-node.js:125:43)
    at NgZone.run (<some-excluded-path>\node_modules\@angular\core\bundles\core.umd.js:3853:69)
    at PlatformRef_._bootstrapModuleFactoryWithZone (<some-excluded-path>\packages\core\src\application_ref.ts:363:1)
    at PlatformRef_.bootstrapModuleFactory (<some-excluded-path>\node_modules\@angular\core\bundles\core.umd.js:4509:21)
    at renderModuleFactory (<some-excluded-path>\node_modules\@angular\platform-server\bundles\platform-server.umd.js:2405:39)
    at Object.<anonymous> (<some-excluded-path>\server.ts:16:1)
    at Module._compile (module.js:570:32)
    at Module.m._compile (<some-excluded-path>\node_modules\ts-node\src\index.ts:392:23)
    at Module._extensions..js (module.js:579:10)
    at Object.require.extensions.(anonymous function) [as .ts] (<some-excluded-path>\node_modules\ts-node\src\index.ts:395:12)
    at Module.load (module.js:487:32)

harshes53 commented 7 years ago

@mrwogu do you have a repo where I can re-produce this and help you fix it?

ghost commented 7 years ago

Here's a universal project with ngx-translate added to demonstrate the import error. Neither of the work arounds above worked for me either.

https://github.com/vanaio/preRenderImportError

dimitriy-k commented 7 years ago

same issue with here with ng-bootstrap

BlankHrt commented 7 years ago

Same Issue .Any good suggestions ?

AnthonyNahas commented 7 years ago

i fix similar issue with another npm module (@ngrx). take a look to my solution (only as work around)

#581

stephanegg commented 7 years ago

Same issue. @hansl any updates or ETA? @AnthonyNahas's answer is a good work around

lichunbin814 commented 7 years ago

@AnthonyNahas I can't open your link to know how you sloved it , can I have other link to slove issue of npm module ?

stephanegg commented 7 years ago

@lichunbin814 the href is wrong in @AnthonyNahas comment. https://github.com/ngx-translate/core/issues/581#issuecomment-326256510

AnthonyNahas commented 7 years ago

@stephanegg thank u @lichunbin814 fixed the url too

intellix commented 7 years ago

@kirillgroshkov did you ever manage to fix the issue that you get after installing import-export? Getting the same thing about the unexpected string regarding:

/node_modules/@angular/compiler/bundles/compiler.umd.js:19810

("ort * as " + prefix + " = require("" + importedModuleName + "");"));

Interestingly, this is the code that it's complaining about:

converter.importsWithPrefixes.forEach(function (prefix, importedModuleName) {
    // Note: can't write the real word for import as it screws up system.js auto detection...
    preambleLines.push("imp" +
        ("ort * as " + prefix + " from '" + importedModuleName + "';"));
});

BlankHrt commented 7 years ago

same issue with ng-cookies. hello ,i followed your advice ,but not working @harshes53 externals: [nodeExternals({whitelist: [/ngx-cookie/]})],

dannypritchard commented 7 years ago

Is there any resolution on this issue yet? Or an ETA? Neither the require('import-export') or the webpack workarounds suggested above have worked for me; the first throws errors when using the @ng-bootstrap datepicker module (Unexpected token export from the npm), and the second throws various @NgModule annotation errors that I can't seem to work around at all (I've played whack-a-mole with them for hours now).

We really need a way to bundle all node_modules for server-side rendering.

freehuoshan commented 7 years ago

https://medium.com/@cyrilletuzi/angular-server-side-rendering-in-node-with-express-universal-engine-dce21933ddce

kirillgroshkov commented 7 years ago

@intellix No, I gave up on SSR with angular. Went with prerendering approach (instead of SSR), used Puppeteer to go through all URLs, render the page, save resulting html file, then served html files as static. Drawback is that prerendering is significantly slower (even in multi-threaded environment).

For future project I'll consider a framework (not Angular) that has SSR fixed (e.g React/Next.js, or stenciljs, or VueJS).

Toxicable commented 7 years ago

I've been doing a bit of investigation on this the last few days trying a few solutions but havn't come up with anything that'll do it easily.

Just to reiterate the actual issue here; The Library being used is for packaged/formatted correctly. For Universal to be able to use a node package is must be commonjs formatted, meaning that it uses require not import since node does not currently support esm modules (import/export). Which means this is not an issue with the CLI, it's an issue with the libraries not packaging their modules correctly.

Work around i've tried: require('import-export')

This module just dosen't support all the right syntax for to be useful here.

Node nightly build

While node does actually have inital experimental support for esm modules as you can see here https://github.com/nodejs/node/commit/c8a389e19f172edbada83f59944cad7cc802d9d5 after trying their latest nightly build: https://nodejs.org/download/nightly/ it dosen't appear to be apart of node just yet

re-packaging libs

I had some success with this: I was able to use tsc to override libs into the correct commonjs format and successfully used that lib in a Universal project. However this is not a good solution at all, it worked for the lib I was testing on but I don't know if it will work for all, and it's a pretty ugly lots of scripts

From my investigations my conclusion is that the best way to work around this is getting libraries to correctly format their packages so that they can be consumed in a node environment without issue

dannypritchard commented 7 years ago

Thanks for looking into this Toxicable

Are you saying that there is no intention for the angular/cli team to allow packaging of 3rd party node_modules imported into a project to run using Angular Universal and that library maintainers will need to convert all packages to commonjs format for distribution in order for them to be used in server side rendering? Because that's a pretty big ask of the vendors, and will, for a team like ours who are using multiple vendor packages, mean we won't be able to go live with our Angular Universal project for potentially months whilst the 3rd parties sort this out

chrillewoodz commented 7 years ago

Not to mention a lot of third party authors are lazy and don't care. So it would require a pull request from an outsider and that is an even bigger ask.

Toxicable commented 7 years ago

@dannyrevenant Please don't twist what i've said into your own opinion, No that is not what I'm saying.

My words are only of my own opinion, I do not speak on behalf of any groups, I am just a regular dev that enjoys contributing to Angular and surrounding communities, most notably Universal as of recent.

As for what the CLI team can do; The CLI does support importing modules from npm into a Universal project, take Material for example, that works with Universal fine since it's correctly packaged, exemplifying that this isn't an issue with the CLI, instead an issue with the 3rd parties packaging format. However I believe there is some webpack magic that they might be able to configure to allow for importing incorrectly formatted packages but it's not obvious, simple or a good solution, I did not investigate this route since I was not looking for a solution by changing webpack config.

For your mention on packaging formats; no I am not suggesting that libs package their packages solely for Universal, if you would take some time to lookup how libs should be distributing packages then you'd find that you can and should distribute your package in multiple formats so that different formats can be correctly resolved for different situations.

The best thing you can do to getting a resolution to this issue is ask maintainers of the projects with incorrect packaging formats to correct it, or better yet, whip up a PR youreself, I'm sure they'll appreciate it.

stephanegg commented 7 years ago

At the moment no guidelines were provided regarding the "packaging format" making a lib working with Angular Universal. If someone could write these guidelines, I would be happy to help filling these guidelines and provide PRs to some libs to move things along. Most of Angular application needs Universal working to go live, we need to fill this SSR gap as soon as possible. @Toxicable do you have at least some bullet points to provide as you deeply investigate the problem? Thank you for your clear explanations by the way

Toxicable commented 7 years ago

@stephanegg See here for guidelines https://docs.google.com/document/d/1CZC2rcpxffTDfRDs6p1cfbmKNLA6x5O-NtkJglDaBVs/preview here for a lib quick start which does the right format https://github.com/filipesilva/angular-quickstart-lib and here for a tool which can output a lib in the right format https://github.com/dherges/ng-packagr

Toxicable commented 7 years ago

Small update here, looks like node has actually released support behind the flag --experimental-modules on v8.5 which was released today https://nodejs.org/en/blog/release/v8.5.0/

So I attempted to run Universal using this flag but ran into a few issues

You cant use import and require in the same file

When using Node's implementation of esm your files must be have the extension .mjs for Node to resolve them, meaning that Libs would have to be packaged in another format before you can take advantage of this feature. It will not be a in-place fix

Which reaffirms my point from earlier that:

the best way to work around this is getting libraries to correctly format their packages so that they can be consumed in a node environment without issue

Toxicable commented 7 years ago

More investigation!

So I think I have something more definitive this time. I noticed that a lot of the packages had UMD bundles, however for some reason they were not being resolved into the Universal application. TLDR: libs that don't produce a FESM build casue cli to produce deep imports where the umd bundle cannot be resolved.

I made a sample repository here showing casing 2 libs. lib1 - a properly formatted lib which can work with universal without problem lib2 - the average kind of lib that you see which runs into Universal issues

The big difference between them is that lib2 will produce a build output where the JS files that use import/export are not concatenated together or in other words are not flat es modules, lib1 does produce a flat esm build.

If you run the demo with the command below (sorry about all the dependency installs) then you'll notice that lib2 throws the error we've seen so much in this thread.

SyntaxError: Unexpected token import

Well if you take a look at app/dist/dist-server/main.bundle.js and a ctrl + f you'll see that there have 2 entries: require("lib2/src/my-module") and require("lib1") AHA! Now we're getting somewhere. So we know that require("lib1") will resolve to the umd bundle but a deep import like require("lib2/src/my-module") has no option to be to resolve the es module, therefore casuing our issue!

What can be done?

As I stated above the best way to fix this issue is for library owners to distribute packages in the correct format, which as I has discovered includes a Flat ES Module option.
There may be something the CLI can do to avoid producing these deep imports but I have no knowledge of that side of their build system. @hansl Any idea if something can be done?

https://github.com/Toxicable/universal-deep-import

cd lib1 && yarn && cd ../lib2 && yarn && cd ../app && yarn && npm run build && npm run build:static

genyklemberg commented 7 years ago

Hi, I have no problems with all the steps starting from "npm run start", but on last step "npm run server" (means - node dist/server) my server do nothing with no erros http://prntscr.com/glippe? Can somebody help me with that?

here is my repo https://github.com/genyklemberg/Universal_Project

MarkPieszak commented 7 years ago

One workaround for now is to basically make your server.js file a TypeScript file, and have Webpack setup to process /.ts$/ included your externals, which are things like node_modules / etc.

Take a look at the comment I made here for more info: https://github.com/angular/universal-starter/issues/428#issuecomment-331558400

Hopefully we can have this as an example in the universal-starter CLI demo as well soon.

angular / angular-cli