Meteor Webpack as NPM module

[tomitrescak]

Hi

I started to work on porting your functionality to be used as NPM module. I have a "half-working" prototype that I decided not to finish as there are several strategic decisions that need to be made and you guys should be the one that should make them ;) Here are some observations that I had while creating the prototype:

1. I had problems running a global package "I named it 'metpack' ;)" as this package first needs to build the core.js before its first run. As a result, it had to be run with administrator priviledges.
   1. Solution 1, is to create a method 'init' ran as 'sudo metpack init' that would build the core.js in the global directory
   2. Solution 2 is to build the core.js into the project directory (but then it would have to be built for every individual project)
2. Here is imaginary workflow of how the project would be created

sudo npm install -g metpack
sudo metpack init // initialise the core.js (can be omitted)
metpack create myProject // see below what all has to be done

// one of below
metpack // run in dev mode
metpack prod // run in prod mode
metpack deploy <meteor | mup | ... > 

1. The meteor create myProject needs to do following:
   1. Build core.js // unless built globally
   2. Create a new meteor project and delete the scaffolded files
   3. Copy the "meteor.html" template and other meteor files to meteor directory OR instead of 2 and 3 just copy the scaffolded meteor_core directory to project directory
   4. Copy the template App files to the project directory
   5. Copy settinngs files into project directory
   6. Copy package.json into project directory with all the dependencies
   7. run npm install

The prototype package is here but it relies on manually copying application file to the project directory and running npm install (the "create" part).

Also, I am still receiving some babel error. Pretty much do not waste time to understand much of it. I just createdwrappers and set correct directories.

Her is the log:

► node_modules/metpack/bin/metpack.js
["node","/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/metpack/bin/metpack.js"]
/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/metpack/node_modules/core-js/package.json
core-js@1.2.1 without ES6 number constructor is up to date
Hash: a06902057b9ed9d33935
Version: webpack 1.12.2
Time: 1356ms
               Asset    Size  Chunks             Chunk Names
    server.bundle.js  162 kB       0  [emitted]  main
server.bundle.js.map  192 kB       0  [emitted]  main
chunk    {0} server.bundle.js, server.bundle.js.map (main) 153 kB [rendered]
 57% 126/160 build modules[[[[[ ~/Documents/Programming/Github/metpacktest/meteor_core ]]]]]

=> Started proxy.
Hash: a55296488b8f642a4086
Version: webpack 1.12.2
Time: 2164ms
           Asset     Size  Chunks       Chunk Names
client.bundle.js  1.24 MB       0       main
chunk    {0} client.bundle.js (main) 993 kB [rendered]

ERROR in ./app/components/BlazeTemplate.js
Module build failed: SyntaxError: /Users/tomi/Documents/Programming/Github/metpacktest/app/components/BlazeTemplate.js: Unexpected token (5:19)
  3 |
  4 | export default class BlazeTemplate extends React.Component {
> 5 |   static propTypes = {
    |                    ^
  6 |     template: React.PropTypes.any.isRequired,
  7 |     component: React.PropTypes.any,
  8 |   }
    at Parser.pp.raise (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/location.js:24:13)
    at Parser.pp.unexpected (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/util.js:82:8)
    at Parser.pp.parseClassProperty (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/statement.js:624:61)
    at Parser.parseClassProperty (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/plugins/flow.js:797:20)
    at Parser.pp.parseClass (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/statement.js:567:32)
    at Parser.pp.parseExprAtom (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/expression.js:405:19)
    at Parser.parseExprAtom (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/plugins/jsx/index.js:412:22)
    at Parser.pp.parseExprSubscripts (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/expression.js:236:19)
    at Parser.pp.parseMaybeUnary (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/expression.js:217:19)
    at Parser.pp.parseExprOps (/Users/tomi/Documents/Programming/Github/metpacktest/node_modules/babel-loader/node_modules/babel-core/node_modules/babylon/lib/parser/expression.js:163:19)
 @ ./app/components/App.jsx 26:21-47
webpack: bundle is now VALID.
=> Started MongoDB.
=> Started your app.

=> App running at: http://localhost:3000/
I20151008-11:49:15.323(11)?
I20151008-11:49:15.324(11)?
I20151008-11:49:15.324(11)? Running on server only
I20151008-11:49:15.325(11)? There are # posts: 4

[tomitrescak]

If you want to "test" my prototype here is what you have to do:

1. Clone the repo
2. Create a new project directory
3. Change to project directory
4. npm install <path to cloned repo>
5. Now we do everything that the "Create" command would do:
   1. Copy package.json from your repo
   2. npm install
   3. Copy _meteorcore, app and settings directories to the project directory
6. run node_modules/metpack/bin/metpack.js

The server runs with no issues, there is a babel error that I'm not sure why is happening. But otherwise it's all good.

With correct pointers I'm happy to further work on this, but I need you opinion on the matter ;)

Done.

[tomitrescak]

I got another version that deals with core.js by building it in the project directory. The order of creation is changed.

1. Copy package.json to project dir
2. Run npm install
3. Build core.js from project directory 4 copy meteor, example app
4. Done

I agree it's quite silly;) I will just wait for what you got to say and if eventually I should continue or drop dead;)

[tomitrescak]

Just to be clear, this is not a PR cuz I think it's not worth it, and because you don't have a branch for npm. I just wanted to open discussion on this;) I got already 3 projects on webpack and wold love to see upgradable version of this amazing project.

[jedwards1211]

That's cool! Getting started with a new project is always a pain, and that would help.

First of all I think you just need a .babelrc with "stage": 0, to fix the babel error.

Yes, I have been thinking about making it more convenient as well. I don't want to assume that our current app skeleton and run config will be ideal for all users, so my strategy (described here) is totally different. But it would be complimentary to yours :smile:

Basically I want to make an NPM package with Webpack plugins that know how to run the generated bundle(s) in Meteor in dev and prod mode, and would support more varied Webpack config (e.g. code splitting, multiple entry points, etc.) more naturally. And also a runWebpackConfigs method that starts all the Webpack instances easily. Also, I could add a postinstall script to the package that does the core-js build, and a require target for custom core-js/regenerator inside the package. With that the run scripts become almost trivial:

require('shelljs/global');
var dirs = require('./dirs');
var runWebpackConfigs = require('webpack-meteor-tools/runWebpackConfigs');
var configs = require('fs-finder').from(dirs.webpack).findFiles('<.*\.dev.js$>');

runWebpackConfigs(configs, function(err) {
  if (err) throw err;
  cd(dirs.meteor);
  exec('meteor --settings ../settings/devel.json', {async: true});
});

So with this capability:

• People can more easily make changes to their webpack config
• People can more easily drop Meteor support into existing webpack configs
• People can easily make more types of run scripts (this can be useful when tracking down the cause of minified exceptions, for example)
• People can change their webpack/launch config however they want, but still easily get upgrades to the core webpack-meteor-tools

Once I've made this thing, there's nothing to preclude us from using it in this app skeleton or your skeleton generator npm package like you're proposing.

[jedwards1211]

A quick and dirty prototype of the new way I want to do things is available in the https://github.com/jedwards1211/meteor-webpack-react/tree/webpack-meteor-tools branch

[tomitrescak]

Nice! And you were right, my scaffolded app worked as it only missed the .babelrc. I really like where this is going.

In the long run, I think all the webpack files should be hidden away from the developer and only some "web(met)pack.json" should be visible in the project where you configure what you need, such as add or remove loaders. Updating webpack meteor would then be only about updating an NPM module.

[rclai]

Yes. The Npm package is nice but there's still a ton of boilerplate. Perhaps a central json file could work? Would there be limitations to that?

[tomitrescak]

Guys, I got quite far. It is possible to do now:

git clone my project
sudo npm install -g . // installs global module,  imagine . will  be the npm  module
metpack create MyApp
metpack

All seems to go well, but when app is launching, I keep recieving this error:

ERROR in /Users/tomi/Documents/Programming/Github/Tests/Test1/app/main_server.js
Module build failed: ReferenceError: Unknown plugin "react-transform"
    at PluginManager.subnormaliseString (/usr/local/lib/node_modules/metpack/node_modules/babel-core/lib/transformation/file/plugin-manager.js:147:13)
    at PluginManager.add (/usr/local/lib/node_modules/metpack/node_modules/babel-core/lib/transformation/file/plugin-manager.js:190:40)
    at File.buildTransformers (/usr/local/lib/node_modules/metpack/node_modules/babel-core/lib/transformation/file/index.js:237:21)
    at new File (/usr/local/lib/node_modules/metpack/node_modules/babel-core/lib/transformation/file/index.js:139:10)
    at Pipeline.transform (/usr/local/lib/node_modules/metpack/node_modules/babel-core/lib/transformation/pipeline.js:164:16)
    at transpile (/usr/local/lib/node_modules/metpack/node_modules/babel-loader/index.js:12:22)
    at Object.module.exports (/usr/local/lib/node_modules/metpack/node_modules/babel-loader/index.js:69:12)
 @ multi main

And I have no ide why is this happening, since the module is installed below global module and for the hell of it I installed it also in the project directory.

Would you be able to check?

[tomitrescak]

Global-js is now detected and built into the project directory into the hidden folder.

[jedwards1211]

@tomitrescak @rclai there's always some inherent limitation to creating a system where the server/dev/client/prod webpack configs can't be customized individually. Also for any production project one probably wants to configure code splitting, which means one must be able to specify entry points and possibly plugins to use. It's also possible to share more things between the various configs, e.g. require-ing in loaders. Are you guys finding it difficult to tweak the webpack configs with the new branch?

[rclai]

It's fine. I can figure out how to adjust, but I'm just wondering there was a way to reduce the boilerplate.

[jedwards1211]

@rclai Sounds like you'll prefer to use Benoit's project, as he likes to make it handle everything automatically like this. I don't really like that approach as it hinders flexibility (for instance in his new system there's currently no way to use 0.0.0.0 instead of localhost as the host) but he'll probably get that stuff ironed out. I'm getting kind of tired of trying to keep up with all this stuff/please everyone.

[rclai]

No problem. I'm still going to continue using this project because of its flexibility though.

[jthomaschewski]

I also really like the flexibility of this project - for me as a webpack newbie it was nice to understand whats happening under the hood to archive meteor compatibility and being able to adjust the scripts and webpack config myself.

But I also like the new approach of benoit (15h ago) - as he somehow completely integrated the webpack stuff into meteor - And some of the (few) problems with this skeleton seems to be fixed in his.

It's great to have multiple webpack skeletons with different approaches: This hopefully helps to find fixes for for the remaining issues, increases the webpack-meteor userbase and draws attention to the meteor core team. Maybe in the future we get official NPM, import/export and maybe even webpack support.

@jedwards1211 Thanks for your great work! Thanks to you, building meteor apps is much more fun now. All these load order issues and slow reloads with growing code base....

[jedwards1211]

Thank you guys too! Yeah, I had tried unsuccessfully to implement the same approach as Benoit, I'll be curious to see his workarounds for isobuild limitations. Well if people find this useful then I guess I'll continue developing this approach. Are you guys using the new webpack plugin in your own projects, or the current master?

[jthomaschewski]

I'm on current master. I've not yet tried your webpack plugin branch but had a look at the webpack config files. I still think this project is still useful - The approach of Benoit seems to have still some issues:

• packages.json !== package.json => NPM updates have to be done by hand, package by package... Sure there might be a solution in the future, but it's not really nice how it's currently manged
• webpack configuration is limited (I tried to set resolve.root - that's not possible as it's overwritten in webpack:webpack, also some things like devtool for the server can't be set)
• A module directory structure with server and client files per module is not easily possible (Or everything has to be put into client - still trying to figure this out)
• I've had some problems with my code when using Benoit's stack - Looked like some kind of load order issues (Problems when using meteor globals). But it's also likely that they where caused by me somehow.

Sure some of these problems might be sorted out in the future, some might be caused by my misunderstanding of the concept or something. This was just my first test of the stack.

I would still prefer a Webpack/NPM first, Meteor second approach (Like your webpack plugin) - also because I'm quite sure that I will be using webpack/npm in almost all of my new web projects - but might change the backend for some of them and replace Meteor with something else.

[tomitrescak]

@jbbr After several days of trying to make things work with Benoit's webpack I've officially dropped it and got back to this project. Here are my observations (pluses and minuses of Benoit's solution)

Positive

• + Small boilerplate makes project look very clean
• + Simple to use, as it creates illusion of a standard meteor project
• + Straightforward webpack configuration

Negative

• - Very slow loading of the project (initial build on my project takes 90 seconds vs 15-20 with this one)
• - No server source maps makes server debugging almost impossible
• - No possibility to use typescript or coffeescript on server as webpack is not watching these files (full server restart required).
• - Weird issues with core-js and number constructor
• - Using older React for meteor
• - Not flexible (this does not bother me that much ... I need a stable platform - typical apple user :)

[jthomaschewski]

@tomitrescak I'm also not 100% convinced of Benoit's webpack. But I can't confirm some of your observations:

• using older react: It's possible to use react 0.14 and react-router from npm instead of Benoits react router ssr
• core-js: This is a general problem when using core-js with meteor. You can use the core-js you built with the corejs build script of this project or any other corejs build with disabled number constructor - e.g https://github.com/xolvio/meteor-core-js
• ts/coffescript with full reload: Are you sure this is any better with this skeleton? Hot reload like with react is not possible with server side code anyway. I didn't had the feeling that benoits webpack behaves different from this.
• server source maps: Yeah... They used to work in a previous version of his package but somehow they are broken now

All in all I also prefer the approach of jedwards project as I understand how it works and it doesn't include as much "magic" as his. But I really need SSR and multiple bundles with lazy loading now - This should be possible with this skeleton too but I would have to get it working myself.

When I have some time I will try the webpack-meteor plugin branch and try to get SSR working etc.

[tomitrescak]

@jbbr what I meant is that benoits approach is not watching for changes in typescript and coffeescript on server, so when a change is performed, you need to reset the whole server to see the change, or just modify some server js file. Not very straightforward. The hot code reload on client works as expected.

About core-js. Not sure that is the issue, but his package behaves strangely. Try to download his flow router pack, or change a ssr for csr reactrouter and then add { limit: 2 } as an option to any query and it will fail during the check process. I opened an issue for this. No response by now.

Possibly for similar reasons, some of my server methods were failing. Could not understand why, as I could not debug them. They were throwing null reference errors, while in this package all works as expected. Possibly load order issues.

With code splitting ... If you use benoits approach and a react router, I'm pretty sure you will be able to replicate this behaviour. If I'll have time I will confirm that.

All and all, I moved a large project to jedwards solution in 10 days (rewrote blaze to react), I lost 5 days just to make things work in the other. But that's probably because I'm lame;)

[rclai]

Benoit's setup's initial build is slow, but after that, it's faster.