Meteor and external libraries

[raix]

We should avoid writing pure wrapper packages.

Why:

1. It requires human ressources - making it unreliable and laggy
2. We risk duplicated code eg. if we have multiple versions of the same wrapped library
3. It opens Meteor for alot of external packages

Meteor wrapper:

1. Allow packages and applications to add external dependencies
2. Clean up the library code removing the amd/umd/globals/export code
3. Have the brilliant package system version resolver work on this

Examples as language syntax:

  // From github
  import "github:Famous:famous/src@0.3.1"
  import "https://github.com/Famous/polyfills.git@0.3.1" as famous.polyfills
  // bower - browserify/npm could be added too
  import "bower:underscore" as _
  // Local lib, use fs watch to reload on lib changes
  import "/Users/raix/locallib/" as localLib
  // Add from some cdn
  import "http://some.cdn.com/cdnlib.js" as cdnlib
  // Add meteor package
  import "raix:library@1.0.0" // Should it be possible to mount exports on a scope?

Examples as current CLI and package.js api's:

  // An example of how to mount a library on a symbol
  api.use('https://github.com/Famous/famous.git/src@0.3.1').as('famous');
  // An example of how to extend a symbol
  api.use('https://github.com/Famous/polyfills.git@0.3.1').as('famous.polyfills');
  // Add underscore from bower and mount it on the _ symbol
  api.use('bower:underscore@1.0.0').as('_');

Examples of CLI:

# moment from bower where its mounted on moment symbol
$ meteor add bower:moment@1.0.0 as moment
# moment as global import?
$ meteor add bower:moment@1.0.0
# cordova plugins
$ meteor add cordova:org.apache.cordova.camera@3.0.0
# npm packages
$ meteor add npm:chokidar@1.0.0
# mount a meteor package exports on a symbol
$ meteor add ground:db@1.0.0 as gdb
# from github
$ meteor add 'https://github.com/Famous/famous.git/src@0.3.1' as famous

We could have a build farm that scans npm/bower/components/browserify and builds wrapped versions... But the folks holding usernames/org names as their meteor login would have to donate/release it.

We could create a meteor release where bower:/npm: etc. were handled specially by meteor add or figure out who reserved the npm: and bower: orgs on meteor and create a wrapper farm.

References: https://github.com/meteor/meteor/issues/2798#issue-45543348 https://github.com/gadicc/meteor-famous-views/issues/136 https://groups.google.com/forum/#!topic/meteor-core/DPd6XK4Ow-w https://github.com/mquandalle/meteor-bower/issues/38 https://github.com/meteor/meteor/issues/2919

Todo: (having a proof of concept)

• [x] Source fetcher for downloading the source from github, bower
• [ ] Source fetcher for downloading the source from npm, http, https, path
• [x] Code translator from requirejs/commonjs/amd/umd/globals to Meteor Wrapper packages
• [x] Modify the meteor constraint solver to accept wrapper packages
• [x] Modify the meteor package api allowing wrapper packages and setting scope
• [x] Modify the meteor CLI tool to allow wrapper packages

Ref to the proof of concept code:

• https://github.com/raix/Meteor-include
• https://github.com/raix/meteor/tree/raix-isohouse

$ meteor --release iso:METEOR@1.0-bower.7 allows:

• $ meteor add bower:underscore@1.7.0 or $ meteor add github:jashkenas:underscore@1.7.0
• Grab scope in code: var myBowerUnderscore = Iso.require('bower:underscore/underscore');

[dandv]

I was working with an original library author to accept a PR for Meteor. Would that be an even better solution than wrapping through Famono?

[raix]

@dandv depends on the library - famous is nicely split up into smaller files, if they added a package.js all of the code would be added - so I'm not a fan - I think we should have better ways for reusing code and only add the code needed.

At the moment MDG isnt concerned about footprint, I guess size doesnt matter and we could have some tool minimizing the code/dead code removal. But as far as I know, none of scoped functions can be matched up / linked. So we still have too much non-sharable code / more code to maintain / more errors etc..

So to me its more of a battle between order and chaos - where imho the chaos could be avoided.

[gadicc]

Oh wow, almost missed this, nice!

1. Yeah, I agree.. finding a way to integrate with meteor build process would be ideal and amazing.
2. If we're removing the amd/umd/export code (nice!) I think we could also try find a pattern to undo common wrapping patterns authors use, maybe. But this is a super optimization which I don't think needs to be a priority.
3. Does server-side really matter? It hurts deploy, but I don't think the extra memory use is such an issue... my issue for me is sending everything to the client. I don't think this can hurt but I think is the least important?

And yes, package dependencies... I thought a lot about it and have a very good idea of how this should work (I could put it down somewhere if you want and see if we see eye-to-eye) but it's a lot of work :)

Versioning remains a big issue that we'll have to figure out. I think despite the initial pain MDG have done a good job here.

[dandv]

No idea... feel free to try, I'm working on getting official package support for Vis's deps first (hammer.js and Moment), so I'll only get to Vis a bit later.

[IstoraMandiri]

Some non-famono-specific discussion: https://groups.google.com/forum/#!topic/meteor-core/DPd6XK4Ow-w

Potential idea could be to integrate 3rd party package managers using the new namespacing:

meteor add npm:async@=x.x.x
meteor add bower:desandro/masonry
meteor add cordova:org.apache.cordova.geolocation

[trusktr]

This would be awesome. I wanted to do this, but if you guys do it, then yeah. :D

jspm makes AMD/UMD/CJS/ES6 work interchangeably with each other. e.g.

import AMDModule from "path/to/AMDModule"

or

var ES6Module = require("path/to/ES6Module")

[raix]

jspm pattern looks very similar to the meteor package naming. We could rig a wrapper farm enabling npm and bower - it wouldnt be too difficult - but who reserved npm: and bower:?

[gadicc]

bower reserved by @dandv for this purpose :) npm doesn't seem reserved on atmosphere at least?

but is a wrapper farm the right approach? it's definitely the fastest, to work through the existing meteor package system. but i'm not sure i'd like to see wrapped packages showing up in atmosphere, nor the extra load of serving them, if it's something we can have the meteor tool do itself straight from npmjs / github.

[trusktr]

@gadicc @raix A wrapper farm just means you're lucky enough to own the username npm or bower on Atmosphere. xD It wouldn't be a legitimate solution IMHO. I don't think meteor-agnostic packages should even exist on Atmosphere since.

Potential idea could be to integrate 3rd party package managers using the new namespacing:

meteor add npm:async@=x.x.x
meteor add bower:desandro/masonry
meteor add cordova:org.apache.cordova.geolocation

This is exactly what jspm does. :D

jspm install npm:async@=x.x.x

jspm has a registry system. It's easy to create new registries. A registry could be used to store build info for each library to make each lib Meteor compatible. But that would also mean that someone needs to create a registry entry for each package if needed, still using man power (it's inevitable, somewhere in the production line it's needed since Metoer's packaging system is it's own system).

IMHO, using jspm would be sweet to use for this. It's a single package manager for all module types. @guybedford has done amazing work with jspm, es6-module-loader, and systemjs and the projects are making super progress.

Plus, when native ES6 Module systems finally land in browsers, jspm will be there to handle all the legacy AMD/CJS/UMD modules in harmony with ES6. It'd be pretty amazing to support/use jspm.

meteor add namespace::package-name@version

(with two colons?) for example, to install meteor-agnostic packages from places besides AtmosphereJS. This could translate to

jspm install namespace:package-name@version

behind the scenes.

[raix]

not sure @gadicc - I have to read up on the build tool etc. but at the moment @glasser seems to be on a sprint rewriting alot of the compiler/constraint resolver etc. But when it settles we could try to create a local package wrapper as a Meteor release - proof of concept.

The current approach I'm puzzling with is:

1. allow the user to add dependencies on bower:/github for starters
2. have the meteor build resolve what version to use and complain about conflicts etc.
3. have the meteor build tool use a bower/github sourceFetcher if package is named ^bower: instead of the default package downloader
4. The source fetcher should create a wrapped package using some of the tech in famono or jspm

The wrapped package could depend on a bower-require package that exposes the bower package scope via var famous = Bower.require('famous'); or a jspm like System.import('bower:underscore');

It might make sense to actually convert the bower package name into the git hub name (since most/all bower packages are on github?) - the reason for this is that if a package includes famous via the github: pattern it should match the bower: version of the package too in the constraint resolver. The version should match normal tags on github like v1.0.0 or 1.0.0 - there will most likely be situations where this is not the case though - we'll have to solve this.

So it should become possible to do:

  meteor add github:Famous/famous@0.3.1
  # or
  meteor add bower:famous@0.3.1

@dandv I'm glad you got the bower:, maybe we should checkout path, url, github: or git:?

@trusktr It would be nice, but I'm not sure if jspm and the meteor build tool work of the same principles internally - Meteor resolves versions and builds the js code into a bundle - Should be investigated. Btw. The registry seems limited - imho we should be able to use all sources local/remote.

EDIT: @trusktr could be two colons like github::Famous:famous@0.3.1 true, eg. if a new package system comes on later on, could be something like famous would do...

[trusktr]

@raix

Btw. The registry seems limited

What do you find limited about it? @guybedford has been super responsive and might be able to help us out with it.

[raix]

Now I'm thinking out loud... I'm working on some prototypes of package wrappers from bower/github/npm - since these will be useful in local/farm approach.

I'm currently thinking of creating a small node-bower-wrapper that could be deployed on a server/farm and would keep 10 bower/npm packages updated on meteors package server. 10 packages for starters - at some point it could do the whole thing

So the reason why the meteor package server publish would be nice is:

1. We would not have to change the Meteor tool
2. Packages would not depend on bowers uptime
3. Packages would be ready and isopacked

The downside is that

1. It requires a server
2. We would have to scan bower/npm
3. Is the meteor pacakge server actually ready for this?
4. Is it the intended pattern?
5. We risk having a lot of garbage packages

@trusktr the registry in their repo.

[queso]

I would hate to see the atmosphere package list junked up with more stuff for me to sift through. What do we really gain by having this automated approach?

[raix]

@queso Agree about spamming atmosphere - We gain having the version constraint resolver doing some work for us?

Update: I'm in the works of writing/adding a wrapper catalog in the meteor-tool - I hope to make a test release soon for you guys to test out a bit. It will build the wrapped packages on client - and connect to bower/github for more.

[dandv]

@dandv I'm glad you got the bower:, maybe we should checkout path, url, github: or git:?

Reserved path too, but url, github and git were already taken.

[raix]

@dandv its super - I'm deep in the meteor-tool, it overrules what ever, but if a problem we could use git::org:repo since users are not allowed to publish :: Status: I can do meteor add bower:underscore (or meteor add github:jashkenas:underscore) and it runs though the version resolver + bower and github, fetches the code and passes it on to a isopackWrapper (that I'm working on atm.)

[mitar]

One reason mentioned multiple times is that for the server side you would not want just to provide a package, but also provide a blocking (fiber enabled) interface to otherwise callback-style API usually available in node.js packages. I do think that this is a valid argument.

On the other hand, I do agree that there it should be easier to bring files into Meteor package, especially when they already have some packaging logic around, which combines files and so on, and then add on top of that blocking interface.

[trusktr]

I have a theory: a package (let's call it "meteor-webpack" for reference) being a dependency of a Meteor application could Npm.require('webpack'); Npm.require('npm'), look in the application's top level (this is possible right?) for a package.json to see npm dependencies, use npm programmatically to install dependencies and build a bundle during Meteor build, and put that file in a place that Meteor automatically sources during runtime.

Since Meteor's API is stored in globals, any module in the bundle can access Meteor API. I think that an entry point (stored somewhere that meteor doesn't source) could require() things, and refer to Meteor APIs, then when it gets compiled into the bundle, it'll work with Meteor references.

This theory seems like it'd work for an application, but I'm not sure how it'd work for packages of an application. For example, suppose we want to make a package that depends on npm modules for the client side and we want to use that package in our app. Maybe meteor-webpack would load itself first, detect dependencies for all packages (???)

@gadicc I was thinking about this because of https://github.com/gadicc/meteor-famous-views/issues/166#issuecomment-87969185. I think that pre-including libraries in a package wouldn't be ideal (suppose your app depends on multiple packages that do this, and they all include different version of the same dependency).

[trusktr]

But, let me look at the meteor-include and isohouse prototype first!

[raix]

@trusktr would be nice to be able to use bower/npm with the version constraint solver but theres a lot of ways to Rome, the Meteor-include is just a package for the METEOR.bower release of Meteor (linked from the include repo) but it basically uses bower and the meteor version solver.

[trusktr]

Using webpack for client code seems to be working nicely. The source code lives outside of the meteor folder, then webpack builds the bundle into the meteor folder somewhere so that Meteor automatically picks it up. To paint a brief picture:

# in the project root
webpack # builds entry point from outside of meteor folder into meteor folder
(cd meteor && meteor)

webpack.config.js might look something like this if it's living next to the meteor folder:

var webpack = require('webpack')
module.exports = {
    entry: "./src/app.js",
    output: {
        path: './meteor/public',
        filename: "app.js"
    }
}

[trusktr]

I started experimenting with meteor+webpack here: https://github.com/trusktr/meteor-with-webpack