Move Tracker and all other dependencies into separate repositories and NPM

[mitar]

For #11 to really work, we have to move all dependencies into separate repositories/NPM.

[stubailo]

Should we create a Blaze GitHub org for this?

[mitar]

I am proposing to use meteor-package. Then git repository URLs would be pretty: https://github.com/meteor-package/blaze, https://github.com/meteor-package/tracker, https://github.com/meteor-package/dpp and so on. :-)

I would also do #12 if we are going into this direction.

[mitar]

But this ticket was mostly about the fact that we have to move tracker out of the Meteor core repository as well. Not necessary to do repository-per-package as #12 suggests.

So if do not want #12, it could be that we do meteor/blaze and meteor/tracker and this is it. And inside tracker there is more than just Tracker, but also its dependencies.

[martijnwalraven]

I agree that it makes sense to also split off tracker and related packages (reactive-var etc.).

[stubailo]

I don't see the benefit of having a new meteor-package org - then what would be left in meteor? It makes sense to me to have a new organization if it's for something that might become more of a standalone project, like Blaze.

[mitar]

Just because it would be easier to give permissions to the community to create new repositories inside meteor-package over meteor. Or would you be willing to allow community members to create repositories inside meteor?

[stubailo]

Ah, I see - creating new repositories at will would be useful. Something like the reactjs organization, which is for community React projects.

Then maybe meteor-community?

[mitar]

You mean https://github.com/MeteorCommunity ? ;-) If not, then this might be a bit confusing to people.

Maybe we could use some ideas from: https://forums.meteor.com/t/searching-for-a-name-for-a-meteor-fork/17897

https://github.com/asteroid-belt ? ;-)

[stubailo]

A better place to talk than meteor-core

Hmm, sounds like the forums kinda took that over! And I don't want some abstract name like asteroid where you can't even tell what it means.

Also, I just looked at the beginning of the thread - why do we need tracker moved out at this time? Just to publish it on NPM?

Anyway, I'm starting to think the best starting point would be a blaze github org, specifically for Blaze.

[mitar]

Just to publish it on NPM?

Isn't this the main thing we want to achieve? Stand-alone use?

Anyway, I'm starting to think the best starting point would be a blaze github org, specifically for Blaze.

I am not sure. Because there are many other packages which will be pulled out sooner or later, no?

[stubailo]

Yes, but if we really want them to get a life of their own, lumping them into one giant organization called meteor-* doesn't make sense. for example, maybe it should be tracker/tracker. Or meteor-accounts/accounts-*.

[mitar]

reactjs makes sense.

Anyway, I am mostly concerned with maintenance overhead of one organization, and you are proposing to have even more. I think we can start with one organization, no? But we could also start with just things under meteor and simply have an easy way to request a repository. The question is also how many new community projects should be under meteor.

Hm, not that I think, I think we really degressed here. In fact we are not talking here about community packages (for that community can create some organization, or more), but where to put core packages, which are moved out of meteor repository. So maybe meteor is OK for that.

[martijnwalraven]

Yeah, I don't think hosting new community projects under meteor makes sense. The way I see it, the main point of splitting off these core packages is to make them more maintainable.

Over time, we may want these split off projects to evolve to be more independent, but right now there are a lot of interdependencies that would make this hard. Would a blaze organization include tracker? But then what about other packages like minimongo, that also use tracker? So having everything under meteor makes more sense to me, as they are still all part of one family (meteor/blaze, meteor/tracker, meteor/ddp, meteor/minimongo, etc.)

[mitar]

Yes. But the question of this ticket is, is it OK to create 20+ repositories inside meteor for all 20+ NPM packages we will be creating just to port Blaze to NPM?

[martijnwalraven]

My preference would be to group packages into coherent repositories, as we've done with meteor/blaze. The only downside is that NPM does not allow you to install from Git when you put more than one package into a repository. But that might not be a problem as long as you have a good development workflow. This is the approach projects like React and Babel also take.

An interesting article comparing different approaches: https://plot.ly/javascript/modularizing-monolithic-javascript-projects/

[mitar]

The only downside is that NPM does not allow you to install from Git when you put more than one package into a repository.

Which pretty limits how much you can use it for forks. If you cannot do a dependency against a git repository, then you have to publish a package. Pretty hard.

[murillo128]

The list of packages used by blaze is the one below:

• Core
• Base64
• EJSON
• Random
• MongoID
• IdMap
• OrderedDict
• Tracker
• ObserveSequence
• HTMLJS
• HMLTools
• ReactiveVar
• Blaze
• BlazeTools
• Templating
• Spacebars
• SpacebarsCompiler

Also, in ObserveSequence i had to add two files from minimongo:

• meteor/packages/minimongo/diff.js
• meteor/packages/minimongo/objectid.js

And tinytest for testing all the above

[murillo128]

I think the best solution for packaging is having 1 meteor package == 1 git repo == 1 npm package. This has the (big) downside that it is really difficult to work with branches/git repos directly.

I overcome it by not requiring each specific package, but exporting a function that expect a Meteor object in which I add the package into.

So instead of doing

var Tracker = require('meteor-tracker');
var Blaze = {};
/*......*/
module.export = Blaze;

I do the following:

module.export = function(Meteor)
{
var Tracker = Meteor.Tracker;
var Blaze = {};
/*......*/
Meteor.Blaze = Blaze;
return Blaze;
}

Then you need a master project my meteor-client [https://github.com/eface2face/meteor-client] that acts as entry point, instantiating all the references:

  var jQuery= require('jquery');
  var lodash = require('lodash');
  var Meteor = require("meteor-core")(lodash);
  require("meteor-base64")(Meteor);
  require("meteor-ejson-safe")(Meteor);
  require("meteor-random-window-crypto")(Meteor);
  require("meteor-id-map")(Meteor);
  require("meteor-ordered-dict")(Meteor);
  require("meteor-tracker")(Meteor);
  require("meteor-observe-sequence")(Meteor);
  require("meteor-htmljs")(Meteor);
  require("meteor-html-tools")(Meteor);
  require("meteor-reactive-var")(Meteor);
  require("meteor-reactive-object-map")(Meteor);
  require("meteor-blaze")(Meteor,jQuery);
  require("meteor-blaze-tools")(Meteor);
  require("meteor-templating")(Meteor);
  require("meteor-spacebars")(Meteor);
  require("meteor-spacebars-compiler")(Meteor);
module.export= Meteor;

It is kind of an ugly trick, and don't like it much, but it allows me to override the phisical location of each of the module, and work with branches or local copies. For example in the meteor-module-generator project [https://github.com/eface2face/meteor-module-generator/blob/meteor-update] I have each meteor package as a git submodule and them load each of the packages by requiring the filename, not the package reference:

var jquery = require("jquery");
var underscore = require("lodash");
var Meteor = require("./modules/meteor-core/dist/meteor.js")(underscore);
require("./modules/meteor-base64/dist/base64.js")(Meteor);
require("./modules/meteor-ejson/dist/ejson.js")(Meteor);
require("./modules/meteor-random/dist/random.js")(Meteor);
require("./modules/meteor-mongo-id/dist/mongo-id.js")(Meteor);
require("./modules/meteor-id-map/dist/id-map.js")(Meteor);
require("./modules/meteor-ordered-dict/dist/ordered_dict.js")(Meteor);
require("./modules/meteor-tracker/dist/tracker.js")(Meteor);
require("./modules/meteor-observe-sequence/dist/observe_sequence.js")(Meteor);
require("./modules/meteor-htmljs/dist/html.js")(Meteor);
require("./modules/meteor-html-tools/dist/html-tools.js")(Meteor);
require("./modules/meteor-reactive-var/dist/reactive-var.js")(Meteor);
require("./modules/meteor-blaze/dist/blaze.js")(Meteor,jquery);
require("./modules/meteor-blaze-tools/dist/blaze-tools.js")(Meteor);
require("./modules/meteor-templating/dist/templating.js")(Meteor);
require("./modules/meteor-spacebars/dist/spacebars.js")(Meteor);
require("./modules/meteor-spacebars-compiler/dist/spacebars-compiler.js")(Meteor);
require("./modules/meteor-tinytest/dist/tinytest.js")(Meteor);
require("./ext/meteor-reactive-object-map/reactive-object-map.js")(Meteor);

//Export to global
global._ = underscore;
global.jquery = jquery;
global.$ = jquery;
global.Meteor = Meteor;

As I said, it is a quite awful trick and but it does the job. If we could rewrite/override the location of a package at npm, then life would be much easier, but I don't know of other way of doing so.

By the way, any estimation about when 1.3 will be released?

[murillo128]

Well, after investigating further, this could help:

https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/

[martijnwalraven]

@murillo128: Can you explain why you think shrinkwrapping would help here?

[murillo128]

still don't know.. :)

...but i will investigate, I don't like to export the function and then pass the variable of Meteor, but somehow we need to replace the dependencies on the subprojects somehow, which seems that is what shinkwrap is doing (although just changing the version numbers not the location). Anyway, just a bit of brainstorming

[murillo128]

BTW, i had to add a couple more packages in order to make Blaze work and pass all the tests from the required packages:

• DiffSequence
• Minimongo (This could be removed if ObserveSequence didn't use some utility functions from LocalCollection or they are moved to an external package)

[mitar]

@stubailo, @zol, now that Meteor 1.3 is released, I think we should really focus on the effort of removing Blaze from Meteor repository and make packages stand-alone. Otherwise I think sooner or later it will fork.

[murillo128]

BTW, even if it is a nightmare to develop at the beginning, I think that what makes more sense is a 1 meteor package = 1 git repo = 1 npm package

[zol]

@mitar totally, that's what we'll be focusing on next.

[mitar]

My wish list:

• easy to run Meteor with your own fork of a core package (or in general any package, probably core packages should not be anything special anymore, just a default list of packages available)
• core packages being such forks
• each package should be in its own repository and its own NPM package, or find a way for NPM packages to work with subdirectories (which NPM does not yet support)

[murillo128]

I have been making some tests with npm packages and git branches, and it seems it would be easier than I though.

I have created three dummy projects a,b and c (https://github.com/murillo128/dummy-a https://github.com/murillo128/dummy-b https://github.com/murillo128/dummy-c).

c is just exporting the name of the github branch:

module.exports="master";

I have then published it to npm (https://www.npmjs.com/package/dummy-c) with v1.0.0

b requires the npm package for c and re-exports the the branch name

module.exports=require('dummy-c');

I have then published it to npm (https://www.npmjs.com/package/dummy-b) with v1.0.0

a requires the npm package b and prints branch name

console.log('Using : ' + require('dummy-b'));

Doing an npm ion a rpovides the expected output

> npm i
dummy-b@1.0.0 node_modules/dummy-b
└── dummy-c@1.0.0

> node index.js
Using : master

Then I created a branch on c changing the label of the branch (https://github.com/murillo128/dummy-c/tree/branch)

module.exports="master";

Then I used it c directly on a:

console.log('Using : ' + require('dummy-b'));
console.log('Using : ' + require('dummy-c'));

But installing the branch instead of the package:

> npm i --save murillo128/dummy-c#branch
dummy-c@1.0.0 node_modules/dummy-c

> node index.js
Using : master
Using : branch

As the version in the package have not changed, removing the node modules and installing them again, makes npm to use c from dependency in a (the branch one) and not the one from b (the npm package one):

> rm node_modules/ -rf && npm i
http GET https://registry.npmjs.org/dummy-b
http 304 https://registry.npmjs.org/dummy-b
dummy-b@1.0.0 node_modules/dummy-b

dummy-c@1.0.0 node_modules/dummy-c
> node index.js
Using : branch
Using : branch

Also, if version differs or it is possible to use npm shrinkwrap to get the tree of references to the packages and override both the version and the source:

{
  "name": "dummy-a",
  "version": "1.0.0",
  "dependencies": {
    "dummy-b": {
      "version": "1.0.0",
      "from": "dummy-b@^1.0.0",
      "resolved": "https://registry.npmjs.org/dummy-b/-/dummy-b-1.0.0.tgz"
    },
    "dummy-c": {
      "version": "1.0.0",
      "from": "dummy-c@git://github.com/murillo128/dummy-c#branch",
      "resolved": "git://github.com/murillo128/dummy-c#d07589983747f3492bf042f4eb0fe68039e340cf"
    }
  }
}

[murillo128]

So IMHO I think the best would be to have one master repo/package blazejs and then one meteor-xxxx repo/pacakge per current meteor package.

blazejs
└── meteor-blaze
    └── meteor-core
    └── meteor-htmljs
         └── meteor-core
         └── meteor-xxxxx
         └── meteor-tracker
              └── meteor-core
               .......

So if you want to work with a branch of one of the "core" packages, you just need to install the dependency on the master package and override with shrinkwrap if necessary.

[murillo128]

@stubailo, @zol, @mitar any feedback? What are your plans on how to proceed?

[stubailo]

@zol have we thought at all about naming of our packages on NPM? Does it make sense to use some kind of namespace like @meteor/blaze to avoid the inevitable collisions? I wonder if all meteor-* packages we want are actually available.

[murillo128]

I have already registered quite a lot of the meteor-xxxx ones :)

https://www.npmjs.com/~sgarcia El 31/3/2016 21:17, "Sashko Stubailo" notifications@github.com escribió:

@zol https://github.com/zol have we thought at all about naming of our packages on NPM? Does it make sense to use some kind of namespace like @meteor/blaze to avoid the inevitable collisions? I wonder if all meteor-* packages we want are actually available.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/meteor/blaze/issues/13#issuecomment-204086126

[laosb]

IMO, Organization meteor should be used by both MDG and community members, and MDG should have a separated organization.

[stubailo]

@laosb you're referring to an NPM organization? What's the benefit of sharing one?

[murillo128]

Hi guys, any update on this?

[zol]

@stubailo I'm not too familiar with Npm scopes but my first thoughts are that using them over meteor- would be preferable and no we haven't thought about naming yet.

@murillo128 Given you've been in the trenches with this you probably know more about it than me. I don't quite understand the need for the master package that sits at the top of the meteor- package dependency tree. Is there a reason this is required?

[martijnwalraven]

We probably need some input from @benjamn on this, but I would really prefer it if we didn't go with an ad hoc solution that takes the result of the existing build tool (concatenated files) and publishes these to npm. Ideally, I think we should find a solution to publish Meteor packages on npm with both CommonJS code and ES2015 module-based code (with jsnext:main).

On the subject of multiple packages per repository, I think there are huge organizational benefits to keeping related packages together (as we currently do with meteor/blaze) because otherwise pull requests and issues would be spread out over multiple small repositories (which is especially problematic if they apply to multiple Meteor packages, because you'd lose context). We will need a solution for development, because npm doesn't allow you to install packages from a subdirectory of a repository. But maybe we could use npm link during development?

[mitar]

We probably need some input from @benjamn on this, but I would really prefer it if we didn't go with an ad hoc solution that takes the result of the existing build tool (concatenated files) and publishes these to npm.

+1

because otherwise pull requests and issues would be spread out over multiple small repositories (which is especially problematic if they apply to multiple Meteor packages, because you'd lose context).

Maybe then you just need a way for Meteor to support overriding packages.

[martijnwalraven]

Maybe then you just need a way for Meteor to support overriding packages.

Could you elaborate on this? What do you think would be needed, and what would it help solve?

[murillo128]

@martijnwalraven

, but I would really prefer it if we didn't go with an ad hoc solution that takes the result of the existing build tool (concatenated files) and publishes these to npm.

Me too. There is no reason for doing that if all the pacakges are available as npm modules.

@martijnwalraven & @zol

We will need a solution for development, because npm doesn't allow you to install packages from a subdirectory of a repository. But maybe we could use npm link during development?

Check my post above, it is perfectly possible to override npm dependencies of the submodules on a npm project.

So if we have each meteor package as a npm module/git repo and one umbrella package for branding/issue tracking/docs/tests as

blazejs
└── meteor-blaze
    └── meteor-core
    └── meteor-htmljs
         └── meteor-core
         └── meteor-xxxxx
         └── meteor-tracker
              └── meteor-core
               .......

You can develop on the blazejs (or any other test project) and override the location of each of the meteor-xxx packages to a local directory or a git branch

[martijnwalraven]

So if we have each meteor package as a npm module/git repo and one umbrella package for branding/issue tracking/docs/tests

But that wouldn't really help you with keeping track of pull requests that apply to multiple related packages, as seems likely with the various templating- packages for instance.

Maybe I'm making this too difficult, but I'm afraid splitting meteor/meteor up into dozens of separate repositories would make it really hard to get a coherent picture of the project, and to keep track of what is going on and what needs to be done.

[murillo128]

Well, AFAIK nothing prevents technically to have a single git repo with subdirectories for each package and publish just a single module to npm. In fact that's jQuery does it IIRC.

For example, if we create the blazejs npm package, you could later import either the whole module or each package by subdirectory. (Talking from memory would need to check and double check):

npm i blazejs --save

var blaze = require('blazejs');

var spacebars = require('blazejs/packages/spacebars');

I think the main problem is how to handle shared modules between blaze and other meteor projects

[mitar]

Maybe I'm making this too difficult, but I'm afraid splitting meteor/meteor up into dozens of separate repositories would make it really hard to get a coherent picture of the project, and to keep track of what is going on and what needs to be done.

So why are we even moving Blaze out of meteor/meteor? Is it just because of permissions for GitHub? Couldn't that be solved than on some other level? For example, by giving commit rights to an user, but with clear understanding that that person should make commits only to a particular set of subdirectories? You could even maintain a simple file in the root of meteor/meteor.

But the reason why we might want to split things into separate repositories is to that they can be used outside of Meteor. And then yes, we could have one repository with multiple NPM packages, but it is really problematic because NPM does not allow installing a package from the git URL, so forking is then hard.

What about having each package in its own branch? NPM can work with that. Then master branch would only have a README explaining repository structure and this is it?

Could you elaborate on this? What do you think would be needed, and what would it help solve?

So the only reason why I think having subdirectories for each package is problematic is because NPM cannot reference that in a fork directly through git. (Branches might solve this problem.) But if we would care only about Meteor, then we could have some other way to get a fork of package to be used in Meteor app. Because for example, putting repository as git submodule inside packages inside the app would not work, if packages are in a subdirectory inside that repository.

But if Meteor would have a way to say for a Meteor app to use mitar:blaze instead of blaze in my whole app, I could use a fork. And then if I would have a way to say that mitar:blaze should come from a subdirectory of a git repository it would be perfect. And this would be useful in general, not just for packages here. This is what I was suggesting.

But as far as I know you want to move away from custom packaging system. So I would suggest that we find a way to have NPM work with forks. So maybe we use branches?

[murillo128]

But the reason why we might want to split things into separate repositories is to that they can be used outside of Meteor. And then yes, we could have one repository with multiple NPM packages, but it is really problematic because NPM does not allow installing a package from the git URL, so forking is then hard.

I don't think that is true (some please correct me if I am wrong). NPM only requires a package.json in order to publish a directory/tarball to the npm repo. So it is perfectly possible to publish a subdirectory from a git repo to to npm.

https://docs.npmjs.com/cli/publish

That would only require that each package subdirectory has their own package.json. Note that the repository tag in package.json is only informative:

https://docs.npmjs.com/files/package.json#repository

repository

Specify the place where your code lives. This is helpful for people who want to contribute. 

[mitar]

I don't think that is true (some please correct me if I am wrong). NPM only requires a package.json in order to publish a directory/tarball to the npm repo.

Yes, you can publish, but you cannot use a git fork without publishing your forked package as well. So developing is much harder because you cannot locally use it as a dependency in your app, to then decide to make a pull request or publish that packages on NPM. (I myself very rarely publish a forked package, but I do use forked git repositories a lot.)

See: https://github.com/npm/npm/issues/2974

[martijnwalraven]

Wat about using npm link for local development?

[mitar]

How can I use npm link so that I can do git clone myMeteorAppWithForkedBlaze and then meteor run without running any extra commands?

[mitar]

So one thing of using forks is not just that you use it for development, but that you can also commit dependency on a forked repository into your repository and it will work. This is why NPM dependencies on git repositories are so powerful. But they work only with monorepositories.

[murillo128]

@mitar I get your point now. You mean that you cannot do an npm install of a git subdirectory, which is true.

Note that you can always checkout the git branch to a local directory, and then an npm install of the subirectory. Which in fact would be easier for developer as you would not neet to commit and push before using the changes

[murillo128]

Also note that I am in favor of 1 package = 1 git repo = 1 npm module, just trying to put all the alternatives on the table.. ;)

[laosb]

Whatever, having totally different things in different branch is not a good practice, and may cause some problems in managing.