Official Meteor integration directly from 3rd party libraries

[dandv]

If you're looking for a popular 3rd party library on Atmosphere, chances are you'll find several wrappers, in various states of maintenance. To avoid this problem, several Meteor community members have started to provide authoritative, high-quality, Meteor integrations of 3rd party libraries, complete with tests and grunt scripts. These are forks of the library rather than Git submodules, because forks allow modifying the source to support Meteor (if necessary) and one can still pull upstream changes without having to re-apply the integration changes.

These forks live in the MeteorPackaging GitHub organization. This org was created so that multiple developers can collaborate on maintaining an integration in a centralized location. Now that Meteor's Package API is stable, we can submit PRs to popular JS library maintainers to add official Meteor integration. Some maintainers will accept the PR - great, we have integration straight from the source, with same-day updates on Atmosphere. Some won't accept the PR, in which case we continue with the MeteorPackaging fork.

I started this effort in November 2014, spoke about it at DevShop SF, and here's where we are now.

Going forward

As of September 2015, see http://autopublish.meteor.com. It's a system authored by @splendido that automatically publishes new versions of a wrapper package automatically on Atmosphere (once you've registered your repo with it), as soon as the upstream package has a new release. See the How it works page. For an example repo, see https://github.com/MeteorPackaging/numeraljs-core-wrapper.

The instructions from here on may be somewhat outdated.

Contribute (minimal effort)

1. If you're working on integrating a 3rd party library with Meteor, please post here. I have also reserved Meteor Developer (Atmosphere) organization namespaces for many libraries; ask if you need one.

2. If you have created an MD organization for a library, please let us know. Namespace is precious. Currently we're looking for:

   • angular
   • bootstrap
   • moment
   • famous
   • jade
   • foundation
   • highcharts
   • stripe
   • bitcoin
   • crypto
   • soundcloud
   • twilio (squatted and never used)
   • echo (squatted and never used)
   • win (for win.js)
   • app (for app.js)

   Who has these?

   Popular libraries with official Meteor support

   • :white_check_mark: d3 - [make, export file]
   • :white_check_mark: Bootstrap Material Design by @FezVrasta. Use the PR as a template.
   • :white_check_mark: Sortable [first merged!]
   • :white_check_mark: Keypress [merged 2014-11-28]
   • :white_check_mark: KeyboardJS [export file]
   • :white_check_mark: summernote - WYSIWYG editor; [2014-11-28]
   • :watch: Semantic UI [discussion in progress]

     PR acceptance pending

   • :clock1: Mousetrap - keyboard events library [pending] [grunt, export file]
   • :clock10: Font Awesome - [acknowledged]. Great example of Meteor resolving CSS URL paths correctly.
   • :ballot_box_with_check: Moment [merged in develop, published]
   • :clock1: Isotope [pending] [grunt, export file]
   • :clock930: jsPlumb [pending, maintainer was in a pissed off mood today]
   • :clock8: Hammer - multi-touch library, ~10k GitHub stars [pending, author has changed jobs] [grunt, export file]. Use it as a template.

     Refused to accept the PR

   • :no_entry_sign: keymaster
   • :no_entry_sign: Bootstrap - with much love and offering a Tweet

     How to do this

     Overview

3. Post here asking @dandv and @splendido to add you to the MeteorPackaging organization and mention the library/libraries you want to package.
4. We'll fork the library's repo into MeteorPackacing and grant you access
5. Add the Meteor integration files,
   • package.js
   • tests
   • gruntfile (or makefile or equivalent) entries to run meteor publish
6. publish the package to Atmosphere yourself (this is very important)
7. submit a PR with the changes to the original library author (optional), or just notify them to get some exposure (they may offer to tweet about the integration or otherwise announce it)

Below are the detailed steps.

Get the integration working before even contacting the maintainer

1. Fork
2. git clone <your_fork_url>
3. cd <repo_directory>
4. git remote add upstream <repo_url_of_the_library>
5. git fetch upstream
6. Read through CONTRIBUTING.md or README.md and figure out which branch the original author wants pull requests against. Some can be very finicky.
7. git checkout -b meteor-integration upstream/that_branch
8. Create a meteor directory in the root of the project and copy there the Hammer.js Meteor package files, or the Font Awesome Meteor package files, depending on whether you integrate a library or CSS/image/font resources.

9. Go through every file and make changes to reflect the actual library you're working on.

   • in particular, give a GOOD summary in package.js (for SEO on Atmosphere), instead of the library's tagline. Example:

     Mousetrap is a simple library for handling keyboard shortcuts in Javascript.

     For Atmosphere, a repository of JavaScript libraries, this is suboptimal: it repeats the library name, and it says "library" and "JavaScript" (well duh). You only have 100 characters. Don't waste them. Better:

     "Mousetrap (official): bind, trigger and handle keyboard events on keys, combinations and sequences"

10. Export the library global(s)

    1. If the package is a jQuery package and adds itself to $, there's nothing to export.

    2. Otherwise, add a meteor/export.js file like this:

       /*global Hammer:true*/  // Meteor creates a file-scope global for exporting. This comment prevents a potential JSHint warning.
       Hammer = window.Hammer;
       delete window.Hammer;

11. Add a package test file, at least checking if the library instantiates correctly.
12. You want to make it as painless as possible for the maintainer to accept the PR. If there's a build process they use to publish, it would be great if you could add to it. Here's a gruntfile patch for Meteor and a Makefile patch (used for d3). If their README mentions Component/Bower/Npm, add a section mentioning Meteor (example).

13. Run the tests using the build process, e.g. grunt meteor-test. If you want to run tests manually, execute this:

    cp meteor/package.js .
    spacejam --mongo-url mongodb:// test-packages ./

    Pick a namespace and publish the package under it

Please be very careful with this, as renaming packages is impossible.

Log into your Meteor Developer account and create an organization that represents the package. Most often it will be the name of library, without any special characters (no -, .). No need to include the JS part, since we're always dealing with JavaScript libraries.

If the organization name is not available, contact me, @dandv. I've registered a bunch of them, so potentially evil individuals don't :)

1. meteor publish the package to Atmosphere for the first time. This prevents the maintainer from running into an error that the package doesn't exist when they publish (meteor would complain about a --create flag).

2. After you've ensured publishing works (namespace is available, the name in package.js is valid etc.), push to your git fork:

   git commit
   git push origin meteor-integration

3. Congratulations! Reply to this issue and add the link to the package's Atmosphere page.

Now you can go to the GitHub page of the library and issue a pull request!

Contact the library maintainers

Here is the recommended PR message.

See d3 and Font Awesome above for addressing objections.

Contact the maintainers of other Meteor wrapper libraries

1. Ask them to deprecate the library by creating a PR overwriting the README.md of their package with the text

   This package is now deprecated. Please use the official Meteor integration package:
   
   http://atmospherejs.com/<org>/<package>

   The commit message should be "Deprecating package in favor of official integration".

   The name of the PR issue (after you click "Create pull request) should be "Kind request to deprecate package in favor of official integration".

   The text of the PR should be similar to this:

   Hi X,
   
   Thanks for publishing this package. In the meantime an
   [official Meteor integration](https://atmospherejs.com/<namespace>/<package>)
   has become available.
   
   Would you like to support the official package?
   
   If so, you can hide [your package](<atmosphere_link_to_their_package>)
   from Atmosphere searches, while still keeping it installable by apps and dependent
   packages, by running the following command:
   
      meteor admin set-unmigrated <their_full:package_name>
   
   Thank you,
   The [Meteor official integations team](https://github.com/raix/Meteor-community-discussions/issues/14)

2. Go to the Atmosphere page of their package and flag it anyway. Ideally, get others to flag the wrapper package. This is important because meteor doesn't warn the user when running meteor add on a deprecated package, so we hope users will check on Atmosphere. However, each package needs several flags from different users, in order to show as flagged globally.

   TODO

   • figure out namespacing exactly. Ideally we'd have an MD organization with the library name, and ask the library maintainers to create MD accounts reflecting their GitHub usernames. Problem:
   • sometimes the names are taken, e.g. moment (pinging @estark37)
   • sometimes the maintainer uses their name as the organization, and you can't have an org and individual with the same name on MD - e.g. RubaXa/Sortable
   • rarely, several libraries have the same name, e.g. RubaXa/Sortable and voidberg/html5sortable (contrived example, need a better one)

[gadicc]

@dandv, re 2, yeah, if there's no module framework found it exports to window. Which is bad for a Meteor package. It's easy to get around, but unfortunately requires extra files. e.g. https://github.com/chuangbo/meteor-marked/commit/e55d9c25ec1b7cce2b29ef7d8e4539821f415350. Maybe part of this issue should be to either support that type of wrap policy natively or allow a { wrap: 'exports' } or something in addFiles.

[dandv]

@gadicc: how about just checking if Package is defined, and assigning to a global var?

[gadicc]

@dandv oh yeah sure, if you have access to the main package itself... forgot about that :> i mostly wrap git submodules, so don't like to touch the actual code. what you describe would work perfectly of course. have a vague memory though that Package name was used by something else too, not sure... it would solve our problems though :>

[PEM--]

:clap:

[dandv]

Today I'm happy to present the official d3 integration.

Please log into Atmosphere with your MD account and,

• star https://atmospherejs.com/d3js/d3
• flag https://atmospherejs.com/meteor/d3

Thanks!

[raix]

@dandv cool - is it the d3js or meteor that maintains the package?

[dandv]

@raix: the best permissions combo I've found so far was to create an organization for the library, add myself to it so I can publish the package (helps to remove barriers for the maintainer and make it as easy as possible for them), then ask them to sign up for an MD at meteor.com and tell me their username. Really a 30-second process. Then I'll add their username to the org so they can meteor publish as well.

[raix]

@dandv that makes sense - it would be nice if possible to add the package.js in their repo (the underscore already got 3 package configs... would be nice with just one...)

[dandv]

Do you mean https://github.com/jashkenas/underscore already has bower.json, component.json and package.json ? It would indeed be nice to have just one, but I'm not sure how that would work with Bower and Component.

A Meteor integration for component would look very much like the d3 integration. I wanted to keep all meteor-related files in the same directory, and package.js is not in the root directory of the project because it sometimes conflicts with Dojo's package.js.

[gadicc]

@dandv just wanted to commend you on your hard work here, and particularly, your vision... to be honest the thought of getting 3rd party libraries to directly publish Meteor had never occurred to me before. you're doing a stellar job, and I can see how it's been infectious.

this is good for so many reasons... i was always against wrapped packages being in atmosphere when they offered nothing new on their own (you and I have discussed this before). but this approach is a real pleasure. latest official release guaranteed, and it means that even if I do an integration package, that say, offers templates etc to use the lib, I don't need to release for every new version and an app developer can still use my package and the latest version of the lib. it's awesome :)

also i feel you've done a great job of instilling meteor's place in the JS community... I'm starting to feel more like a 1st class citizen now :>

[raix]

@dandv should we hit famous? EDIT: it would be nice if possible to have famous split up in alot of small packages - eg. have a famous:core that included famous:core.Surface package etc. Getting a more finegrained approach.

[splendido]

@gadicc: are you chasing me?! ;-)

Since I was already in contact with Semantic-UI maintainers I decided to get their opinion before starting chasing meteor wrapper maintainers asking to deprecate packages. I felt the request might be even more robust and acceptable by MD having the official maintainers behind it...

In the discussion @dandv already linked we're also trying to figure out which is the more appropriate style language to be used as the main official package: CSS, LESS, SASS, etc. So its not only the way/script to be used to publish the package (@dandv is doing a great job about this!) but also how to select the proper meteor way. I think it would be also great to have your opinion about this. Do you think we should open another issue on this repo?

Tnx, Luca

[gadicc]

@splendido, i'm following everyone doing good work in the community :> i'll wait and see how the discussion plays out, I'm less knowledgeable about styling formats. but back to the JS discussion, which I guess is related...

@raix, personally i find it annoying to both use and maintain split-up packages like that. also, it's quite likely they might want e.g. famous.views.GridLayout and not everything else in famous.views.*.

It's very clear where this is going :) Packages should include everything, and then you are going to rework famono to work directly on installed packages too :) It's good that you already got familiar with publishing meteor releases :> This is no small feat, and I think also, a revolutionary gamechanger, but your track record is impeccable, so expecting the proof-of-concept tomorrow sometime :>

[raix]

@gadicc Haha, right on

[zimme]

I just wanted to tell you guys that I'm sitting on the "documentcloud" organization for the day underscore/underscore-contrib/backbone wants the namespace.

Working on getting meteor support into the official repos of the wrapper packages I'm maintaining too

[raix]

Thanks @zimme :+1:

[dandv]

@dandv just wanted to commend you on your hard work here, and particularly, your vision... to be honest the thought of getting 3rd party libraries to directly publish Meteor had never occurred to me before

@gadicc: I'm stoked to read those kind words. While I do believe my ideas tend to be on the radical side (e.g English should be the universal language, or we should have the ability to start startups bypassing current bureaucracy), I'm glad this one is also practical :-)

Now if only I could convince @HedCET to deprecate his Font-Awesome package... It is hard to kill your own projects when something clearly superior comes along.

[dandv]

Hey @awwx, should we maybe merge efforts? Your writeup on dealing with exports is very helpful, and now it seems the community has agreed that submitting PRs to library maintainers is the way to go.

[dandv]

All, please flag https://atmospherejs.com/benjaminrh/bootstrap-timepicker - the original library only works with Bootstrap 2 but doesn't specify that, and the Meteor wrapper hasn't been updated for Meteor 0.9+, yet has 15 installs.

[gadicc]

@dandv, heh, we should have a chat sometime. at my current rate i'm expecting some free time in Q3 2015 :)

[awwx]

@dandv I have nothing to add :) Keep up the good work :+1:

[dandv]

We got the first explicit refusal.

[dandv]

Another one. I'm losing hope.

[simison]

See also https://github.com/Nemo64/meteor-bootstrap for bootstrap-less

[raix]

@dandv You are doing a great job, sure some will say no - I kind of follow the argument about this repo not being an "official" Meteor community repo etc. But we could ask Meteor about some way of getting your back clean. cc @n1mmy

I personally think it would be great to have library writers publish packages for Meteor - but I also respect that not all even know about Meteor and doesnt see it as worth the effort (yet). The only thing we can do is stay polite and patience about it, letting them know where to get help if they want to later on.

Maybe give them some tooling like discussed in jsplumb/jsplumb#15

The https://github.com/raix/Meteor-include Meteor release looks and feels good - and would be a nice fallback if libraries aren't supported directly or not published anywhere as packages.

[dandv]

Comment from @madrobby, the maintainer who refused to merge the PR. How feasible is this?

I strongly suggest for Meteor to have centralized repository to store manifests/recipes/formulas (like Homebrew or CocoaPods). That way I wouldn't have to do anything, and someone else could easily keep files up-to-date for packing.

[zimme]

Well, I'm sorry to say that the choice is up the the maintainer and even with this effort from our side to try and get meteor support into the official repos there are gonna be some maintainers who won't go for it. In these cases we're gonna have some wrapper packages until we can convince the them to come over to the dark side... or until they are enlightened.

[zimme]

Maybe some maintainers looks at the PR's and see the meteor folder and script files, thinks it's too much hassle than just a package.json/package.js/bower.json.

I know there are multiple packaging tools that uses package.json

Maybe the same can be done with package.js or maybe convince MDG to add support for meteor.js in the packaging system.

Just a thought

I think it might be a wee bit easier to convince some maintainers if we can get support for grunt/gulp and such ready. Would have started work on this if I've ever used any of them.

[dandv]

In these cases we're gonna have some wrapper packages until we can convince the them to come over to the dark side...

Or we can continue maintaining the fork. In this case, I've updated the README to explain the situation, and pointed the git URL in package.js to the fork. Still a bit easier to git pull and run the publish script than it would be to update a wrapper library.

As for package.json, we now have a Gruntfile patch to publish to Atmosphere, but should meteorpublish be added to the default task?

Gulpfile is TODO.

[gadicc]

Agree with @raix. We can't expect 100% adoption, especially not in the beginning. @dandv, it helps that you've emphasized that Meteor is the most popular full stack framework, maybe we could also mention it's the 7th most starred JS project on GitHub with 20k stars (do we know the total install count?) -- to show it's a sizeable user base. It also helps that we're getting support from some of the other most starred projects, e.g. font-awesome is no 7, and if they come through, bootstrap is no 1 and d3 is no 5.

As for a centralized repo for meta data, sounds like a chore but if that's a common existing approach for a lot of package authors - and a deal breaker for them - then I guess we should do it (or at least investigate, I don't know this approach).

As for fork vs wrapper, I guess it depends how the wrapper is maintained. At the end of the day, we're unfortunately in the same problem, where you'll be responsible for updating all these packages where potentially they could become out of date. So I guess then it just falls on the community maintainer. When I wrap I package, I do it via git submodules... pretty much the same as a fork with no merge conflicts. I'd definitely prefer not to wrap any packages at all though :>

[Program3r]

Actually I didn't delete the repo, and it's not old. I provided the v1 and v2 versions of isotope just in case there were problems. I had an issue once where isotope kept failing to initialize because of it's requirejs integration plus some conflict (can't remember). Will you provide a version with v1? Not sure if you can post past package versions.

-J

--Don't worry I'm willing to remove the package.

On 11/29/2014 12:41 AM, Dan Dascalescu wrote:

Hey @Program3r https://github.com/Program3r, looks like you had published an old Isotope package on Atmosphere https://atmospherejs.com/clode/isotope and then deleted the repo. We have an official one https://atmospherejs.com/isotope/isotope now. Can you please rm yours by running

meteor admin set-unmigrated close:isotope

Thanks!

— Reply to this email directly or view it on GitHub https://github.com/raix/Meteor-community-discussions/issues/14#issuecomment-64943491.

[dandv]

Moved the discussion with @Program3r to a specific thread.

[dandv]

We now have Grunt files for most of the packages in the OP, and I've hooked into Bootstrap's Gruntfile so the maintenance load could be effectively zero.

Published out of the fork at http://atmospherejs.com/twbs/bootstrap

Any feedback/thoughts before submitting the PR? Maybe a different PR message would be helpful, since mine haven't always been successful :)

[splendido]

Might be a silly proposal but you'll say it in case...

Instead of forking official library repos here and there putting them under different users, what do you think about creating a github organization (lets call it meteor-3rd-party-libraries or whatever you like...) to be used specifically for this purpose?

Initial forks could be put there to keep everything under control is only one place and avoiding spreading them even more (e.g. fontawesome published by @dandv under name fortawesome:fontawesome forked under dandv/fontawesome repo and having a readme on atmosphere pointing to the official non-meteor repo) which might a bit frustrating for anyone trying to understand in which state this 'official' porting is...

All forked repo initially put under the same organization page, modified to be rady to publish, PRs made to official repos/maintainer. In a separate empty repository under the same org we could keep an issue for each porting in progress. Mark them appropriately (discussion in progress, awaiting publish, published, PR sent, PR accepted, eventually migrated....)

And also, possibly keep al this discussions together under the same org with this specific purpose.

We might meep list of packages we'd like to be deprecated, log ongoing attemps. Etc.

The long story short: a place known by the most where to go to know about 3rd party library published for Meteor! ...or propose new ones :-)

[splendido]

@dandv: about bootstrap, are you publishing compiled css files? For Semantic it seems we're agreeing on publishing a package with all .less.import source files from which another package (the main one) will simply import everything to get compiled by Meteor with the less package. From the same source package dev could import only what they're interested in and possibly, before the actual import, changing variable values allowing for theming.

It would be awesome if we all could agree on a same approach to be consistently applies to every style library to get devs used on a similar pattern which once leraned would allow them to switch from one front-end framework to the one other in zero time!

Thoughts?

[awatson1978]

Would it be better to have a GitHub organization, or could it simply be a published track in its own namespace?

I ask because it seems like meteor-3rd-party-libraries has a sort of similar interest as clinical:meteor track in keeping track of quality packages. For comparison, you can find the package list for clinical:meteor here. Also, the Meteor Cookbook has also kept a list of community favorite packages for a long time.

The definition of quality is different. Instead of "stable packages that are known to work together", or "packages that are popular with the community and get a lot of use", it seems the goal of that meteor-3rd-party-libraries track might be something along the lines of "packages that are the most up-to-date with their upstream author's version".

But, otherwise, there are a lot of similarities. Why not simply treat it as it's own track? And rather than trying to police other people, or moderate people, and whatnot, just create a space within Meteor that's up-to-date. If unmaintained packages are a bad thing in Atmosphere searches, maybe we can add a way to search by Tracks. Then people could search within the Meteor Up To Date 3rd Party Libraries Track, and know they're getting the latest versions of packages. Or the Clinical Track to get packages that are FDA and HIPAA compliant. Or the Video Game track to get packages that support video games. Or whatever.

[dandv]

@splendido excellent idea to create GitHub organization so initial forks could be put there to keep everything under control. Welcome to https://github.com/MeteorPackaging !

[splendido]

Thanks @dandv, lets see how it goes then ;-)

@awatson1978, I didn't know you're keeping such tracks/lists (almost) up to date, nice discover btw!

...@dandv I might be in line with Abigail about not pushing too much with the 'police' action Still, it is remarkably true we'd need a cleaner way to do searched on Atmosphere.

...might it be just a matter of time? Meteor just turned 1.0... Atmosphere went out of Beta a few months ago... Package maintainers are starting to side up to make the packaging atmosphere less polluted...

I think these kind of discussion are great and I guess MDG and Atmosphere are interested to make things better and better too!

Lets keep this up (perhaps without pushing too much...) and lets see whether this can make developers a bit more sensible collaborative about publishing packages!

[paralin]

This is a good idea, but I'm not sure if package developers will be willing to actually run the Meteor build cycle every time they make an update. Can we write some kind of app to track releases (bower, npm) for a package and pull the latest commits, run buildtool on them?

[dandv]

@paralin: @mbostock was pointing out exactly that, that he'd much rather have Meteor watch GitHub repos and automatically build new releases.

[paralin]

I'm willing to write a quick proof of concept app that could pull from various third party repositories, check their bower or npm package definitions, and update Meteor accordingly.

[manduks]

Deleted manduks/meteor-sencha-extjs#4 repo :)

[dandv]

@manduks I would've said "thanks" in that repo if I could ;)

[samhatoum]

This is such an amazing effort, thank you @dandv so much for doing all this for the community (and all others involved)

I have an idea that I'm going to throw out here that could potentially be less friction and achieves the same outcome: timely releases of officially maintained packages.

It seems to me the process is a two-part process.

1. Updating the meteor package of a port when the source code of the non-meteor library has changed
2. Writing an "adaptor" to get the specific library into a Meteor package.

Part one can be solved with fork + a webhook on the non-meteor package repo that triggers a Travis build (See next point) Part two can be solved with a Travis build that re-creates the adaptor, and runs some tests and makes sure the package works in Meteor

This puts the onus on the Meteor community to make the adaptors (most should be templateable, some will be custom), and a PR may still be submitted to the author to put a mention in the README, or a "Ported to Meteor" badge if you're lucky!

Then we can have a single Meteor namespace that is called official-community-ports (or some better name) and we would find all these official package ports under one namespace. We can still ask for non-official ports to be taken down too.

What are your thoughts?

[zol]

:+1:

[dandv]

Hi @samhatoum, thanks for the kind words!

Part one I believe is something I've also thought would be nice to do - essentially a "Meteor builds" machine that watches repos and triggers meteor publish.

Part two sounds like @raix's wrapper package. The adaptor could be based on the template I've been using for the packages linked in the OP. Hammer.js has the most updated set of files.

Oh and :+1: for this :)

We can still ask for non-official ports to be taken down too.

[dandv]

:cool: :sunglasses:

[paralin]

Well done

On Mon, Dec 8, 2014, 3:57 PM Dan Dascalescu notifications@github.com wrote:

[image: :cool:] [image: :sunglasses:] [image: image] https://cloud.githubusercontent.com/assets/33569/5349949/c925a140-7ef2-11e4-9b04-43760c19bbe6.png

Reply to this email directly or view it on GitHub https://github.com/MeteorCommunity/discussions/issues/14#issuecomment-66211469 .

[PEM--]

I've just starred them :star:

It is incredibly nice to see that coming on Atmosphere. @dandv :+1:

[mizzao]

Is it just me or does that say fortawesome? :P