Investigate swapping Underscore w/ Lo-Dash or upgrading Underscore

[jdalton]

_86 Upvotes_ It's been mentioned a few times, so I figured I'd track progress/questions with an issue.

_85 Upvotes_

[dwelle]

+1

[glasser]

At some point we should move off of Underscore 1.5.2, either to its latest version or to Lodash. Both of these changes will require thorough QA, of course.

[JamesMGreene]

At some point we should move off of Underscore 1.5.2, either to its latest version or to Lo[D]ash. Both of [T]hese changes will require thorough QA, of course.

:wink:

[rclai]

+100,000,000.

There, it's gonna happen. :P

[hanchang]

@glasser , I really hope the decision is to move to LoDash and off of Underscore!

For those of you who want to switch before it's officially supported, I just ran meteor add stevezhu:lodash in my Meteor project root, added _ = lodash as the first line to my lib/constants.js (or any other initializer) and all seems to be well.

[ai10]

+1
comments re lodash 3.0 announcement https://news.ycombinator.com/item?id=8952291 third party opinions can provide perspective absent from autobiography.

[jdalton]

A better source for info may be the official lodash v3.0.0 release notes, though comments from non-JS users and anonymous trolls can provide some amusement.

[rgoomar]

The new changes for lo-dash 3.0 are awesome! And huge performance boosts.

[JamesMGreene]

So excited for lazy evaluation in Lo-Dash v3.0.0! Now if only the Meteor core team would get on board.... :wink:

[digioak-zz]

+1

[donflopez]

+1 too!

[sabind]

+1

[fermuch]

+1!

[matteodem]

+1

[robertlagrant]

+1

[comerc]

Wanted translators: https://github.com/comerc/lodash-ru

[orlaqp]

+1

Please add it and let me/us know if there is anything we can do to help with the transition. Main reasons for me:

• Speed
• API consistency
• cloneDeep :-)
• plus all the new goodies from version 3.0.0 (https://github.com/lodash/lodash/releases/tag/3.0.0)

[javierbyte]

+1

[Tarang]

Just one thing to be careful with lodash is its not 100% compatible with underscore. Lodash always produces what .chain() would produce if you use _(xx) so it may have some issues on existing projects.

i.e _([1,2,3]).map(function(x){return x}) => [1,2,3] //Underscore _([1,2,3]).map(function(x){return x}) => (Wrapper Object) //Lodash

[jdalton]

With v3 Lodash has discontinued its Underscore compat build in favor of devs using Lodash modules to supplement Underscore until the full transition to Lodash is doable. That said, if Meteor required it, we'd work something out for it.

[JamesMGreene]

@jdalton: Is there some v3.x-based documentation that explains potential conflicts/issues if using LoDash as a drop-in Underscore replacement (without the compat build)? Their base APIs are obviously very similar one way or another but I'm curious about the actual gotchas when using the full build.

[awatson1978]

Lodash looks great. But just like every time we upgrade a Node minor version, the day I swap in lodash will be a day where I do a full automated and manual QA of Meteor. Which is something we're trying to make more and more automated, but we're not there yet.

What exactly would a fully automated and manual QA of Meteor look like? There's obviously TinyTest on the package side. And on the end-to-end side, the Meteor Cookbook has some two dozen example apps that are running on Travis.

https://github.com/awatson1978/meteor-cookbook

I'm still in the work of writing tests for all of these example apps (and will be for the next few months). But I suspect something like these benchmarks are what @glasser and folks are looking for with regard to the QA process needed to do the LoDash refactor?

If people can start brain storming the edge-cases that we would need to test to migrate to LoDash, we could start writing tests for them. For instance, _.each() would need to iterate over an array. Clinical Workqueues has a tagging infrastructure that uses _.each(), as I recall, so we could use it to benchmark that functionality. And so, on for each of the Underscore functions.

Do we just go through the entire Underscore API docs and find an applet or demo that implements that API? Or should we create a kitchen-sink style app that uses every underscore call, and use that as a benchmark?

[jdalton]

@JamesMGreene

Is there some v3.x-based documentation that explains potential conflicts/issues if using LoDash as a drop-in Underscore replacement (without the compat build)? Their base APIs are obviously very similar one way or another but I'm curious about the actual gotchas when using the full build.

I've created a wiki page for migrating that lists known differences.

[JamesMGreene]

Thanks, @jdalton! Very useful and detailed information as per usual. Seems like pretty minor differences overall but a few little potential gotchas — and mostly just more LoDash benefits. :wink:

[matteosaporiti]

+1 for improved speed (third party benchmark where lodash is one of the fastest library while underscore is one of the slowest), better documentation and many useful additions (cloneDeep being one of them).

[cslamber]

+1

[madvas]

+1

[benstr]

I felt left out... throwing my hat in. +1

[yoh]

+1

[smowden]

+1

[rclai]

Is there a way to swap underscore to lodash as a local hijacked underscore package in order to test? I tried to copy the files of the underscore package into a local package and simply replaced the underscore files/includes with lodash files and for some reason when starting Meteor it is complaining that there is a circular dependency issue with underscore and meteor (the package). I expected it to just work since local packages take precedence right? So my hijacked version of underscore should override the other one?

I'd really like this thread to be less +1s and more of figuring out what initial steps we can take to start testing this at least unofficially.

[jonagoldman]

+1

[tungv]

+1

[firdausramlan]

+1

[mitar]

Please at least upgrade the underscore to the latest version.

[tmikoss]

+1

[jonagoldman]

Guys at meteor should see this talk (and everyone really): https://www.youtube.com/watch?v=cD9utLH3QOk (Lo-Dash and JavaScript Performance Optimizations)

[mazswojejzony]

+1

[voxlet]

+1

[bryankennedy]

It seems like one step forward would be getting this on the Meteor roadmap. I'm guessing that until we see it on there, it isn't a priority for the MDG.

[rclai]

Hey guys, so I've created a repository with a solution to actually replace underscore under the hood with lodash: https://github.com/rclai/meteor-lodash-replace-underscore Please feel free to contribute or let me know of any suggestions.

[stachrom]

+1

[Tarang]

Gosh I wish there weren't so many '+1's in here. It makes it really difficult to scroll through the actual issues that need to be addressed with the differences between the two.

I've been trying to narrow down some of the differences so a 'drop-in' replacement can be used such as the one by @rclai or any others on Atmosphere.

In addition to the chaining issue with the previous comment I made there is another difference:

Some methods on Lodash inconsistently exit the chaining method to give out values. Examples of this are .reduce. There are also a few more methods on Lodash so .sum also exits the chain:

Underscore (Chains)

_([1,2,3]).map(function(x){return x}) => [1,2,3]
_([1,2,3]).map(function(x){return x}).filter(....) => [1,2]  //Assuming 3 is filtered out
_([1,2,3]).map(function(x){return x}).min() => 1
_([1,2,3]).map(function(x){return x}).sum() => [Error] no method 'sum'

No method on sum- but that's not an issue if lodash were to be the replacement

Lodash (Chained already by using the _(xx) notation)

_([1,2,3]).map(function(x){return x}) => [Chain object]
_([1,2,3]).map(function(x){return x}).filter(....) => [Chain object]
_([1,2,3]).map(function(x){return x}).min() => 1 //Chain is exited - inconsistent
_([1,2,3]).map(function(x){return x}).sum() => 6 //Chain is exited - inconsistent

So Lodash does use chaining with the _(xxx) notation and returns chains but often can exit the chain on certain methods instead of keeping the chain.

[jdalton]

@Tarang

Your summary of Underscore chaining is incorrect. Without chain() Underscore won't make it passed the first method call. So your example of _([1,2,3]).map(function(x){return x}).min() is incorrect as map would return the unwrapped array which would lack a min method.

Also Lodash chaining behavior is more jQuery-like. Methods that operate on and return arrays, collections, and functions can be chained together. Methods that return a boolean or single value will automatically end the chain returning the unwrapped value. Explicit chaining may be enabled using _.chain.

[Tarang]

@jdalton Yes, you're correct I mistakenly copied out the wrong line. The inconsistency with Lodash's chaining issue still remains though which was what I was trying to demonstrate using the underscore example.

We need to find a way to allow Lodash to work as a drop-in replacement for Underscore that way MDG would perhaps look into the issue a bit deeper. It may be similar to JQuery and the design could be more consistent with other public libraries, but this argument isn't helping the case about existing projects using underscore at the moment as they would be rendered incompatible.

If .chain() forces explicit chaining the _(xx) chaining style is a bit confusing and inconsistent. I think one of the things that could help is this notation followed underscore's lead. At least this would be needed for it to be a drop-in today replacement over underscore a viable replacement in the future.

[jdalton]

The inconsistency with Lodash's chaining issue still remains though which was what I was trying to demonstrate using the underscore example.

Your wording makes it sound like Lodash has a problem with its chaining. It's not an inconsistency, as in some oversight or bug, it's a different behavior.

We need to find a way to allow Lodash to work as a drop-in replacement for Underscore that way MDG would perhaps look into the issue a bit deeper.

That's not a good route. Lodash 3 has features that would be nixed if it were a drop-in replacement.

If .chain() forces explicit chaining the _(xx) chaining style is a bit confusing and inconsistent.

I disagree. It gives devs an intuitive-feeling chain by default but allows them to explicitly force chaining if they want. It also allows writing code that works in both Underscore and Lodash.

I think one of the things that could help is this notation followed underscore's lead.

Lodash is leading now.

[Tarang]

Your wording makes it sound like Lodash lodash is leading now.

Apologies for this. I'm looking at it from the point of view where I can use Lodash instead of Underscore in my apps. I prefer Lodash.

This issue isn't about Lodash vs Underscore which is alot of where you're basing your arguments on. I think we all know Lodash has plenty of advantages over Underscore.

The problem is over the past 2 years there are tonnes of Meteor apps people have made using Underscore and I'm quite sure the primary reason MDG can't immediately swap Underscore with Lodash is these apps would suddenly start crashing due to issues such as the chaining issue as the code would operate as unintended.

This issue will remain +1s unless we can find a meaningful way to transition these users to Lodash without breaking their projects. It's the performance that is the advantage, perhaps the syntax but a broken app is a regression when the argument for it is an all out transition to Lodash with its syntax as it stands now.

[jdalton]

but a broken app is a regression when the argument for it is an all out transition to lodash.

Save it for a major bump. Roadmap that bump. Even jQuery 3.0 is introducing back-compat breaking changes ;)

[Tarang]

@jdalton Meteor is used by lots of people and version 1.0's motto was there would be no major breaking changes going forward. JQuery & Underscore are the 'core' 3rd party packages used. Realistically the JQuery issue is still to come up forth and the package included with Meteor will probably not be upgraded as soon because of this.

If you look at everything MDG has done so far with breaking changes they have mostly put out paths for backwards compatibility with the exception of Meteorite's upgrade where they & Percolate actively engaged package maintainers to update their code and even helped them in many cases. The other was the file scoped variables introduced where eventually they added the compatiblity folder to ensure no one was left behind.

You can't expect everyone to suddenly adapt to use Lodash in the way you expect for the sake of syntax. There are a good deal of Stackoverflow questions and blog posts that break down suddenly. Lets not forget Meteor is a learner's platform too.

Perhaps eventually users can use the Lodash syntax but there needs to be a realistic upgrade path. Where the user can remove a compat like package (like this one: https://github.com/meteor/meteor/blob/devel/packages/deps/package.js) to get it to behave naturally.

I think alot of us, you included, want to start using Lodash with Meteor today and not on Meteor 2.0.

[jdalton]

You can't expect everyone to suddenly adapt to use Lodash in the way you expect for the sake of syntax.

I don't expect anyone to suddenly adopt. This issue is more than two years old now. The version of Underscore Meteor uses is almost as old. There's nothing sudden about this process.

I think a lot of us, you included, want to start using Lodash with Meteor today and not on Meteor 2.0.

I have no preference. If compat is a concern put Lodash on the roadmap for 2.0.

For a start MDG could maintain an official Lodash package, as they do with Underscore, and move on from there. Since Lodash follows semver they can update it more often without risk of compat issues and then separately, over time, work it into a future release.