un-migrate d3 from Atmosphere?

[dandv]

Now that d3 is deprecated, perhaps it should be hidden from Atmosphere searches? It's already gotten 6 stars there.

[awatson1978]

It's super sad that MDG has stopped supporting D3. As a data scientist and bioinformaticists, the fact that Meteor tied together Mongo and D3 together into a single platform is the exact reason I started working with Meteor two years ago. To see them deprecate D3 and to start focusing on SQL integration... it's just kind of sad to see.

[awatson1978]

Adding a bit of SVG support is suppose to replace the all the internals of D3? Hardly. If anybody thought that, they severely under-estimated the functionality of what D3 provides and over-estimated what SVG powered Blaze templates provide. Admittedly, having SVG support opens up all sorts of interesting possibilities. I'm not complaining about the SVG support. Just that the overall focus has been going towards lowest-common-denominator and widest-appeal.

At least we have the different tracks. I'll be supporting D3 via the Clinical Track, although it kinda sucks to have the maintenance burden shifted my way. But, whatever.

One of the other things I had planned for the Clinical Track was to hack meteor-tool, and finally add a couple extra commands I've been asking about for... well, for well over a year now. Currently on the list are:

meteor clone
meteor nightwatch
meteor certify

The meteor clone command I had imagined being very similar to meteor add, so that somebody could do something along the lines of:

cd myapp/client/workflows
meteor clone https://github.com/awatson1978/eula-page.git

Which is basically what you're suggesting with meteor add https://github.com/mbostock/d3.git@3.0.0. I had envisioned it being more for UI scaffolding and components; but I could also imagine it being very useful for registering libraries. The libraries would need to be well formatted in their respective repositories though; which is what the wrappers provide... a good way to manage inconsistent library structure. But as far as just getting the library into the app, that might work.

[glasser]

d3 isn't deprecated.

Shipping a copy of d3 with Meteor whose version is pinned exactly to your Meteor release and which can't be upgraded without also upgrading your Meteor release is deprecated.

[glasser]

(And I'm not sure if our long term plan is "encourage somebody in the community to maintain the best d3 package" or "keep maintaining the d3 package, just not pinned to a particular release".)

[dandv]

@raix:

I personally think there should be a better way to use libraries directly instead of creating wrapper packages. (had hoped that we would have a meteor add bower:d3@3.0.0 or meteor add https://github.com/mbostock/d3.git@3.0.0)

There is a discussion precisely about extending the package system to search among npm and bower modules/packages and a meteor-core thread. Not all packages are directly usable, and many will require manual integration, but the number of discoverable packages would increase by an order of magnitude.

[paralin]

@raix Please do see our discussion on this. I have been arguing against wrapper packages cluttering up the Meteor ecosystem when the software developers will almost always keep an official up-to-date entry in the Bower database.

[awatson1978]

Thanks for chiming in, @glasser. Glad to hear that D3 isn't deprecated (yet).

[awatson1978]

We have different definitions of 'maintained'. The D3 package has been upgraded quite a few times over the past two years, which I'd call maintenance. And since I'm looking for stability, and not bleeding-edge development, the pace of maintenance has been entirely satisfactory.

[awatson1978]

And that's perfectly reasonable, as far as I'm concerned. Please take the time to realize that the packages in core are run through suites of unit tests, used in demo applications, define the overall feature set, and used by organizations when evaluating the overall platform. There are many implications to a package being in core; not just who maintains it, or how often it is maintained.

Up until Meteor 1.0, there was a commitment by Meteor Development Group that Meteor would be guaranteed to work with D3, and was reflected by the Parties example which showcased D3 in action. This made Meteor a compelling choice for people interested in data analytics, data science, bioinformatics, economics, financial analysis, and so forth. With D3 being removed from core and the Parties example no longer being showcased, that guarantee or commitment is no longer there in the same way. It's very sad to see.

[glasser]

We haven't even said that we won't continue to maintain the d3 package (not sure if a decision has been made). We've just changed the fact that in the past, if we wanted to put out a new version of the d3 package, you wouldn't be able to get it without also waiting for a new release of Meteor; and conversely, if we upgraded d3 with a new version of Meteor in a way that you didn't want for your app, you wouldn't be able to upgrade Meteor without also upgrading d3.

[awatson1978]

Thank you for the clarification, David. I understand that things are in flux, and am mostly responding to D3 no longer being in the meteor/packages directory on GitHub. Perhaps the new approach is a good one. I hope so. My gut reaction, however, is that in an effort to appeal to the widest possible audience, Meteor is pivoting in a way that it is loosing focus of what made it so great to begin with. Mongo integrated with D3 through a pure-JavaScript stack was the killer feature for me, as a data scientist. The reactive templating was gravy; not the main dish.

[awatson1978]

@paralin - I have every right to advocate my position and explain that there are alternative viewpoints in the community. While I agree with you that unmaintained packages in Atmosphere are a problem, I seriously disagree with your proposed solutions. I'm sorry that this is an inconvenience for you. But I have a professional responsibility to the companies I work for, as well as a personal interest having worked with Meteor for two years now.

Also, you should check yourself on the personal attacks.

[awatson1978]

@glasser - Just to reiterate and to make sure this is documented, I maintain the claim that institutional organizations care about the presence of a supported graphing library in core Meteor. It's going to be a critical evaluation point for larger institutions evaluating whether to adopt Meteor.

Rather than deprecate D3 completely, I'd suggest that there's a compelling case to take the D3 library, perhaps combine it with NVD3, add some tests which never got fleshed out as well as they should have, and provide a data analytics pipeline package. If Meteor is ever interested in such a sub-project, please let me know, and we can have that conversation offline. I have all the pieces in place for reactive mini-mongo driven D3 graphs, and could piece together a demo-app in short order.

[paralin]

I quote @glasser: "We haven't even said that we won't continue to maintain the d3 package" although I do agree some tests might be helpful.

[dandv]

@paralin: what was listed under core projects? Don't see D3 listed anywhere on https://www.meteor.com/projects.

[paralin]

Apologies for my response in this thread. I believed @awatson1978 to be complaining that d3 was somehow not going to be maintained after the MDG group removed it from the core packages. I responded attempting to explain (albeit in a somewhat frustrated manner) that d3 would continue to be supported by its original authors. In reality she is somewhat upset that they seem to be moving away from it as a priority, and feels a Blaze<->d3 integrated system should be written to allow "Mongo integrated with D3".

@dandv I was referring to Mongo <-> Templating support. I didn't realize she meant Mongo<->d3 support as that does not exist at the moment (last I checked) and @awatson1978 was describing it like a feature that was being phased out.

I've cleared out some of my older post since it all really says the same thing: there has not been any real change YET for d3 from before. With the release of 1.0 and the new package system the MDG team has been separating out all of the packages into separate pieces so they can be independently versioned and updated. I don't think this really qualifies for an upset over the project "pivoting in a way that it is loosing focus", but that is my opinion, and I cannot force that on anyone else.

[dandv]

@glasser: I've submitted a PR to @mbostock for official d3 <-> Meteor integration.

Perhaps time to hide the mdg package from Atmosphere?

[awatson1978]

How about waiting until after the pull request is accepted at least? What is up with this need to purge D3? It's not hurting things as-is, and there are other people using the package still. Why do you feel this need to rock the boat and disrupt the processes that I and the companies I work for and many other people use?

I was letting this chill after paralin's reply (thank you by the way, it's a rare apology that addresses circumstances and acknowledges situational differences; apology accepted, and no hard feelings). But I feel compelled to speak up, again, and say 'leave this alone, already'.

Look, of all the packages that are included in core, D3 is the one that brings in money. On the East Coast, which is the financial center of the US, and also heavy in advertising and biotech, the best paid jobs in programming all involve analytics to some degree. I can tell you right now that a Parties-like demo, using D3 and an ActiveRecord pattern of some type and a user accounts package, has a market rate of about $20,000 right now and a turn-around time of about 6 weeks. Do the math on what kind of consulting and what kind of salaries can be made.

Every project I have has some request for analytics. Last year, I was a full-time Meteor programmer (one of the first in the community, I dare say) because I found a company that needed an analytics platform with a half-dozen graphs that kept me busy all year long. Same story with each other consulting opportunity that pays real money.

One of the Meteor Development Group's stated goals is to provide an ecosystem for consulting and professional services. And D3 is bread-and-butter for those kinds of services on the East Coast, where financial and biotech industries are common. D3 shouldn't be removed and purged. It should be given some TLC now that some of the other core components have been taken care of.

[paralin]

Abigail nothing has changed with d3, it is just namespaced differently. This actually allows for more free updates and separate versioning. It is not a sign of lack of support.

On Tue, Nov 25, 2014, 6:44 AM Abigail Watson notifications@github.com wrote:

How about waiting until after the pull request is accepted at least? What is up with this need to purge D3? It's not hurting things as-is, and there are other people using the package still. Why do you feel this need to rock the boat and disrupt the processes that I and the companies I work for and many other people use?

I was letting this chill after paralin's reply (thank you by the way, it's a rare apology that addresses circumstances and acknowledges situational differences; apology accepted, and no hard feelings). But I feel compelled to speak up, again, and say 'leave this alone, already'.

Look, of all the packages that are included in core, D3 is the one that brings in money. On the East Coast, which is the financial center of the US, and also heavy in advertising and biotech, the best paid jobs in programming all involve analytics to some degree. I can tell you right now that a Parties-like demo, using D3 and an ActiveRecord pattern of some type and a user accounts package, has a market rate of about $20,000 right now and a turn-around time of about 6 weeks. Do the math on what kind of consulting and what kind of salaries can be made.

Every project I have has some request for analytics. Last year, I was a full-time Meteor programmer (one of the first in the community, I dare say) because I found a company that needed an analytics platform with a half-dozen graphs that kept me busy all year long. Same story with each other consulting opportunity that pays real money.

One of the Meteor Development Group's stated goals is to provide an ecosystem for consulting and professional services. And D3 is bread-and-butter for those kinds of services on the East Coast, where financial and biotech industries are common. D3 shouldn't be removed and purged. It should be given some TLC now that some of the other core components have been taken care of.

Reply to this email directly or view it on GitHub https://github.com/meteor/meteor/issues/3082#issuecomment-64409074.

[awatson1978]

You're discussing technical, I'm discussing business. The change in namespacing has implications from a business perspective that I don't think people are considering.... core product, core services, brand-identity, marketing. This proposed change of namespace has all sorts of impact with how consultants and service providers can market Meteor as a solution to financial and biotech applications.

As somebody who is liable to be impacted by this namespacing change, I'm putting it in writing that taking D3 out of of the core namespace signals all sorts of things. Most importantly, it signals that Meteor doesn't have an officially supported analytics and graphing package. From that, it also signals that Meteor isn't serious about trying to develop financial or biotech applications, or catering to those markets.

Speaking in semi-official capacity on behalf of the NY Meteor Meetup and the Meteor Healthcare Working Group, those proposed changes in brand identity and marketing and namespacing are worrisome. Taking the Parties example down has been a big enough cause for concern. For those of us who have been establishing careers around Meteor being applicable in the financial, advertising, and/or biotech space, and who pay our rents with D3 being integrated into our products... it's more than just a namespace change.

[awatson1978]

Agreed with regard to bootstrap and underscore being in the same boat as D3.

I'm not entirely unsympathetic to the desire to keep core lean and small. At my last job, we had some weird constraints with our datacenter, and were working with 6GB SAN of disk space for our virtualized server instances. For a B2B application that's only 30 or 40MB large, that should have been plenty. But the core Meteor installation takes up 500 to 600MB of space, and we wound up with issues compiling directly on the server when Meteor got installed 3x times in different user accounts. So, I get it. I really do. Small core footprint is a good thing.

But what's being asked is to take out a core package that's been there for nearly two years, and which people have built up best practices for, documentation for, tech demos, sales pitches, niche markets, and consulting practices around.

In theory, the tracks that publish-release creates may be able to provide a sort of middle ground. A track doesn't necessarily have the level of support, resources, and visibility that the core platform has. But specialized libraries could be used to create new stacks. Maybe somebody wants to use Famo.us to create a next-generation video-game stack using Meteor. Well, I don't necessarily want Famo.us in core. So there could be a Video Game Track. And D3 can wind up in the Financial Track or Clinical Track.

But as a data scientist who came to Meteor early because it was the only pure-javascript platform around to have native D3 integration, I think the track solution is still a poor compromise compared to keeping D3 in core and building on it. As I've offered before and elsewhere; if there's a willingness from the MDG to accept a pull-request, I'd be happy to provide an analytics pipeline with D3, NVD3, statistics writers, aggregation, etc.

[paralin]

Meteor takes up 500mb of space? Really?

At least bundled it should be pretty small. On Tue Nov 25 2014 at 9:22:25 AM Abigail Watson notifications@github.com wrote:

@mizzao https://github.com/mizzao and @lourd https://github.com/lourd... care to chime in on this discussion?

Reply to this email directly or view it on GitHub https://github.com/meteor/meteor/issues/3082#issuecomment-64436811.

[glasser]

@awatson1978 I really don't think you're taking this discussion in a productive direction.

I'm sorry that we've been a little distracted from GitHub issues for the past few weeks and haven't had a chance to really chime in, but I think you're making a mountain out of a molehill here and are reading a lot into a tiny technical change. I think this philosophical discussion is better suited to the mailing lists or your blog than to our bug tracker.

[paralin]

@awatson1978 In addition to what glasser said, literally all you have to do to update your apps is change "d3" to "d3js:d3" which is supported by the official D3 team.

By the way that comes straight from the official D3 github repository so you're getting very frequent updates and development. Check the commit log. Last one was 12 days ago.

[awatson1978]

@paralin - Indeed. I was surprised at the size. And its more if the mobile SDKs are installed.

@glaser... maybe it's a molehill for others, but this tiny technical change has many business consequences. Like how supporting Strings vs ObjectIds with timestamps in record _id field impacts whether business can use Meteor on pre-existing Mongo databases or not. Small technical difference between Strings and ObjectIds with timestamps, but big business impact.

This is the same kind of thing. Tiny technical change. But there are business consequences.

And I think this discussion should be taken to the mailing list. This isn't a bug, and this ticket should never have been logged in the bug tracker to begin with. This is a business/philosophical change that Dan has proposed, logged as a bug.

@paralin - Indeed. And when I produce the .meteor/versions file to a client during an audit, and they inspect all the packages that are being used for intellectual property, HIPAA compliance, SOX compliance, SLA support responsibilities, etc. They look at whether it's d3js:d3 versus mrt:d3. The bean-counter doing the audit won't appreciate the difference, but will start saying things like 'Oh, so it's not a core package. We're going to need extra staff to maintain this, right? How will this impact the total cost of ownership? etc.'

Look, I pick my arguments fairly carefully. There are hundreds other bug tickets I don't make a mountain out of. I'm making a mountain out of this one because it has business consequences. Namespacing matters. Namespacing impacts business/strategy, and should be taken to the mailing list. This should never have been logged in the bug tracker to begin with.

[lourd]

Sorry Abigail, I agree that d3 should be taken out of the core platform. It was great that d3 was part of core for these past couple years, but that was only necessary because there wasn't a robust package manager. I see this as a success of the community and pledge of faith from MDG in Atmosphere. There's no bigger endorsement of the packaging system than using it themselves.

3rd party components should never be inseparably intertwined. Even 1st party components shouldn't be. Shaving down core and maintaining its components independently will enable better maintenance and innovation, not less.

For example, if I wanted to make a reactive-d3 package, implementing Tracker-aware data sources, I don't want my package to be tied to an old version of Meteor because I used an old version of d3. On the flip side, if @mbostock wanted to make reactive-d3 himself, we shouldn't have to wait for MDG to integrate the newer version. It's separation of concerns.

I don't think that a namespace is going to make a difference in an audit, especially when any large project is going to have at least 15-20 different packages in them, most of them with different namespaces. It's not like a namespace means anything anyway, when so many packages are just wrappers of other libraries anyway. This is open-source, it's not all first party, it's made together, and that's better! You know that.

[awatson1978]

Hmm. Okay. Well, I've said my piece. If D3 is removed, I hope that we're at least consistent, and also remove Underscore, Bootstrap, etc.

This isn't something that should have wound up in the bug tracker. D3 is a feature, not a flaw.

[paralin]

@awatson1978 "If D3 is removed" it was not removed. It was separated. The people who originally wrote D3 (not the MDG people!) are maintaining it and have full control over the package, rather than relying on MDG to pin the package to a newer version.

In terms of auditing it should be MORE reliable. You have the official repository for the project linked to the package. They will say, oh, there are hundreds of contributors here and it's the same codebase used in hundreds of other apps. This is better than seeing an out of date package loosely pinned to the latest D3 but also bound to an older version of Meteor.

Those other packages are slowly being migrated as well. It's not wise to keep everything in one repository like this, especially when version pinning is involved.

[lourd]

Agreed, underscore and bootstrap should be separated as well. A brief search doesn't turn up any current issues filed to do that; is there any reason not to file those @glasser ?

[mizzao]

Bootstrap 2 should certainly be taken out; underscore is being used by Meteor itself and it makes sense for developers to use the unipackage version for that.

Not sure what is going on in this thread, but yesterday I made a small issue of Meteor not having an updated, well-maintained d3 package (sergeyt/meteor-d3#1) and today there is an officially supported package released by d3.js. As far as I'm concerned, that was the fastest fix I've ever seen, and I'm just going to switch over to using that.

In fact, I'm going to try taking a cue from @dandv and try this for other prominent wrapper packages I'm maintaining, because it sure reduces the amount of work I have to do. Also, the willingness of package authors to directly publish in Atmosphere is a testament to Meteor's growing endurance.

[awatson1978]

Fair enough. I created a thread in meteor-core to discuss the implications of un-migrating packages that have been in core and associated integration testing issues.

[lourd]

@mizzao nice! Here's what Andrew's talking about:

• @dandv's pull request on the d3 repo
• The commit in the pull request
• The new, official d3 package on Atmosphere!

Thanks @mbostock!

[paralin]

Is there any real technical reason ALL of these packages can't be separated out? Would it be possible for Meteor to depend on packages in its own ecosystem?

[lourd]

I think maintained rather than separated out would be more accurate. Meteor already is a collection of packages, just take a look . Each package is separately declared and published on Atmosphere. I don't know what the difference would be for to have a packages manifesto like .meteor/packages rather than the /packages directory, but for all intents and purposes Meteor is dependent on packages in its own ecosystem.

I think it would be more clear if the core packages were published under the meteor namespace e.g. meteor:appcache rather than just the name. Or maybe mdg instead. Are there any reasons why each core package doesn't have its own repository?

[paralin]

Sorry, that was what I meant by separate. They ought to have their own revision histories at this point.

On Tue, Nov 25, 2014, 12:45 PM Louis DeScioli notifications@github.com wrote:

I think maintained rather than separated out would be more accurate. Meteor already is a collection of packages, just take a look https://github.com/meteor/meteor/tree/devel/packages. Each package is separately declared and published on Atmosphere. I don't know what the difference would be for to have a packages manifesto like .meteor/packages rather than the /packages directory, but for all intents and purposes Meteor is dependent on packages in its own ecosystem.

I think it would be more clear if the core packages were published under the meteor namespace e.g. meteor:appcache rather than just the name. Or maybe mdg instead. Are there any reasons why each core package doesn't have its own repository?

Reply to this email directly or view it on GitHub https://github.com/meteor/meteor/issues/3082#issuecomment-64467717.

[mizzao]

@lourd I spoke too soon; confused @dandv's published package and his pull request with an officially supported package. Hopefully he'll be successful at getting the buy-in, though.

[awatson1978]

Integration testing is the big technical hurdle I can think of. Keep in mind the meteor test-packages command. It's behavior has changed a bit of late, so it doesn't run through all the packages by default like it used to. That makes it more of a unit testing tool nowdays, rather than an integration testing tool like it used to be. Part of everything being in the github/meteor directory has traditionally involved it being run through test-packages and showing up green before releases. The packages might be able to be separately maintained, but what's the process to ensure that they work together?

Tracks seem like they're the way to go, but Ronen and Morten seem to be the only people who have managed to get them to work yet. Get them in place so the public can use them, and un-migrating core packages becomes less of an issue.

[mizzao]

@awatson1978 There are no tests for d3 in Meteor; it's just thrown in the client side and assumed to work. What integration tests are you talking about?

If you are writing your own tests you can just api.use("d3js:d3") or whatever in Package.onTest.

[glasser]

I thought I requested that the abstract philosophical discussions use a forum other than a bug report that I haven't gotten around to closing yet in our bug tracker. Please find a better forum for this discussion.

[raix]

Last comment from me: I've finally managed to published a test release that allows: $ meteor add bower:d3 --release iso:METEOR@1.0-bower.7 https://github.com/raix/Meteor-include Also feel free to use https://github.com/raix/Meteor-community-discussions for tracking technical issues etc. Edit: or abstract philosophical discussions

[paralin]

@raix Wow, cool! That's very elegant. I like the GitHub option for pulling from repositories. I'll refrain from posting anymore in this issue, though.

[dandv]

@mbostock has accepted the Meteor integration PR. Can the powers that be please update the Meteor docs and point to it?

[mizzao]

Maybe it would be helpful to have a way to alias package names - i.e. d3 alias for d3js:d3, that way meteor add d3 installs the official d3 package just as @mbostock requested.