search

dc-js / dc.js

Multi-Dimensional charting built to work natively with crossfilter rendered with d3.js
Apache License 2.0
7.42k stars 1.8k forks source link

Support D3 v4 #1173

Closed jfsiii closed 6 years ago

jfsiii commented 8 years ago

D3 is being broken up into smaller modules for v4 (e.g. d3-scale, d3-selection, etc) and there are some API changes.

Discuss version support & approach. Should dc.js support d3 v3 and v4 or only one of them? One option to support both is a jQuery-style approach where 2.x supports v3 and 3.x supports v4.

ruhley commented 8 years ago

I would love to move to d3 v4, but I know it would be a big task. I will have a quick go at upgrading on my branch to gauge how hard it would be. I would also love dc to take some of the best practices that d3 now uses:

D3 4.0 is modular. Instead of one library, D3 is now many small libraries that are designed to work together. You can pick and choose which parts to use as you see fit. Each library is maintained in its own repository, allowing decentralized ownership and independent release cycles.

You can also cat D3 microlibraries into a custom bundle, or use tools such as Webpack and Rollup to create optimized bundles. Custom bundles are great for applications that use a subset of D3’s features; for example, a React chart library might use D3 for scales and shapes, and React to manipulate the DOM. The D3 microlibraries are written as ES6 modules, and Rollup lets you pick at the symbol level to produce smaller bundles.

The default UMD bundle is now anonymous. No d3 global is exported if AMD or CommonJS is detected. In a vanilla environment, the D3 microlibraries share the d3 global, even if you load them independently; thus, code you write is the same whether or not you use the default bundle. (See Let’s Make a (D3) Plugin for more.)

In short: modular, extendable and anonymous.

gordonwoodhull commented 8 years ago

Nods. All good things.

Actually I don't think this is the most difficult task ahead for d3 - some of the design and debugging problems are really fiendish.

And I think it's a pretty safe change.

However, I do want to watch the adoption of v4 because it is likely there will be some gotchas and best practices to be learned as it rolls out.

(Does modularity require splitting dc.js into a dozen or more repos? Because that will definitely increase the cost of maintenance.)

XaserAcheron commented 8 years ago

Presumably the modularity just translates to "simply include the parts of d3 that dc.js requires," without any sort of funkiness on dc's side.

I'm not sure if there's any part of dc.js itself that would really make sense to split off into a separate bit -- anonymous-ness, perhaps, though.

ruhley commented 8 years ago

I think modular charts would be great. If I only want to have a bar and pie chart then I only need to include those, not the whole library. Thought at 82.6 KB minified it's not that bad, but going forward adding more charts to dc will make it bigger.

I tested d3 v4 on my branch and managed to fairly easily get something to work. Most things are just renames (e.g. d3.time.format to d3.timeFormat)

dc js bar chart example

However it seems that d3.layout.stack (now d3.stack) has changed quite a bit so there is no data rendering. Though as @gordonwoodhull says there will be more gotchas during the implementation.

gordonwoodhull commented 8 years ago

The idea that plugins are the same as built-ins is really good, too.

I mean you can already plug in most chart PRs safely, but it's awkward and requires more git-fu than most people need. I think if we adopt the same architecture then we can just have a gallery of contributions and when they get to a certain level (usually involving tests, unfortunately) they can migrate to the main repo or organization. It should help with getting more feedback as the charts evolve.

I think there are breaking changes in every d3 component, even d3.selection is a little different IIUC.

That's regular migration stuff.

I'm more worried that we understand the right way to include modules and watch as best practices for modules and plugins emerge. This, for example --

https://github.com/d3/d3/issues/2793#issuecomment-229074764

-- is still Greek to me, but I'm sure in a couple of months it will make perfect sense!

XaserAcheron commented 8 years ago

Huh, modularity would indeed be useful for third-party charts & PRs & whatnot. Hadn't thought of that.

Dunno if I have anything else to add, but there you go.

ruhley commented 8 years ago

So what is the plan then? Use @jfsiii suggetion of finishing off dc v2 using d3 v3 and then create dc v3 using d3 v4? Would upgrading to d3 v4 warrant an almost complete re-write?

gordonwoodhull commented 8 years ago

Something like that. I've been putting breaking changes into 2.1 (semantic version without quite so much inflation), but I don't really care if we call it 2.2 or 3.

I doubt it's a rewrite as there are a lot of subtle details to the way the code is now. Of course if you want to write dc.js from scratch, no one's stopping you, but I'd rather see this repo evolve... Many chart libraries have died from trying to change everything at once.

gordonwoodhull commented 8 years ago

I'd also rather port to the v4 interface before modularizing, if we do break dc.js up into lots of repos: it will become a lot more difficult to port fixes, and I think there is value to fixing the hundred or so issues identified as interface - compatible with 2.0.

Perhaps I am too conservative. Unfortunately this is not a full-time gig for me.

ruhley commented 8 years ago

I was playing around with Angular 2 the other day and implementing the material design components - https://github.com/angular/material2. The whole thing is modular but it is all still housed in the one repository. Then each area has it's own guide (e.g. https://github.com/angular/material2/tree/master/src/components/input). I found this much easier to use as all the code is still in one place, but it is organised and it even means the documentation is modular (unlike the dc documentation which is bigger than Ben-Hur).

gordonwoodhull commented 8 years ago

Cool! I will check it out.

BTW, thanks to @mtraynham, we now have HTML documentation which you may find more navigable. We should advertise it better!

Suggestions welcome.

mtraynham commented 8 years ago

I've put together this table which should help identify all uses of d3 in dc.js and references to the d3 4.0 counterpart.
Note: This table does not currently outline any inconsistencies and breaking changes, but I will add them if they are identified.

D3 V3 Reference D3 V4 Package D3 V4 Global Reference D3 V4 ES6 Import
d3.functor NA NA NA
d3.dispatch d3-dispatch d3.dispatch import dispatch from 'd3-dispatch/src/dispatch'
d3.select d3-selection d3.select import select from 'd3-selection/src/select'
d3.mouse d3-selection d3.mouse import mouse from 'd3-selection/src/mouse'
d3.event d3-selection d3.event import event from 'd3-selection/src/selection/on'
d3.scale.ordinal d3-scale d3.scaleOrdinal import scaleOrdinal from 'd3-scale/src/ordinal'
d3.scale.linear d3-scale d3.scaleLinear import scaleLinear from 'd3-scale/src/linear'
d3.scale.quantize d3-scale d3.scaleQuantize import scaleQuantize from 'd3-scale/src/quantize'
d3.scale.category20c d3-scale d3.schemeCategory20c import category20c from 'd3-scale/src/category20c'
d3.interpolate d3-interpolate d3.interpolate import interpolate from 'd3-interpolate/src/value'
d3.interpolateHcl d3-interpolate d3.interpolateHcl import interpolateHcl from 'd3-interpolate/src/hcl'
d3.interpolateNumber d3-interpolate d3.interpolateNumber import interpolateNumber from 'd3-interpolate/src/number'
d3.quantile d3-array d3.quantile import quantile from 'd3-array/src/quantile'
d3.min d3-array d3.min import min from 'd3-array/src/min'
d3.max d3-array d3.max import max from 'd3-array/src/max'
d3.sum d3-array d3.sum import sum from 'd3-array/src/sum'
d3.range d3-array d3.range import range from 'd3-array/src/range'
d3.extent d3-array d3.extent import extent from 'd3-array/src/extent'
d3.ascending d3-array d3.ascending import ascending from 'd3-array/src/ascending'
d3.descending d3-array d3.descending import descending from 'd3-array/src/descending'
d3.timer d3-timer d3.timer import {timer} from 'd3-timer/src/timer'
d3.set d3-collection d3.set import set from 'd3-collection/src/set'
d3.nest d3-collection d3.nest import nest from 'd3-collection/src/nest'
d3.svg.arc d3-shape d3.arc import arc from 'd3-shape/src/arc'
d3.svg.area d3-shape d3.area import line from 'd3-shape/src/area'
d3.svg.line d3-shape d3.line import line from 'd3-shape/src/line'
d3.svg.symbol d3-shape d3.symbol import symbol from 'd3-shape/src/symbol'
d3.layout.pie d3-shape d3.pie import pie from 'd3-shape/src/pie'
d3.layout.stack d3-shape d3.stack import stack from 'd3-shape/src/stack'
d3.svg.axis d3-axis d3.axisTop import {axisBottom, axisLeft} from 'd3-axis/src/axis'
d3.svg.brush d3-brush d3.brush import {brushX, brushY, default as brush} from 'd3-brush/src/brush'
d3.format d3-format d3.format import {format, utcFormat} from 'd3-format/src/defaultLocale'
d3.time.format d3-time-format d3.timeFormat import {timeFormat, utcFormat} from 'd3-time-format/src/defaultLocale'
d3.geo.path d3-geo d3.geoPath import path from 'd3-geo/src/path/index'
d3.behavior.zoom d3-zoom d3.zoom import zoom from 'd3-zoom/src/zoom'
gordonwoodhull commented 8 years ago

Here is the change log, for anyone who is interested. There are minor breaking changes to every single module, it seems:

https://github.com/d3/d3/blob/master/CHANGES.md

Warning: it's a long read!

gordonwoodhull commented 8 years ago

This seems to be the best overview and rationale of the major changes: https://medium.com/@mbostock/what-makes-software-good-943557f8a488#.m7uanuhgo

You can skip the philosophizing at the beginning and go straight to the cases, which are the real point.

The biggest change is case 1: enter().append() no longer modifies the update selection. It will take some staring at our code to determine where we rely on this.

And that's the point: it wasn't explicit before, it was hidden. Our code will become clearer by explicitly merging selections. But we have to read it carefully to figure out where we need to.

gordonwoodhull commented 8 years ago

Looks like brushes might be one of the biggest changes that affects dc.js; some hints in this comment: https://github.com/d3/d3-brush/issues/14#issuecomment-250993767

stillesjo commented 8 years ago

It's been a couple of months since this was first brought up 😄. What's your thoughts about updating to use v4 of d3, @gordonwoodhull ?

gordonwoodhull commented 8 years ago

I'd love to do it, however it's a ton of work. I'm not concerned about the component renames, but the APIs and semantics of pretty much every component have changed, as detailed in the CHANGES doc I linked above.

If anyone wants to start working on this in a fork, that would be excellent.

In terms of versioning, I figure I want to first pull in all the big pull requests into dc.js 2.1, and then port them to d3v4 in dc.js 3.0.

I have a bunch of branches where I fixed the worst problems in the 2.0 betas, and in most cases I discovered that I couldn't fix them without changing the API, colors, or behavior in incompatible ways. So I'm inclined to just go ahead and release 2.0 pretty much as it is, and merge those fixes for 2.1.

christophe-g commented 8 years ago

Shameless plug here - but started a new approach (based on Polymer, d3.v4 and Universe) available here for very early(!) preview.

The idea is to use the modular nature of Polymer to construct dc-like charts with web-components (i.e. handle building blocks directly in the markup - and importing only relevant components). There are two independent libraries: multi-verse for handling filtering/grouping of dataset, and multi-chart for rendering charts and handling selection (brush or click behaviors).

One simple example looks like:

<!-- Load the data -->
    <multi-csv url="flight.csv" data="{{data}}"></multi-csv>
 <!-- Start a multi-verse (similar to creating a new crossfilter)-->
  <multi-verse id="universe" data="[[data]]" universe="{{universe}}">
        <!-- Group the data by distances, count only)-->
    <multi-group universe="[[universe]]" data="{{data-chart-distance}}" group-by="distances">
            <!-- Render this group in a bar chart-->
      <multi-verse-bar title="distance" data="[[data-chart-distance]]"> </multi-verse-bar>
    </multi-group>
        <!-- Group the data by day-->
    <multi-group universe="[[universe]]" data="{{data-chart-day}}" group-by="day">
            <!-- Render this group in a pie chart-->
      <multi-verse-pie title="day (pie)" data="[[data-chart-day]]" color-scale="{{colorScale}}" width="{{width}}">
        <!-- Add a color scale legend to the chart -->
        <multi-legend legend chart-width="[[width]]" scale="[[colorScale]]" position="top-right"></multi-legend>
      </multi-verse-pie>
    </multi-group>
  </multi-verse>

and a more mature example: image

jakobzhao commented 7 years ago

Agree, not an easy task to V4. Do not know how to move the a.functor variable.

gordonwoodhull commented 7 years ago

Hi @jakobzhao, d3.functor isn't all that complicated and it's not hard to replace.

https://github.com/d3/d3/blob/master/CHANGES.md#internals

It will be a big task to convert dc.js to d3v4 but not that hard I think. Just will take a lot of time to sort out all the subtly changed functionality. I hope to get to it this year. If anyone makes partial progress on this please lmk.

bentwonk commented 7 years ago

Any update on progress supporting d3v4? Thanks.

gordonwoodhull commented 7 years ago

I'd really like to do this, and I've been learning all I can about the differences - just need to find 2-3 weeks of solid time to actually do the work. :-/

akuji1993 commented 7 years ago

Don't stress too much about it Gordon. Open source projects with only a few maintainers like this, it's hard to adopt stuff fast. If I'd have more time away from my actual job I'd be willing to help, but I won't promise stuff that I can't keep.

jakobzhao commented 6 years ago

agree, don't stress too much!

HamsterHuey commented 6 years ago

Is there any branch where anyone has taken any initial steps at implementing this? Just curious if there is some place I can muck around and potentially contribute. It would really be nice to switch to d3v4, but I understand how large a change that is for dc.js and it is pretty daunting to tackle it when you don't quite understand the inner workings of a good chunk of the library :-)

kum-deepak commented 6 years ago

Please check #1363 for current progress.

I have started work on this task. My current understanding:

My overall attach direction is as follows:

Well I took a plunge (#1363), following is current status:

As of now all test cases for pie chart are passing.

My next course of action:

gordonwoodhull commented 6 years ago

We are really close to a release. The final thing that needs to be ported is d3.layout.stack -> d3.stack, which has a completely different API.

I encourage people to try the 3.0 branch - once this goes beta we'll start publishing to NPM.

gordonwoodhull commented 6 years ago

All porting to d3 v4/5 is complete and merged to develop/master.

Published 3.0.0-beta.1 to npm.

Only remaining known issue caused by the port is that the range-series example crashes #1424. We need to deal with this in a comprehensive way which will change some assumptions (#1408)

gordonwoodhull commented 6 years ago

Working on a porting guide here: https://github.com/dc-js/dc.js/wiki/Changes-in-dc.js-version-3.0/_edit

Meanwhile, most of the changes are in Changelog.md.