Using d3-x3d within a d3.csv callback

dburkhardt commented 4 years ago

Please forgive me if this is a dumb question, but I'm trying to create a 3d scatterplot with d3-x3d.

This is my first d3 and first javascript project; I spend most of my time in the Python/Matplotlib ecosystem.

I can run the examples just fine, but when I try to load data using d3.csv and then create the chart inside the callback function (which seems to be the recommended way to work with externally loaded data), the chart doesn't appear.

My index.html looks like this:

<!DOCTYPE html>
<html>
<head>
  <meta charset="utf-8">
  <title>d3-x3d : 3D Scatter Plot Example</title>
  <script src="https://d3js.org/d3.v5.min.js"></script>
  <script src="https://x3dom.org/download/dev/x3dom-full.js"></script>
  <link rel="stylesheet" href="https://x3dom.org/download/dev/x3dom.css" />
  <script src="https://raw.githack.com/jamesleesaunders/d3-x3d/master/dist/d3-x3d.js"></script>
</head>

<body>
  <div id="chartholder"></div>

  <script>
    d3.csv("https://experimentaldesign.io/data/MNIST_PHATE_3D.head.csv")
        .then(function(data){
          myData = {
                key:"Bubbles",
                values: data
                 };

           // Select chartholder
           var chartHolder = d3.select("#chartholder");

           // Declare the chart component
           var myChart = d3.x3d.chart.scatterPlot();

           // Attach chart and data to the chartholder
           chartHolder.datum(myData).call(myChart);
        });

  </script>
</body>

I've also put this example into a JSFiddle if that's helpful.

https://jsfiddle.net/zg4x2ywo/3/

This seems like a great project, and I really appreciate the work you all put into it!

jamesleesaunders commented 4 years ago

Hi @dburkhardt Thanks for the feedback and interest in d3-x3d.

Coincidently I was working on a very similar thing tonight.

For some reason x3dom render times out if the loading of the csv takes a while. I am working with the x3dom developers on this presently. But, for the time being, there is a workaround, simply re-call x3dom.reload() at the end to re-trigger render.

  <script>
    d3.csv("https://experimentaldesign.io/data/MNIST_PHATE_3D.head.csv")
        .then(function(data){
          myData = {
                key:"Bubbles",
                values: data
                 };

           // Select chartholder
           var chartHolder = d3.select("#chartholder");

           // Declare the chart component
           var myChart = d3.x3d.chart.scatterPlot();

           // Attach chart and data to the chartholder
           chartHolder.datum(myData).call(myChart);
        })
        .finally(() => {
          x3dom.reload();
        });
  </script>

jamesleesaunders commented 4 years ago

I also notice your data has negative values in it. I am also working on a fix which adds support for negative axis to the Scatter and Bubble charts. You may be interested in a WIP branch I have which add a few new features to the scatter plot:

Bubble Color Scale
Bubble Size Scale
Negative Axis

Example in use: https://raw.githack.com/jamesleesaunders/d3-x3d/polys/examples/polys/3DScatterPlotExample_N3_3.html

Or see code: https://github.com/jamesleesaunders/d3-x3d/blob/polys/examples/polys/3DScatterPlotExample_N3_3.html

dburkhardt commented 4 years ago

Thanks @jamesleesaunders! Adding the x3dom.reload() call did fix it (along with a parsing fix).

Now I have a new issue. That csv file was a small test file with only 5 points, but the actual file I want to plot is 60K points (all of MNIST digits). Loading this data doesn't seem to scale well (i.e. the plot never loads). Is this expected? Is this a d3 csv loading issue or does the slowness come from x3dom/d3-x3d?

The full file is at https://experimentaldesign.io/data/MNIST_PHATE_3D.csv.

Any suggestions you have on the best way to tackle this problem is appreciated!

jamesleesaunders commented 4 years ago

Hi, So 60k point! Thats a lot! Again I am working with a prof Polys on a similar issue. To date I have not really tested d3-x3dom with large amount of points (but I am keen to make it work).

In this slight different example to the one previously: https://raw.githack.com/jamesleesaunders/d3-x3d/polys/examples/polys/3DScatterPlotExample_N3.html This also has lots of points, and although slow it does eventually render.

This will not be a d3.csv loading issue, this is likely more a x3dom rendering issue (60k x each sphere (bubble) is a lot of vertex!) I may experiment with a simpler shape for the point.

For now, you may want to try filtering for less points and ramping it up to see how many points work until it breaks:

// Just pick the first 200 points
myData = {
    key: "Bubbles",
    values: data.slice(0, 200)
};

It will no doubt get slow.

I will have a play with your data to see what I can do.

dburkhardt commented 4 years ago

Interesting, thanks! I can get it to work with 10K points. I wonder if it's possible to turn off shading for the spheres? I'm comparing to this: https://pair-code.github.io/understanding-umap/ and you can see in Figures 2 and 5 they have 3D scatter plots. You'll notice that the points in the plot are much simpler than what you're using in the bubble plots, but we'd rather more points and less shading for our application.

At the end of the day, we're interested in producing a lot of the same kinds of plots as in that example from Google PAIR, and our bar for what the plots should look like is about what they've produced.

That project uses a library Google built for 2D/3D point clouds (https://github.com/PAIR-code/scatter-gl) which looks like it's built off Three.JS. I think we'd rather stay in the D3 ecosystem for our project.

dburkhardt commented 4 years ago

Another question I've been having a hard time figuring out: does X3DOM natively use WebGL rendering? I imagine it must be, but I can't find any documentation on GPU acceleration.

jamesleesaunders commented 4 years ago

@dburkhardt Just to let you know I have not forgotten about you. I have managed to make a more efficient version of the bubble chart which I am working on on a separate branch.

As a sneak peek I have managed to an example going with your data: https://raw.githack.com/jamesleesaunders/d3-x3d/polys/examples/polys/dburkhardt.html

It still takes a little while to load and is still work in progress (and at present still limited to 10,000 points). But it does perform much better. I achieves this by swapping out individual x3d spheres for x3d particle shapes.

Code: https://github.com/jamesleesaunders/d3-x3d/blob/polys/examples/polys/dburkhardt.html

I will get back to you once I have polished it off and published it properly.

dburkhardt commented 4 years ago

@jamesleesaunders this looks excellent! I'm super excited to see this running. I greatly appreciate your help with this.

I wonder if there are some shading parameters that we could scale back on in order to speed up loading or increase the number of points? At the end of the day, I'm happy to have un-shaded circles instead of spheres like in this Figure from https://pair-code.github.io/understanding-umap/

Please let me know if there's anything that I can do to help out or contribute. I understand that my skillset is not well adapted for this project, but at the very least I'm happy to review documentation and test out the code.

jamesleesaunders commented 4 years ago

Hi @dburkhardt Just catching up on a few old issues and suggestions and wanted to follow up on our conversation above.

You may be please to know that x3dom has now added PointProperties support and as such I have now been able to complete the 'Particle Plot' component in d3-x3d, example here: https://raw.githack.com/jamesleesaunders/d3-x3d/master/examples/X3DOM/chart/ParticlePlot.html

I am not sure where you are now with your project but if you do have any questions let me know. I'd love to hear any feedback or thoughts on d3-x3d.

All the best. Jim

jamesleesaunders / d3-x3d

Using d3-x3d within a d3.csv callback #123