Can we stop requestAnimationFrame since it started for 10s?

[gzboy007]

Hi Vasturiano, RaulPROP and friends,

How you are doing good and everthing goes smoothly. I used "3d-force-graph" recently to spot the network, it's really amazing. However I found a problem that can't be solved by a few days time and hope you can suggest the solution. Thanks in advance!

The backgroud is "3d-force-graph" continuous perform requestAnimationFrame(debug in Google Chrome) and never stop for large graph(>5000 nodes)... It largely consume browser's resources and it becomes slower and slower when we replace the input data and then generate the visualization again.

My question is that any approach that we can stop requestAnimationFrame since it started for 10s? After requestAnimationFrame stopped, we can still drag the node to a fixed position(can requestAnimationFrame again for a little bit when we drag the node)

I tried "d3AlphaMin", "cooldownTime", "pauseAnimation" however node of them works. "pauseAnimation" closed to the solution just it gives me a static picture and we can't zoom and span. In D3, I use forceSimulation.stop() to solved this problem, but I don't know how to achieve it in "3d-force-graph".

[vasturiano]

@gzboy007 thanks for reaching out.

You can control how long the force simulation runs for using any of the two attributes: cooldownTicks (default to Infinity) and cooldownTime (default to 15s). Whichever expires first will cause the force simulation to stop running. At that point, normally the only things that are kept running at every frame is the controls (zoom/pan), and the objects raycaster (for node/links click/hover/drag interaction). It can also happen that link particles need to be animated if you have that feature enabled.

Do you see the same pattern occurring with this example? https://vasturiano.github.io/3d-force-graph/example/large-graph/

If you're still having slow-down issues, could you make a reproducible example on https://codepen.io/ so we can try to debug and see where the bottleneck may be?

[gzboy007]

@vasturiano Hi Vasturiano, so sorry for my late reply. Kind of busy these two days. Just try the cooldownTime again, yes it can stop the force simulation.

Why it doesn't work when I tried last time is because I used d3.json to read the json file. Maybe the Synchronous machanism cause this. I used d3.json instaed of jsonUrl is because I need to work with Link and Node entities.

For example,
const Graph = ForceGraph3D()(elem) .jsonUrl('../datasets/blocks.json')

I can't say Graph.links or Graph.nodes.

Would you please suggest how I can access links or nodes using "jsonUrl"? Thank you so much~ Regards

[vasturiano]

@gzboy007 jsonUrl is just a thin wrapper around an http request. You should be able to do the same with d3-fetch or similar without any impact to the functionality. The force-engine lifecycle shouldn't be affected by it.

In any case, if you load data via jsonUrl you can always access the data object (after it's loaded) via the getter graphData:

const { nodes, links } = myGraph.graphData();

[gzboy007]

@vasturiano d3.json is one of the module in d3-fetch, however it's no luck cooldownTime doesn't work if I use d3.json. Would you please explain a little bit more about "after it's loaded"? For example below code will return empty nodes and links array. Thanks~

const elem = document.getElementById('3d-graph'); const Graph = ForceGraph3D()(elem).jsonUrl('../datasets/blocks.json') console.log(Graph.graphData(););

[vasturiano]

@gzboy007 it's very unexpected that d3-json would conflict with cooldownTime. Those are two very separate functionalities. After it's loaded, means the data fetch needs to finish and received by the browser browser before it's available via the module. Just for checking you can delay the console log statement:

setTimeout(() => console.log(Graph.graphData()), 10000);

[gzboy007]

@vasturiano Thank you so much for your answer, yes I can get the links and nodes by using setTimeout(). Today I found "cooldownTime" doesn't work either use jsonUrl() or d3.json. However I can't upload the data so I record a vedio and then change to gif maybe you can take a look. The app become slower and slower after several rounds of "requestAnimationFrame" when refreshing the graph with different data.

https://gfycat.com/oddballvaguearawana

[vasturiano]

@gzboy007 at this point it would be easier if you could make a reproducible simple example on https://codepen.io so we can debug it better.

[gzboy007]

@vasturiano OK. Let me create an example with dummy data, upload to https://codepen.io and then post it in here again.

[gzboy007]

@vasturiano Hi Vasturiano, I published the codes and files in here. Kindly take a look. This is my first time upload project to github, kindly let me know if you can't download it. Thanks. https://github.com/gzboy007/Network/tree/master

The problem is I used "cooldownTime" to set the graph cool down in 3s. Yes the graph has freezed in 3s however when we open the console in Chrome, it persist "requestAnimationFrame". It become very slow after we generate graph several times.

[vasturiano]

@gzboy007 thanks for making that example. I looked in your example data and there's more than 16000 links. This is likely what's causing the performance slow-down in your case. Do you need to render all those links?

[gzboy007]

@vasturiano Actually in our real case, we are not rendering 16000 links. We create a level 3 network with specific entity so all of the nodes are connected and without isolated nodes. I can't send out the data so I use Python to re-create the Json file to show the problem. Currently our tool mainly used D3.js as library, we use forceSimulation.stop() to freeze the graph and stop "requestAnimationFrame" so the graph can cool down in 10s. But in D3.js if we simulate more than 2000 nodes it become a little bit slow at the very beginning , but we can still use it because the graph will be totally freeze after 10s. Why we try to change to use your "3d-force-graph" library is because it's excellent 1. good looking 2. much faster if we simulate 2000 ~ 10000 nodes 3. Can have function like particles which is not available in D3.js 4.Code is much easier to maintenance.

We found that using "3d-force-graph" if the network have more than 2000 links which link to each other, it create a problem that it persist "requestAnimationFrame" at the backend so it become very very slow if we change the map for a few times. So it can't fit most of our case because 90% of our investigation target's network are more than 2000 links.

Vasturiano do you think you or yuor team can enhance the library that we can use "coodownTime" to truly stop the graph after 10s without "requestAnimationFram" at back end? I think this function is a little bit common for example in social network, we only need to get the roughly clustering and don't need to wait "Alpha" value to be "0".

Thanks & regards.

[vasturiano]

@gzboy007 thanks for your explanation.

There are a few principles that are important to understand: 1) There are two main sections of this module, one is the force-directed graph engine, the other is the rendering cycle. 2) The graph engine is just an in-memory structure with no connection to the browser DOM. This engine can be halted after the layout has cooled down using cooldownTime or cooldownTicks. This part appears appears to be working perfectly fine, and doesn't look like the issue on your side. 3) The rendering part is what controls the display, by manipulating the DOM using WebGL. This mechanism uses a continuous rendering cycle of requestAnimationFrame to maintain the visuals of your objects (nodes/links) on the scene. This requestAnimationFrame is standard operation when manipulating WebGL and is not considered an anomaly. You'd expect this cycle to keep on running even after your graph engine has stopped. The only way to halt this is to invoke .pauseAnimation() on the module. In that case all interaction and navigation controls is expected to stop working because they require the render cycle to properly function, but sometimes that's the intended goal - to get a static still picture. 4) The more objects you have on the scene, the more work the rendering cycle needs to do at each render frame. If you have a high volume of complex objects you'll start getting some frame slowdown as your browser will fail to maintain an optimal frame rate, you'll also notice the CPU spiking. This however also depends on the GPU capabilities of the machine's hardware where you're running the app. 5) Given that the performance issues in your case are likely related to the volume and complexity of the objects (I can't know for certain as the data differs from the example you posted on github), I would suggest to simplify your data set and potentially reduce your rendering to simpler shapes, until it fits your performance expectations. Do you do any custom representation of nodes or links? Do you use particles (and how many) in your links? Can you aggregate some of your nodes into some hierarchy so to reduce the number of items on the scene (see this example)? If you have a disproportionate high number of links compared to nodes (full mesh), can you trim some of those connections?

Hope this helps demistify to some extent the inner mechanics of the component and help with solving your performance experience.

[gzboy007]

@vasturiano Hi Vasturiano, Thank you so much for your patient explanation and valuable suggestion. You are so kind and impressed me a lot! Regarding to some suggestions you mention about, here are my answers.

Given that the performance issues in your case are likely related to the volume and complexity of the objects (I can't know for certain as the data differs from the example you posted on github), I would suggest to simplify your data set and potentially reduce your rendering to simpler shapes, until it fits your performance expectations. I didn't use complexity of the objects, all of the objects are the defualt one - sphere

Do you do any custom representation of nodes or links? No, they are the normal one. the nodes and links are the same as your "large-graph" examples.

Do you use particles (and how many) in your links? No this moment, but want to use in the future

Can you aggregate some of your nodes into some hierarchy so to reduce the number of items on the scene? Actually when our users open the graph, the first sight should be the network for the target entity. They won't click and expand the node one by one. So expanding nodes is not an option at the moment. Even that they are very cool and a very nice presentation for complicated network.

If you have a disproportionate high number of links compared to nodes (full mesh), can you trim some of those connections? Do you mean decrease the number of links between two nodes? Yes we have already did this. source + target is the unique key in link file. For the other fields in link, they are either concatenated or roll up. By this mean, we have already trim the connections.

Finally, I understood the in WebGL, continuous rendering cycle of requestAnimationFrame keep on running even after the graph engine has stopped. However, in the 'large-graph' example, requestAnimationFrame stopped and finalized after requestAnimationFrame around 20 times. In this case means requestAnimationFrame is not running all the time? Even I use this library to create some simple network, requestAnimationFrame will stop after a few seconds. The requestAnimationFrame runs all the time seems it's about the data structure. If nodes are connected with each other with many links, the requestAnimationFrame will run all the time. When we take a look the graph for several minutes, the graph is completed freezed however requestAnimationFrame is still running. The only one thing I can think about is requestAnimationFrame is keep rendering the object. But it seems like it's not this assumption because I use the most simple one like 'large-graph' example-sphere