Large Graph Processing - Githubissues

arangoml / networkx-adapter

The NetworkX adapter for ArangoDB

Apache License 2.0

23 stars 8 forks source link

Large Graph Processing #87

Open cw00dw0rd opened 2 years ago

cw00dw0rd commented 2 years ago

Currently, we do not have a way to limit graph sizes. For example, if we have a large graph stored in ArangoDB that may take up 1TB if loaded into memory. It would be nice to explore limiting the extracted graph from ArangoDB via hard limits or sampling.

Additionally, this relates to optimizing vertex/edge dictionaries currently used to create the NetworkX graph.

aMahanna commented 2 years ago

Some things to consider:

Large Graph sampling
Parallel Batch Processing via multithreading in python
Providing arangoexport and arangoimport as an option for ArangoDB to NetworkX & vice-versa
GPU-enabled adapter operations

aMahanna commented 2 years ago

(low priority for now)

nealrauhauser commented 2 months ago

This really should be expanded so that there is a bridge to Graphistry - it's a fairly slick GPU enabled web based network visualization tool - similar in spirit to Gephi, but much more capable in terms of volume. They have a free plan that you can use for an hour, then you have to reset the API key. It's not much of a burden, an hour is a LONG time to be messing with any given graph visualization.