Clarification of gelly-stream features

vasia / gelly-streaming

An experimental Graph Streaming API for Apache Flink

Apache License 2.0

138 stars 44 forks source link

Clarification of gelly-stream features #40

Closed sdht0 closed 4 years ago

sdht0 commented 4 years ago

Hi there.

I read in the presentation on gelly-stream (https://www.slideshare.net/vkalavri/gellystream-singlepass-graph-streaming-analytics-with-apache-flink) that it only supports single-pass graph computations, and moreover with only approximate results. Is that (still) correct?

I'm just getting started with Flink and did not find a way yet to get gelly-streaming running. Do you have step-by-step instructions somewhere that can help me get say ConnectedComponents running?

vasia commented 4 years ago

Hi @sdht0,

The README more or less reflects the current state of the repository. You can find the examples in the example folder and they should all have documentation. The Connected Components example for instance returns exact results. To run the examples, you can either import the project to your IDE (I use IntelliJ) and run them as normal Java applications or build the project with Maven and use the generated jar.

I hope this helps! Let me know if you have further questions.

sdht0 commented 4 years ago

Hi again! After a haitus, I was able to try out gelly-streaming again and actually got the connected component examples to work (on vscode but not on IntelliJ). I had a few question I want to clarify:

Is my understanding correct: ConnectedComponentsExample merges the incoming edges into time batches and performs the entire computation for each batch, while IterativeConnectedComponents applies each edge as soon as it comes it to the single maintained output session. Is ConnectedComponentsExample reusing the results of the previous batches?
It looks like a bug in IterativeConnectedComponents, if I input 0 1\n1 2\n1 3 (using nc), it endlessly outputs 4> (3,0) until I input another edge.
How can I scale the execution of the example to multiple threads/workers or run it within a existing flink deployment?

Thanks!

sdht0 commented 4 years ago

Hi @vasia, pinging again in case you missed the above message.

vasia commented 4 years ago

Hi @sdht0, Thank you for your patience. I've been busy with a deadline.

Your understanding about the connected components algorithms is correct. Both incrementally compute the connected components of an edge-only stream and "remember" the entire graph history.
That was a bug indeed. Thank you for reporting. I think I have pushed a fix. Let me know if it works for you.
Gelly-streaming does not have its own runtime. It is simply a library that you can run on top of Flink, i.e. you can define parallelism and run programs in parallel as you would normally do in Flink.

sdht0 commented 4 years ago

Thanks! I had a couple of more questions, but I'll try the new code again and get back to you.