search

chrislusf / glow

Glow is an easy-to-use distributed computation system written in Go, similar to Hadoop Map Reduce, Spark, Flink, Storm, etc. I am also working on another similar pure Go system, https://github.com/chrislusf/gleam , which is more flexible and more performant.
3.2k stars 246 forks source link

any ideas to add Lua(LuaJIT)? #63

Closed plandem closed 7 years ago

plandem commented 8 years ago

it's quite fast (already using at high load systems).

It would nice to have: 1) define flow dynamically 2) use lua code as map/reduce/filter and etc.

chrislusf commented 8 years ago

Not familiar with Go + Luajit together. I want to see an example that go streams []bytes to luajit code and get output as []byte, probably via an external luajit process.

plandem commented 8 years ago

well...right now i'm not ready to answer your question, but maybe this projects helps: https://github.com/mozilla-services/heka

p.s.: what would be nice - to have ability to make custom DSL or just make some strict rules for built-in scripting language to defines flows without recompilation. e.g.: we need daemon that spawn calculations with flows. flows will be changing time from time and recompilation is not possible in our case. So we are just looking for solution. Your library is looking very powerful and yet easy to learn.

flowchartsman commented 8 years ago

Probably worth mentioning at this point that (unless I missed it on the dependency graph) LuaJIT would be the first cgo dependency glow has, thus complicating the build process and bringing all of the other fun nonsense that cgo comes with. If he's looking to keep the library pure-go, this isn't going to happen. There are pure-go Lua VMs, but none of them are on par with the C interpreters.

chrislusf commented 8 years ago

If going with LuaJIT, this would be a separate github project.

plandem commented 8 years ago

there is also another library about "flow", but mainly about flow programming: https://github.com/trustmaster/goflow

Well, so maybe i miss something, but is it possible to: dynamically "recreate" flow without recompilation? E.g.: we have UI to build flow for processing incoming data. As output of designed at UI flow we have something: DSL, json or whatever. After submit, "daemon" reload logic and recreate flow as designed. We can't have compiler to recompile each time. So any ideas?

plandem commented 8 years ago

@alaska the problem with pure Lua VM is in speed, i suppose. LuaJit is very fast. If we omit speed, then any other language can be taken, e.g. javascript. I started topic about LuaJit because it's already using at projects where speed is critical:

tarantool, nginx, snabb switch, luvit.io, terralang and etc.

So it's already battle tested in highload. But if we are talking about our case - we need good speed, but also we need dynamic behaviour - ability to reload flow after some actions at UI side. I like Glow for learning curve - very easy to set/use, but right now can't decide how it can help us :(

we need maybe only small subset of functionalities - something like mini DSL to add some kind of UI to build/use it. For more advanced cases - golang is still the right way.

I'm thinking about something like this: http://leafo.net/guides/dsl-in-lua.html

chrislusf commented 8 years ago

working on https://github.com/chrislusf/gleam

It's a Glow + Luajit combination. Currently only single computer only. Please try it out first.

plandem commented 8 years ago

Wow, will try it soon. Looks like totally separate project. Looking forward for benchmarks :) it would be nice to see cons/pros of each library :)

Sent from iPhone

On 12 Sep 2016, at 08:17, Chris Lu notifications@github.com wrote:

working on https://github.com/chrislusf/gleam

It's a Glow + Luajit combination. Currently only single computer only. Please try it out first.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

tejasmanohar commented 7 years ago

@chrislusf is gleam the future of this project, or are they both active in separate lines?

chrislusf commented 7 years ago

@tejasmanohar Glow is left as is. It is working fine by itself. But the performance is about 10x or 20x slower than Gleam.

tejasmanohar commented 7 years ago

@chrislusf sorry for issue hijacking :) - but more what i meant is... is there any reason you'd use glow over gleam once gleam becomes more mature / finishes up features on roadmap?

chrislusf commented 7 years ago

Glow has nice syntax and is more suited to some pure Go advocates' taste. That's it.

tejasmanohar commented 7 years ago

Gotcha, cool! Do Glow/Gleam optimize for in-memory computations or ...? Do you have any benchmarks on how this compares to something like Spark?

chrislusf commented 7 years ago

No Spark comparison. Its setup is too much.

Gleam supports both on disk and in memory operations.

On Sun, Jan 29, 2017 at 10:20 PM, Tejas Manohar notifications@github.com wrote:

Gotcha, cool! Do Glow/Gleam optimize for in-memory computations or ...? Do you have any benchmarks on how this compares to something like Spark?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/chrislusf/glow/issues/63#issuecomment-275987705, or mute the thread https://github.com/notifications/unsubscribe-auth/ABeL77cE7czC3n0pfY5gZ84zFPkkCMg_ks5rXYESgaJpZM4JjqGx .