search

grosser / parallel

Ruby: parallel processing made simple and fast
MIT License
4.17k stars 254 forks source link

Use Ox as serializer #194

Open najamelan opened 7 years ago

najamelan commented 7 years ago

Ox is an object serializer which is faster than Marshal. I wonder if you'd be interested in a pull request that uses Ox in parallel?

Question is whether it's ok to add a dependency which requires compilation or rather use Ox if available and Marshal otherwise?

https://github.com/ohler55/ox http://www.ohler.com/dev/ruby_object_xml_serialization/ruby_object_xml_serialization.html

grosser commented 7 years ago

How about allowing the serializer to be configured ? :)

On Wed, Jan 25, 2017 at 5:19 AM, Naja Melan notifications@github.com wrote:

Ox is an object serializer which is faster than Marshal. I wonder if you'd be interested in a pull request that uses Ox in parallel?

Question is whether it's ok to add a dependency which requires compilation or rather use Ox if available and Marshal otherwise?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/194, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAsZ7zjA9REn5gqzMaC7DFGbv0GcTlNks5rV0v3gaJpZM4LteaO .

ms-ati commented 7 years ago

+1 for allowing the serializer to be configured, and giving an example in the README of using Ox!

grosser commented 7 years ago

how about a PR ? :)

On Fri, Feb 10, 2017 at 7:41 AM, Marc Siegel notifications@github.com wrote:

+1 for allowing the serializer to be configured, and giving an example in the README of using Ox!

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/grosser/parallel/issues/194#issuecomment-278977992, or mute the thread https://github.com/notifications/unsubscribe-auth/AAAsZ7iTBOQlI0jykL8SIUPWIvh6uUmtks5rbIUQgaJpZM4LteaO .

najamelan commented 7 years ago

I'm sorry I didn't get back to you about this. I was looking into dealing with performance issues by parallelizing, and when considering Parallel this came to mind. But then I realized I had quite big data structures that would need to be available to my working processes and that seemed a hassle, passing through redis which needs two more serialize operations to get data around. Then I though, damn, I would really be better of doing real multithreading, so I looked into jruby, which didn't really run refinements the way I need them to, rubinius didn't work either for some reason I already forgot, and so now I'm extending my app with a lib in rust that I'm working around the clock to implement. Faster work in the first place and I can multithread it on top of that without serializing. That's pretty much better than anything I can do with ruby directly.

So, since I'm working against a deadline, I won't be able to work on this proposal this week. In itself it looks like it's trivial to implement, so I wouldn't mind but I can't make promises right now. I will try to find some time next week to look into it.