basho / riak

Riak is a decentralized datastore from Basho Technologies.
http://docs.basho.com
Apache License 2.0
3.93k stars 534 forks source link

Questions #988

Closed runtman closed 4 years ago

runtman commented 4 years ago

Apologies, this isn't for questions but I'm not sure where else to direct them.

I was curious if the community editions support map reduce? Basho's documentation always stated they were going to ditch map reduce, now the company I work for rely on it heavily and are hitting performance issues which we can't seem to resolve with beefing up our infrastructure anymore, Riak 2.2.3 doesn't appear to make much use of anything over 32 cores.

martinsumner commented 4 years ago

All releases currently continue to support map/reduce. For release 3.0 (where we move to OTP 20), the javascript map/reduce has been taken out - but the Erlang map/reduce will continue.

If you are using the Javascript map/reduce, there are commercial Riak suppliers that would no doubt assist in converting to Erlang.

There is a medium term intention to look at things that were previously done by map/reduce and see if there is a more efficient way of supporting that feature more directly in Riak. For example there is a use case I know of that does Map/Reduce over objects which are all bloom filters. whereby it would be probably better if we could make these indexes, and allow bloom-checking functions to roll over them. Happy to consider other scenarios as well.

In terms of stressing > 32 cores, this may be tuneable. Heavy use of map/reduce is not something that has been discussed in any developer meetings - but if you have more information on your scenario it is something I would be happy to spend some time thinking about it.

runtman commented 4 years ago

Thanks for the response, so if for example. I take my 2.2.3 cluster and install community 2.9 over it. Map reduce will continue to work as expected.

martinsumner commented 4 years ago

Yes

runtman commented 4 years ago

I will try to install this over the top or 2.2.3 and see how i get on, thanks for the responses.