Eleveldb bug with non-default Erlang schedulers on Linux

basho / eleveldb

Erlang LevelDB API

263 stars 176 forks source link

Eleveldb bug with non-default Erlang schedulers on Linux #260

Open maartenvanmalland opened 4 years ago

maartenvanmalland commented 4 years ago

It seems that there's a pretty hefty bug with Eleveldb when you're using non-default Erlang schedulers ( https://erlang.org/doc/man/erl.html#+sbt ). When you use one of those scheduler flags to start your Erlang application, all the Eleveldb threads are being scheduled to just one core! In our case, this completely trashed the performance of our application.

You can easily check this behavior yourself:

Run your Erlang application with another scheduler (i.e. erl +sbt nnts $application)
Check the affinity of the Eleveldb processes in /proc/sched_debug . You'll see that all the Eleveldb threads are bound to the same core.

hb0nes commented 1 year ago

Is anyone looking at this?

Our company is unable to use isolcpu due to Eleveldb not honoring the Erlang scheduler settings.

If it's not related to Eleveldb, but instead a configuration error on our side, that is also an answer. However, it does not appear that this is the case.

martinsumner commented 1 year ago

I made a comment on a leveldb issue about supporting eleveldb going forward - https://github.com/basho/leveldb/issues/240#issuecomment-1341310851.

Those with commit rights to this repo are focused now on making the minimal changes necessary to maintain eleveldb within Riak. So I'm not aware of anyone actively working on eleveldb/basho-leveldb who would be able to resolve an issue like this.

hb0nes commented 1 year ago

Thanks for the reply, Martin. A shame there aren't enough contributors!

I see you've mentioned leveled or bitcask there, we will look into this.

martinsumner commented 1 year ago

Be aware, that Leveled is heavily optimised for the Riak use case; and bitcask doesn't support indexing. It may also be worth looking at https://gitlab.com/barrel-db/erlang-rocksdb.