search

erlang / otp

Erlang/OTP
http://erlang.org
Apache License 2.0
11.38k stars 2.95k forks source link

ERL-140: ERTS single pollset in a single thread is a bottleneck for any usage of file descriptors in Erlang #3453

Closed OTP-Maintainer closed 3 years ago

OTP-Maintainer commented 8 years ago

Original reporter: okeuday Affected version: OTP-18.3 Component: erts Migrated from: https://bugs.erlang.org/browse/ERL-140

I understand that OTP team members are aware of this problem, but it has not been addressed in the past and it appears to lack visibility, so it seems best to track here.  I see this problem as slightly less important than JIT (which is focused on Erlang-only source code) but similar due to its impact on Erlang port type handling efficiency (representing sockets/file_descriptors which often are connected to non-Erlang source code).

I am not sure about all the history surrounding this problem, but I do know that a patch was submitted in 2012 to solve the problem but was never merged (the patch is mentioned at http://erlang.org/pipermail/erlang-patches/2012-July/002860.html on the branch https://github.com/weicao/otp/tree/pollset_per_scheduler).  More recently, in 2016, a NIF was made public which attempts to solve the problem without contributing to Erlang/OTP (mentioned at http://erlang.org/pipermail/erlang-questions/2016-February/087775.html with the repo at https://github.com/max-feng/erlang_multi_pollset).  It is unclear what testing this particular approach with a NIF has received, which is an important aspect of this problem, because of how important the Erlang port type is for general use of Erlang (and due to the importance of fault-tolerance with the usage of Erlang).

I know that for benchmark games, this particular bottleneck can make things like cowboy (https://github.com/ninenines/cowboy) look bad (e.g., https://www.techempower.com/benchmarks/#section=data-r12&hw=peak&test=db).  By extension, that can make Erlang as a programming language look like a limitation on scalability, when considering anything connected by a file descriptor (sockets, files, Erlang port programs, etc.).

I have done testing recently on CloudI with the results at https://github.com/CloudI/CloudI/blob/develop/src/tests/request_rate/results/results_v1_5_1/results.txt#L45-L104 .  The results show that throughput is limited by 30k-55k req/s when utilizing file descriptors, but that throughput is able to rise without extreme resistance when using Erlang-only messaging (with rates exceeding 100k req/s for a period of time exceeding 2 minutes).
OTP-Maintainer commented 8 years ago

raimo said:

See also http://bugs.erlang.org/browse/ERL-154 since it may be related
OTP-Maintainer commented 8 years ago

okeuday said:

Just to be clear, http://bugs.erlang.org/browse/ERL-154 is good to use as a test once this issue (ERL-140) is fixed, since if the bug manifests in the same way it should lock up the machine (due to having all pollset threads consume 100% of their cpus, assuming a sufficient number of Erlang port objects being used).
OTP-Maintainer commented 7 years ago

okeuday said:

Lukas, based on https://twitter.com/garazdawi/status/855513428521345025 you have been adding this to 20.0.
OTP-Maintainer commented 7 years ago

okeuday said:

Recent update from mailing list:
[http://erlang.org/pipermail/erlang-questions/2017-June/092547.html]
OTP-Maintainer commented 7 years ago

rickard said:

Issue resolved in the {{master}} branch.