Just wanted to say that I enjoyed your talk and found it very interesting about your team's findings with the limits of ETS. I looked over the bench code and wanted to point out a possible issue that makes the bench a bit misleading:
For each process that is spawned, it performs the same amount of iterations. 10 processes with 1,000 iterations will be 10,000 counter updates. I was expecting that it would be 10 processes doing 100 counter updates each for a total of 1,000 counter updates. This means that some of the bigger tests with 1 million processes each doing a million updates will give a total of 1 trillion counter updates! This is several orders of magnitude different than the test where only 10 processes do 1 million updates (10 million total).
Also, a possible suggestion/scenario is to have your processes write to the same table but to different rows. With write concurrency set to true (don't set read concurrency to squeeze a bit more performance out of it), it shouldn't be locking the entire table if a process is only writing to a particular row. On my macbook pro, I was able to get around 20 million writes per second (8 core machine, 10 async threads, OTP 19 built from source w/hipe) with this method.
Anyways, just wanted to point this out. This is not to say that this is faster than the NIF but the report might be more accurate. I'll send a PR shortly.
slezakattack
commented
6 years ago
Hello,
Just wanted to say that I enjoyed your talk and found it very interesting about your team's findings with the limits of ETS. I looked over the bench code and wanted to point out a possible issue that makes the bench a bit misleading:
For each process that is spawned, it performs the same amount of iterations. 10 processes with 1,000 iterations will be 10,000 counter updates. I was expecting that it would be 10 processes doing 100 counter updates each for a total of 1,000 counter updates. This means that some of the bigger tests with 1 million processes each doing a million updates will give a total of 1 trillion counter updates! This is several orders of magnitude different than the test where only 10 processes do 1 million updates (10 million total).
Also, a possible suggestion/scenario is to have your processes write to the same table but to different rows. With write concurrency set to true (don't set read concurrency to squeeze a bit more performance out of it), it shouldn't be locking the entire table if a process is only writing to a particular row. On my macbook pro, I was able to get around 20 million writes per second (8 core machine, 10 async threads, OTP 19 built from source w/hipe) with this method.
Anyways, just wanted to point this out. This is not to say that this is faster than the NIF but the report might be more accurate. I'll send a PR shortly.
Regards, Michael