search

seomoz / pyreBloom

Fast Redis Bloom Filters in Python
MIT License
289 stars 67 forks source link

performance problem with remote machine in same network #20

Open Cphilo opened 8 years ago

Cphilo commented 8 years ago

I use the benchmark.py to test the performance. My local machine ip is 192.168.0.201 and I have another redis-server running on 192.168.0.101. Here is the parameter with the benchmark.py file.

count    = 50000
capacity = 1e9
error    = 1e-6

Then the benchmark result with 192.168.0.201 is as follows:

Generating 100000 random test words
Generated random test words in 1.082357s
Filter using 20 hash functions and 28755175137 bits
Batch insert : 11.030547s (4532.866814 words / second)
Serial insert: 14.348155s (3484.768594 words / second)
Batch test   : 2.664293s (18766.704355 words / second)
Serial test  : 5.634645s (8873.673520 words / second)
False positive rate: 0.000000 (0.000001 expected)
Redis set add  : 0.119999s (416670.375433 words / second)
Redis pipe chk : 0.884489s (56529.826959 words / second)
Redis pipe sadd: 0.888721s (56260.626908 words / second)
Redis pipe chk : 0.884423s (56534.032935 words / second)

And the benchmark with 192.168.0.101 is as follows:

Generating 100000 random test words
Generated random test words in 1.091547s
Filter using 20 hash functions and 28755175137 bits
Batch insert : 14.148405s (3533.967238 words / second)
Serial insert: 30.745577s (1626.250183 words / second)
Batch test   : 6.328274s (7901.048518 words / second)
Serial test  : 20.952494s (2386.350744 words / second)
False positive rate: 0.000000 (0.000001 expected)
Redis set add  : 0.140992s (354630.235574 words / second)
Redis pipe chk : 0.909973s (54946.676504 words / second)
Redis pipe sadd: 0.906465s (55159.324459 words / second)
Redis pipe chk : 0.887314s (56349.832669 words / second)

It is obvious that the performance result with 192.168.0.101 is much slower than 192.168.0.201. It seems to be a bug. Any ideas?

b4hand commented 8 years ago

I seriously doubt this is an issue with pyreBloom. It's much more likely an issue with your system or network configuration on the two different machines.

Cphilo commented 8 years ago

@b4hand , please give some test data before doubting it.

b4hand commented 8 years ago

Please give some hardware and OS details, as well as the version of Redis you are using. Are there any other processes running at the same time?

Do you have any firewall rules setup on either of the machines? How are the two machines connected and what network devices are in between the two of them?

The information you've given is simply not enough information to reproduce a test.

Cphilo commented 8 years ago

@b4hand , add some details about my environment. 192.168.0.101 and 192.168.0.201 are on same network with 1Gb network card redis version: Redis server v=3.0.3 redis-cli 3.0.3 os detail: Linux node1 4.2.0-16-generic #19-Ubuntu SMP Thu Oct 8 15:35:06 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

b4hand commented 8 years ago

This still doesn't tell me anything about the hardware of the two machines. For example, if you run the client and server on a single CPU machine, you would expect the performance to be worse than if you run the client and server on separate machines.