search

benmfaul / XRTB

A Real Time Bidding (RTB) 2.1 bidding engine written in Java 1.8
http://rtb4free.com
Apache License 2.0
121 stars 67 forks source link

QPS testing #7

Closed lionux2 closed 7 years ago

lionux2 commented 7 years ago

Hi Ben;

First Happy New Year 2017!

Also, I have built a local server (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, 32 cores, with 128GB of ram), and ran maxload on it.

Based on your own testing (i.e. 50k QPS), I expected to reach a phenomenal number of QPS myself on this server; although I only have the following (with an exception ;((.

Threads=32, QPS=16599.5 com.aerospike.client.AerospikeException: Error Code 14: Hot key at com.aerospike.client.command.WriteCommand.parseResult(WriteCommand.java:72) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:56) at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:299) at com.aerospike.redisson.RedissonClient.hmset(RedissonClient.java:294) at com.xrtb.bidder.Controller.recordBid(Controller.java:789) at com.xrtb.bidder.Handler.handle(RTBServer.java:881) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)

Can you give your feedback on this, and potential issues why I have such a slow performance (I would think that the server is resourced enough to reach much higher QPS)?

Thanks.

benmfaul commented 7 years ago

Your super-charged RTB4FREE system has overloaded the aerospike server. Too many updates are hitting the same keys.

All of these caching systems are a pain in the neck in this regard. Trust me. REDIS is even worse. Network latencies to the cache will always slow you down.

For maximum performance use the already embedded cache2k system instead of aerospike.

It is very easy to implement.

  1. Make sure database.json (the campaigns file) is in the same directory you run RTB4FREE from.

  2. In the config file Campaigns/Payday.json, simply remove the "aerospike" object.

  3. Your RTB4FREE JVM memory resources will need to be increased however. Use at least -Xmx8g.

More information can be found here:

http://www.rtb4free.com/noaerospike.html

On Wed, Jan 4, 2017 at 9:20 AM, Lionux notifications@github.com wrote:

Hi Ben;

First Happy New Year 2017!

Also, I have built a local server (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, 32 cores, with 128GB of ram), and ran maxload on it.

Based on your own testing (i.e. 50k QPS), I expected to reach a phenomenal number of QPS myself on this server; although I only have the following (with an exception ;((.

Threads=32, QPS=16599.5 com.aerospike.client.AerospikeException: Error Code 14: Hot key at com.aerospike.client.command.WriteCommand.parseResult( WriteCommand.java:72) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:56) at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:299) at com.aerospike.redisson.RedissonClient.hmset(RedissonClient.java:294) at com.xrtb.bidder.Controller.recordBid(Controller.java:789) at com.xrtb.bidder.Handler.handle(RTBServer.java:881) at org.eclipse.jetty.server.session.SessionHandler. doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.session.SessionHandler. doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ScopedHandler.handle( ScopedHandler.java:141)

Can you give your feedback on this, and potential issues why I have such a slow performance (I would think that the server is resourced enough to reach much higher QPS)?

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7, or mute the thread https://github.com/notifications/unsubscribe-auth/ACoB4kfq4dz5hsSFuRHS1TO-mbV5wArdks5rO9TVgaJpZM4La25M .

lionux2 commented 7 years ago

Thanks you. I will try and let you know the QPS I could get out of my system.

On 1/4/2017 6:40 PM, benmfaul wrote:

Your super-charged RTB4FREE system has overloaded the aerospike server. Too many updates are hitting the same keys.

All of these caching systems are a pain in the neck in this regard. Trust me. REDIS is even worse. Network latencies to the cache will always slow you down.

For maximum performance use the already embedded cache2k system instead of aerospike.

It is very easy to implement.

  1. Make sure database.json (the campaigns file) is in the same directory you run RTB4FREE from.

  2. In the config file Campaigns/Payday.json, simply remove the "aerospike" object.

  3. Your RTB4FREE JVM memory resources will need to be increased however. Use at least -Xmx8g.

More information can be found here:

http://www.rtb4free.com/noaerospike.html

On Wed, Jan 4, 2017 at 9:20 AM, Lionux notifications@github.com wrote:

Hi Ben;

First Happy New Year 2017!

Also, I have built a local server (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, 32 cores, with 128GB of ram), and ran maxload on it.

Based on your own testing (i.e. 50k QPS), I expected to reach a phenomenal number of QPS myself on this server; although I only have the following (with an exception ;((.

Threads=32, QPS=16599.5 com.aerospike.client.AerospikeException: Error Code 14: Hot key at com.aerospike.client.command.WriteCommand.parseResult( WriteCommand.java:72) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:56) at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:299) at com.aerospike.redisson.RedissonClient.hmset(RedissonClient.java:294) at com.xrtb.bidder.Controller.recordBid(Controller.java:789) at com.xrtb.bidder.Handler.handle(RTBServer.java:881) at org.eclipse.jetty.server.session.SessionHandler. doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.session.SessionHandler. doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ScopedHandler.handle( ScopedHandler.java:141)

Can you give your feedback on this, and potential issues why I have such a slow performance (I would think that the server is resourced enough to reach much higher QPS)?

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7, or mute the thread

https://github.com/notifications/unsubscribe-auth/ACoB4kfq4dz5hsSFuRHS1TO-mbV5wArdks5rO9TVgaJpZM4La25M .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7#issuecomment-270434642, or mute the thread https://github.com/notifications/unsubscribe-auth/AFW3viWkTTGdvkpp-XzdtbgB3dUIk3Baks5rO9mFgaJpZM4La25M.

lionux2 commented 7 years ago

Hi Ben;

I indeed try to run ... and this time no exception from aerospike.

However, the QPS did not improve although I added 16gb of memory instead of 8gb that you suggested.

The cpu managed to settle at 70% then. I have noticed that the memory is not really an issue for the RTB.

So, I guess my server is somehow limited ... 17K QPS for me is the max I can get from this hardware. Still not bad for two old generation CPUs.

Thanks,

On 1/6/2017 1:25 PM, Patrick wrote:

Thanks you. I will try and let you know the QPS I could get out of my system.

On 1/4/2017 6:40 PM, benmfaul wrote:

Your super-charged RTB4FREE system has overloaded the aerospike server. Too many updates are hitting the same keys.

All of these caching systems are a pain in the neck in this regard. Trust me. REDIS is even worse. Network latencies to the cache will always slow you down.

For maximum performance use the already embedded cache2k system instead of aerospike.

It is very easy to implement.

  1. Make sure database.json (the campaigns file) is in the same directory you run RTB4FREE from.

  2. In the config file Campaigns/Payday.json, simply remove the "aerospike" object.

  3. Your RTB4FREE JVM memory resources will need to be increased however. Use at least -Xmx8g.

More information can be found here:

http://www.rtb4free.com/noaerospike.html

On Wed, Jan 4, 2017 at 9:20 AM, Lionux notifications@github.com wrote:

Hi Ben;

First Happy New Year 2017!

Also, I have built a local server (Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz, 32 cores, with 128GB of ram), and ran maxload on it.

Based on your own testing (i.e. 50k QPS), I expected to reach a phenomenal number of QPS myself on this server; although I only have the following (with an exception ;((.

Threads=32, QPS=16599.5 com.aerospike.client.AerospikeException: Error Code 14: Hot key at com.aerospike.client.command.WriteCommand.parseResult( WriteCommand.java:72) at com.aerospike.client.command.SyncCommand.execute(SyncCommand.java:56) at com.aerospike.client.AerospikeClient.put(AerospikeClient.java:299) at com.aerospike.redisson.RedissonClient.hmset(RedissonClient.java:294) at com.xrtb.bidder.Controller.recordBid(Controller.java:789) at com.xrtb.bidder.Handler.handle(RTBServer.java:881) at org.eclipse.jetty.server.session.SessionHandler. doHandle(SessionHandler.java:226) at org.eclipse.jetty.server.session.SessionHandler. doScope(SessionHandler.java:189) at org.eclipse.jetty.server.handler.ScopedHandler.handle( ScopedHandler.java:141)

Can you give your feedback on this, and potential issues why I have such a slow performance (I would think that the server is resourced enough to reach much higher QPS)?

Thanks.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7, or mute the thread

https://github.com/notifications/unsubscribe-auth/ACoB4kfq4dz5hsSFuRHS1TO-mbV5wArdks5rO9TVgaJpZM4La25M .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7#issuecomment-270434642, or mute the thread https://github.com/notifications/unsubscribe-auth/AFW3viWkTTGdvkpp-XzdtbgB3dUIk3Baks5rO9mFgaJpZM4La25M.

venediktov commented 7 years ago

Hi Guys,

I don't have such powerful hardware as you guys , can someone test my C++11 stack for QPS ?

https://github.com/venediktov/vanilla-rtb

you possibly need boost 1.61 and Linux , most Ubuntu boxes have Boost pre-installed , if not simple sudo add-apt-repository ppa:boost-latest/ppa sudo apt-get update sudo apt-get install libboost1.61-all-dev

then read instructions on my home page, trivial 3 line commands. I currently have simepl http_bidder_test in my examples dir , that supports most time consuming targeting ( others will be added later) . I wonder what it will show on your 32 core machine , help academia our project is just a sandbox for students. On my 4 core ubuntu 1.7Ghz I am getting 1124 QPS with apache benchmark tool , see my ab_bidder.sh ( needs to be changed to concurrency 32 , -c 32 option) for your system.

Thanks, Vlad.

lionux2 commented 7 years ago

Hi Vlad.

My machine seems impressive, although I have not seen impressive QPS on it so far.

Still I am happy to try your software (I am very curious about it ;)).

Also, from a high level point of view; what makes vanilla-rtb different from RTBKit?

Boost seems to enable fast development, and performance might not be its strong point.

Thanks,

P.

On 2/24/2017 4:51 AM, Vladimir Venediktov wrote:

Hi Guys,

I don't have such powerful hardware as you guys , can someone test my C++11 stack for QPS ?

https://github.com/venediktov/vanilla-rtb

you possibly need boost 1.61 and Linux , most Ubuntu boxes have Boost pre-installed , if not simple sudo add-apt-repository ppa:boost-latest/ppa sudo apt-get update sudo apt-get install libboost1.61-all-dev

then read instructions on my home page, trivial 3 line commands. I currently have simepl http_bidder_test in my examples dir , that supports most time consuming targeting ( others will be added later) . I wonder what it will show on your 32 core machine , help academia our project is just a sandbox for students. On my 4 core ubuntu 1.7Ghz I am getting 1124 QPS with apache benchmark tool , see my ab_bidder.sh ( needs to be changed to concurrency 32 , -c 32 option) for your system.

Thanks, Vlad.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/benmfaul/XRTB/issues/7#issuecomment-282196873, or mute the thread https://github.com/notifications/unsubscribe-auth/AFW3vncGkIwRpREVJWUq7jsBbAKgFjNRks5rflO6gaJpZM4La25M.

venediktov commented 7 years ago

Hi Lionux, Thanks for willing to take a look at our yet in progress framework. We are different from RTBkit in respect we started project early on with few lines of code , we want RTB community to contribute to ideas. RTBkit is very difficult even to build , not sure how people use it. It's like application with json configuration written in-house by 5-10 developers then released after 3 years to github. I don't think Boost is as slow as it's claimed . For example Boost.IPC measured only 20-30 microseconds to do filtering in shared memory. Boost.Serialization in binray mode is compatible with google protocol buffers. We constantly intend to run performance tests including use of our perf_timer class which we use to identify slow parts of the code. Our code is not replacement for RTBkit , but we intend to add functionality RTBkit does not have and simplify interfaces for loading data into Bidders. Currently we think our model with single bidder used with 1000 different geo campaigns ( see geo_add 98000 records in shared memory) can sustain the load on a single machine , however we work on scale it out with addition to communicator pattern ( still in progress ) Let me know if you have problems loading our stack. P.S. I am working on setting up Travis CI ( Continuous Integration ) , so we'll know if it builds for different flavors of Boost,gcc,clang,etc. Cheers, Vlad.