search

brianfrankcooper / YCSB

Yahoo! Cloud Serving Benchmark
Apache License 2.0
4.85k stars 2.22k forks source link

Benchmarking sharded cluster mongodb with YCSB #1673

Open aaa646123 opened 1 year ago

aaa646123 commented 1 year ago

Hi, I am new with mongodb and YCSB.

I have 2 type of mongo db:

  1. Standalone mongodb.
  2. Sharded cluster mongodb with 3 replica shards.

I assume that sharded cluster db would have better performance than single db. But after test with YCSB, I got the test result as below:

image

Seems like Shard cluster didn't have better performance than single mongodb. I don't know why it happened, did I miss something?

Below are my environment information, let me know if you want to know more information.

Environment info :

OS: Red Hat Enterprise Linux release 8.6
CPU: 2vCPU / Ram: 16G
Mongo db version: 4.2.24

The way I test single db

  1. Only enable 1 mongod service on 1 server.
  2. Run YCSB with url below on another windows client. 
    mongodb.url="mongodb://ycsb:ycsb_1234@192.168.0.1:20020/ycsb"

The way I test cluster db

  1. Enable 3 mongo shard, 1 config server, 1 mongos service on each 3 server.
  2. Run command as below to enable shard 
    sh.enableSharding("ycsb")
sh.shardCollection("ycsb.usertable",{_id:1})
  3. Run YCSB with url below on another windows client. 
    mongodb.url="mongodb://ycsb:ycsb_1234@[192.168.0.1:20020],[192.168.0.2:20020],[192.168.0.3:20020]/ycsb"
twblamer commented 1 year ago

Your database size is about 10 GB, which fits completely inside the RAM of one server. In that scenario, no performance benefit from sharding is possible. This is assuming you left fieldcount and fieldlength with default values.

You should experiment with larger record counts, like 40,000,000 for example.

You should also run the db.stats() command to examine the size of your database.

aaa646123 commented 1 year ago

@twblamer Sorry for the late reply. Here are the result of insert 40,000,000 records.

But single db still fater than sharded cluster...

image

Besides, my database is new and I will clean data before I run ycsb.

allanbank commented 1 year ago

With the sharding you used (sh.shardCollection("ycsb.usertable",{_id:1})) the database will be quite busy during and after the load phase with splitting the data into, what MongoDB calls, 'chunks' and migrating those across the different shards. I previously wrote a script to perform a 'pre-split' on the collection. You can find that in this comment.

I recommend doing the pre-split on a clean, unused cluster. Then load and run the benchmark.

Your virtual machines are probably a little undersized CPU wise (2 vCPUs) given you are throwing 500 threads of concurrency at the cluster probably from a Windows desktop with 16 or more real core/threads...

aaa646123 commented 1 year ago

@allanbank Thanks for the reply. I ran the ycsb-presplit-balance.js and here is the execute record.

MongoDB shell version v4.2.24
connecting to: mongodb://192.168.0.1:20020,192.168.0.2:20020,192.168.0.3:20020/admin?compressors=disabled&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("75fba283-b272-4db1-9367-3e8a56a08ec5") }
MongoDB server version: 4.2.24
Splitting ycsb.usertable across shards:

  shardrs1

  shardrs2

  shardrs3

Splitting at user00...
 moving to shardrs1...
 done.
Splitting at user01...
 moving to shardrs2...
 done.
Splitting at user02...
 moving to shardrs3...
 done.
Splitting at user03...
 moving to shardrs1...
 done.
Splitting at user04...
 moving to shardrs2...
 done.
Splitting at user05...
 moving to shardrs3...
 done.
Splitting at user06...
 moving to shardrs1...
 done.
Splitting at user07...
 moving to shardrs2...
 done.
Splitting at user08...
 moving to shardrs3...
 done.
Splitting at user09...
 moving to shardrs1...
 done.
Splitting at user10...
 moving to shardrs2...
 done.
Splitting at user11...
 moving to shardrs3...
 done.
Splitting at user12...
 moving to shardrs1...
 done.
Splitting at user13...
 moving to shardrs2...
 done.
Splitting at user14...
 moving to shardrs3...
 done.
Splitting at user15...
 moving to shardrs1...
 done.
Splitting at user16...
 moving to shardrs2...
 done.
Splitting at user17...
 moving to shardrs3...
 done.
Splitting at user18...
 moving to shardrs1...
 done.
Splitting at user19...
 moving to shardrs2...
 done.
Splitting at user20...
 moving to shardrs3...
 done.
Splitting at user21...
 moving to shardrs1...
 done.
Splitting at user22...
 moving to shardrs2...
 done.
Splitting at user23...
 moving to shardrs3...
 done.
Splitting at user24...
 moving to shardrs1...
 done.
Splitting at user25...
 moving to shardrs2...
 done.
Splitting at user26...
 moving to shardrs3...
 done.
Splitting at user27...
 moving to shardrs1...
 done.
Splitting at user28...
 moving to shardrs2...
 done.
Splitting at user29...
 moving to shardrs3...
 done.
Splitting at user30...
 moving to shardrs1...
 done.
Splitting at user31...
 moving to shardrs2...
 done.
Splitting at user32...
 moving to shardrs3...
 done.
Splitting at user33...
 moving to shardrs1...
 done.
Splitting at user34...
 moving to shardrs2...
 done.
Splitting at user35...
 moving to shardrs3...
 done.
Splitting at user36...
 moving to shardrs1...
 done.
Splitting at user37...
 moving to shardrs2...
 done.
Splitting at user38...
 moving to shardrs3...
 done.
Splitting at user39...
 moving to shardrs1...
 done.
Splitting at user40...
 moving to shardrs2...
 done.
Splitting at user41...
 moving to shardrs3...
 done.
Splitting at user42...
 moving to shardrs1...
 done.
Splitting at user43...
 moving to shardrs2...
 done.
Splitting at user44...
 moving to shardrs3...
 done.
Splitting at user45...
 moving to shardrs1...
 done.
Splitting at user46...
 moving to shardrs2...
 done.
Splitting at user47...
 moving to shardrs3...
 done.
Splitting at user48...
 moving to shardrs1...
 done.
Splitting at user49...
 moving to shardrs2...
 done.
Splitting at user50...
 moving to shardrs3...
 done.
Splitting at user51...
 moving to shardrs1...
 done.
Splitting at user52...
 moving to shardrs2...
 done.
Splitting at user53...
 moving to shardrs3...
 done.
Splitting at user54...
 moving to shardrs1...
 done.
Splitting at user55...
 moving to shardrs2...
 done.
Splitting at user56...
 moving to shardrs3...
 done.
Splitting at user57...
 moving to shardrs1...
 done.
Splitting at user58...
 moving to shardrs2...
 done.
Splitting at user59...
 moving to shardrs3...
 done.
Splitting at user60...
 moving to shardrs1...
 done.
Splitting at user61...
 moving to shardrs2...
 done.
Splitting at user62...
 moving to shardrs3...
 done.
Splitting at user63...
 moving to shardrs1...
 done.
Splitting at user64...
 moving to shardrs2...
 done.
Splitting at user65...
 moving to shardrs3...
 done.
Splitting at user66...
 moving to shardrs1...
 done.
Splitting at user67...
 moving to shardrs2...
 done.
Splitting at user68...
 moving to shardrs3...
 done.
Splitting at user69...
 moving to shardrs1...
 done.
Splitting at user70...
 moving to shardrs2...
 done.
Splitting at user71...
 moving to shardrs3...
 done.
Splitting at user72...
 moving to shardrs1...
 done.
Splitting at user73...
 moving to shardrs2...
 done.
Splitting at user74...
 moving to shardrs3...
 done.
Splitting at user75...
 moving to shardrs1...
 done.
Splitting at user76...
 moving to shardrs2...
 done.
Splitting at user77...
 moving to shardrs3...
 done.
Splitting at user78...
 moving to shardrs1...
 done.
Splitting at user79...
 moving to shardrs2...
 done.
Splitting at user80...
 moving to shardrs3...
 done.
Splitting at user81...
 moving to shardrs1...
 done.
Splitting at user82...
 moving to shardrs2...
 done.
Splitting at user83...
 moving to shardrs3...
 done.
Splitting at user84...
 moving to shardrs1...
 done.
Splitting at user85...
 moving to shardrs2...
 done.
Splitting at user86...
 moving to shardrs3...
 done.
Splitting at user87...
 moving to shardrs1...
 done.
Splitting at user88...
 moving to shardrs2...
 done.
Splitting at user89...
 moving to shardrs3...
 done.
Splitting at user90...
 moving to shardrs1...
 done.
Splitting at user91...
 moving to shardrs2...
 done.
Splitting at user92...
 moving to shardrs3...
 done.
Splitting at user93...
 moving to shardrs1...
 done.
Splitting at user94...
 moving to shardrs2...
 done.
Splitting at user95...
 moving to shardrs3...
 done.
Splitting at user96...
 moving to shardrs1...
 done.
Splitting at user97...
 moving to shardrs2...
 done.
Splitting at user98...
 moving to shardrs3...
 done.
Splitting at user99...
 moving to shardrs1...
 done.

I also ran the sh.status() in shell to check shard status. But I got result as below:

 databases:
        {  "_id" : "ycsb",  "primary" : "shardrs3",  "partitioned" : true,  "version" : {  
        "uuid" : UUID("6edbd870-e5e3-4650-8033-1271010aa332"),  "lastMod" : 1 } }
                ycsb.usertable
                        shard key: { "_id" : 1 }
                        unique: true
                        balancing: true
                        chunks:
                                shardrs3        1
                        { "_id" : { "$minKey" : 1 } } -->> { "_id" : { "$maxKey" : 1 } } on : shardrs3 Timestamp(1, 0)

Seems like usertable still not have chunk after ran the script. Did I miss something?

allanbank commented 1 year ago

That certainly does not look like the split worked. I think the bug is in the script that the lines:

sh.splitAt( fullName, { shardField : splitLocation } );
print(" moving to " + destination + "...");
sh.moveChunk( fullName, { shardField : splitLocation }, destination );

Should be:

sh.splitAt( fullName, { "_id" : splitLocation } );
print(" moving to " + destination + "...");
sh.moveChunk( fullName, { "_id" : splitLocation }, destination );

Can you edit the script and see if that does a better job?

aaa646123 commented 1 year ago

@allanbank Thanks for the reply. I edit and run the script. But it stock when it try to moving chunk to shardrs2. I've been wait for about 5 minute but it still stop at this process. I am not sure if it means have db lock or I should keep waiting... Here is the script I used. ycsb-presplit-balance.txt

Splitting ycsb.usertable across shards:

  shardrs1

  shardrs2

  shardrs3

Splitting at user00...
 moving to shardrs1...
 done.
Splitting at user01...
 moving to shardrs2...
 done.
Splitting at user02...
 moving to shardrs3...

And I collect some weird log as below, hope it could help. I found that these log keeps show up.

192.168.0.2 shardrs2.log
2023-04-26T11:11:03.522+0800 I  SHARDING [conn71] moveChunk data transfer progress: { waited: true, active: true, sessionId: "shardrs2_shardrs3_644892efa8b16c208151163f", ns: "ycsb.usertable", from: "shardrs2/192.168.0.1:20002,192.168.0.2:20002,192.168.0.3:20002", fromShardId: "shardrs2", min: { _id: "user02" }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, supportsCriticalSectionDuringCatchUp: true, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0, $gleStats: { lastOpTime: { ts: Timestamp(1682477807, 51), t: 3 }, electionId: ObjectId('7fffffff0000000000000003') }, lastCommittedOpTime: Timestamp(1682478659, 2), $configServerState: { opTime: { ts: Timestamp(1682478659, 1), t: 4 } }, $clusterTime: { clusterTime: Timestamp(1682478661, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, operationTime: Timestamp(1682478659, 2) } mem used: 0 documents remaining to clone: 0
2023-04-26T11:11:04.523+0800 I  SHARDING [conn71] moveChunk data transfer progress: { waited: true, active: true, sessionId: "shardrs2_shardrs3_644892efa8b16c208151163f", ns: "ycsb.usertable", from: "shardrs2/192.168.0.1:20002,192.168.0.2:20002,192.168.0.3:20002", fromShardId: "shardrs2", min: { _id: "user02" }, max: { _id: MaxKey }, shardKeyPattern: { _id: 1.0 }, supportsCriticalSectionDuringCatchUp: true, state: "ready", counts: { cloned: 0, clonedBytes: 0, catchup: 0, steady: 0 }, ok: 1.0, $gleStats: { lastOpTime: { ts: Timestamp(1682477807, 51), t: 3 }, electionId: ObjectId('7fffffff0000000000000003') }, lastCommittedOpTime: Timestamp(1682478659, 2), $configServerState: { opTime: { ts: Timestamp(1682478659, 1), t: 4 } }, $clusterTime: { clusterTime: Timestamp(1682478661, 1), signature: { hash: BinData(0, 0000000000000000000000000000000000000000), keyId: 0 } }, operationTime: Timestamp(1682478659, 2) } mem used: 0 documents remaining to clone: 0

192.168.0.3 shardrs3.log
2023-04-26T11:08:23.274+0800 I  COMMAND  [conn71] command admin.$cmd command: _recvChunkStatus { _recvChunkStatus: "ycsb.usertable", waitForSteadyOrDone: true, sessionId: "shardrs2_shardrs3_644892efa8b16c208151163f", $clusterTime: { clusterTime: Timestamp(1682478501, 2), signature: { hash: BinData(0, 61A4F2F46688AC5210EF8FB65A8E49CF9F685DE7), keyId: 7226173683998392352 } }, $configServerState: { opTime: { ts: Timestamp(1682478501, 1), t: 4 } }, $db: "admin" } numYields:0 reslen:755 locks:{} protocol:op_msg 1001ms
2023-04-26T11:08:24.276+0800 I  COMMAND  [conn71] command admin.$cmd command: _recvChunkStatus { _recvChunkStatus: "ycsb.usertable", waitForSteadyOrDone: true, sessionId: "shardrs2_shardrs3_644892efa8b16c208151163f", $clusterTime: { clusterTime: Timestamp(1682478501, 2), signature: { hash: BinData(0, 61A4F2F46688AC5210EF8FB65A8E49CF9F685DE7), keyId: 7226173683998392352 } }, $configServerState: { opTime: { ts: Timestamp(1682478501, 1), t: 4 } }, $db: "admin" } numYields:0 reslen:755 locks:{} protocol:op_msg 1001ms
aaa646123 commented 1 year ago

I tried upgrade my mongo db to 6.0.3 and redeploy sharded cluster db. When I run the script, it pop the error message as below.

Splitting ycsb.usertable across shards:

  shardrs1

  shardrs2

  shardrs3

Splitting at user00...
 moving to shardrs1...
 done.
Splitting at user01...
 moving to shardrs2...
 done.
Splitting at user02...
 moving to shardrs3...
MongoServerError: Data transfer error: ExceededTimeLimit: Failed to delete orphaned ycsb.usertable range [{ _id: "user02" }, { _id: MaxKey }) :: caused by :: operation exceeded time limit
allanbank commented 1 year ago

Do you have data in the database? If so the moves can take minutes to 10's of minutes to run. (The first move could move about 99% of the data, The second, 98%. etc. Data migration is intentionally not a speedy process.

If not then the issue might be that your config server cluster is not healthy so the move is not progressing.

If you want to leave the data in the script then do the splitAt(...) in one loop and then the moveChunk(...) in a separate loop after all of the splitAt(...) commands have completed.