YCSB and MongoDB shards

[annisaMS]

My name is Annisa, I'm new with MongoDB and ycsb I have some questions about MongoDB benchmarking using YCSB :

1. I have 6 shards, but YCSB only work for 3 shards, is YCSB has limitation in working with MongoDB?
2. can I use spesific data file in YCSB?

may you please help me to solve my problems?

thanks

[allanbank]

What do you mean that "YCSB only work for 3 shards"? What exactly are you seeing that makes you say that?

You will want to to pre-split and balance the collection (for best results) before starting the ycsb load.

Rob

[annisaMS]

thaks for the reply @allanbank

I have MongoDB sharded cluster that consist of 6 shards and I use basic configuration of YCSB for benchmarking my sharded cluster, and I already enabled my ycsb database for all shards. It just happened like that. I don't know why, can you please tell me if I need to do somtehing or if I've missed a step?

I'll try to find about pre-split and ballance the collection later

In previous comment, I read that "YCSB is not designed to test against your specific data. It is only designed to load test and perform some certain types of predefined data. The data that is loaded can be modified to change a few properties like number of records, data size, and some other parameters" what's the right way to change the data size in YCSB?

[allanbank]

YCSB throws a lot of data into MongoDB in a very short amount of time. MongoDB internally tracks the mapping of what documents are on what shards via ranges over the shard key. Initially, there is a single range entry (-infinity, infinity) [called a chunk] on a single shard. Over time MongoDB looks at the data and will split the range/chink to allow documents to move to other shards. It is this second step (actually moving chunks of documents to balance the cluster) that takes time as all of the documents have to be moved to the new shard. A normal YCSB run is not long enough to allow all of the chunks to get spread across the cluster effectively.

The pre-split and balance manually creates the ranges in the shard key range and then manually moves the chunks onto different shards.

I currently have limited internet and have not found a good guide on pre-splitting I can point you to. Once I get back to civilization I will put something together for you.

Rob

[annisaMS]

@allanbank oh I see ,thanks for your explaination , wah I'll wait for that

[allanbank]

I wrote a quick script that should do the pre-split and balance for you. I added comments so you can modify it to your needs. My version drops the ycsb database which (obviously) will delete data but makes the pre-split and balance faster.

To run the script save it with the name ycsb-presplit-balance.js (the '.js' extension is required) and use the mongo command something like:

mongo <mongos_host>:<mongos:port>/admin <path_to>/ycsb-presplit-balance.js 

I have not actually run this against a MongoDB sharded cluster yet. If you run into problems let me know.

Rob.

ycsb-presplit-balance.txt

[annisaMS]

Thanks .. I do it in my VM, because my real sharded cluster is in my campus laboratorium, I run it without change anything, and it goes like: mongo mongos.com:27017/admin /home/mongo/Downloads/ycsb-presplit-balance.js MongoDB shell version: 3.2.17 connecting to: mongos.com:27017/admin Waiting for active hosts... Waiting for the balancer lock... Waiting again for active hosts after balancer is off... Splitting ycsb.usertable across shards:

rs1

rs0

Splitting at user00... moving to rs1... done. Splitting at user01... moving to rs0... done. Splitting at user02... moving to rs1... done. Splitting at user03... moving to rs0... done. Splitting at user04... moving to rs1... done. Splitting at user05... moving to rs0... done. Splitting at user06... moving to rs1... done. Splitting at user07... moving to rs0... done. Splitting at user08... moving to rs1... done. Splitting at user09... moving to rs0... done. Splitting at user10... moving to rs1... done. Splitting at user11... moving to rs0... done. Splitting at user12... moving to rs1... done. Splitting at user13... moving to rs0... done. Splitting at user14... moving to rs1... done. Splitting at user15... moving to rs0... done. Splitting at user16... moving to rs1... done. Splitting at user17... moving to rs0... done. Splitting at user18... moving to rs1... done. Splitting at user19... moving to rs0... done. Splitting at user20... moving to rs1... done. Splitting at user21... moving to rs0... done. Splitting at user22... moving to rs1... done. Splitting at user23... moving to rs0... done. Splitting at user24... moving to rs1... done. Splitting at user25... moving to rs0... done. Splitting at user26... moving to rs1... done. Splitting at user27... moving to rs0... done. Splitting at user28... moving to rs1... done. Splitting at user29... moving to rs0... done. Splitting at user30... moving to rs1... done. Splitting at user31... moving to rs0... done. Splitting at user32... moving to rs1... done. Splitting at user33... moving to rs0... done. Splitting at user34... moving to rs1... done. Splitting at user35... moving to rs0... done. Splitting at user36... moving to rs1... done. Splitting at user37... moving to rs0... done. Splitting at user38... moving to rs1... done. Splitting at user39... moving to rs0... done. Splitting at user40... moving to rs1... done. Splitting at user41... moving to rs0... done. Splitting at user42... moving to rs1... done. Splitting at user43... moving to rs0... done. Splitting at user44... moving to rs1... done. Splitting at user45... moving to rs0... done. Splitting at user46... moving to rs1... done. Splitting at user47... moving to rs0... done. Splitting at user48... moving to rs1... done. Splitting at user49... moving to rs0... done. Splitting at user50... moving to rs1... done. Splitting at user51... moving to rs0... done. Splitting at user52... moving to rs1... done. Splitting at user53... moving to rs0... done. Splitting at user54... moving to rs1... done. Splitting at user55... moving to rs0... done. Splitting at user56... moving to rs1... done. Splitting at user57... moving to rs0... done. Splitting at user58... moving to rs1... done. Splitting at user59... moving to rs0... done. Splitting at user60... moving to rs1... done. Splitting at user61... moving to rs0... done. Splitting at user62... moving to rs1... done. Splitting at user63... moving to rs0... done. Splitting at user64... moving to rs1... done. Splitting at user65... moving to rs0... done. Splitting at user66... moving to rs1... done. Splitting at user67... moving to rs0... done. Splitting at user68... moving to rs1... done. Splitting at user69... moving to rs0... done. Splitting at user70... moving to rs1... done. Splitting at user71... moving to rs0... done. Splitting at user72... moving to rs1... done. Splitting at user73... moving to rs0... done. Splitting at user74... moving to rs1... done. Splitting at user75... moving to rs0... done. Splitting at user76... moving to rs1... done. Splitting at user77... moving to rs0... done. Splitting at user78... moving to rs1... done. Splitting at user79... moving to rs0... done. Splitting at user80... moving to rs1... done. Splitting at user81... moving to rs0... done. Splitting at user82... moving to rs1... done. Splitting at user83... moving to rs0... done. Splitting at user84... moving to rs1... done. Splitting at user85... moving to rs0... done. Splitting at user86... moving to rs1... done. Splitting at user87... moving to rs0... done. Splitting at user88... moving to rs1... done. Splitting at user89... moving to rs0... done. Splitting at user90... moving to rs1... done. Splitting at user91... moving to rs0... done. Splitting at user92... moving to rs1... done. Splitting at user93... moving to rs0... done. Splitting at user94... moving to rs1... done. Splitting at user95... moving to rs0... done. Splitting at user96... moving to rs1... done. Splitting at user97... moving to rs0... done. Splitting at user98... moving to rs1... done. Splitting at user99... moving to rs0... done.

am I succeed ? and what does "user1,....,user99" means?

[allanbank]

Yes - That looked like it worked. There should have been a message about the balancer starting at the end but that is not critical. You can make sure the balancer is running via the following in the Mongo shell:

// Start the balancer again.
sh.startBalancer();

I had put a comment at the top of the script to try and explain what was going on. Does this explain where the user01, user02, ... are coming from?

// YCSB uses the database/collection ycsb/usertable. The documents have
// a key of the form 'user<large_decimal_number>' so we create splits 
// at 'user<digit><digit>' to make sure there are enough chunks (10*10=100) 
// to spread fairly evenly across shards. With 100 chunks a cluster should 
// only see a 1% variance in the number of documents per shard (no shard 
// will have more than 1 extra chunk of data). 10 chunks will produce a 10% 
// variance, etc.

[annisaMS]

I see, thank you but @allanbank can I ask 1 more question ? for example, I had a data file named data.csv (4GB), can I use it in YCSB load phase ?

[allanbank]

There are tools to load csv files, e.g., mongoimport. If the data.csv file was created with mongoexport after a YCSB load then you can reload the data via mongoimport. That is a good way to ensure that different runs use the same data set. If you use mongoimport then you would not do a ycsb load before the ycsb benchmark run. (I'll also say that mongoexport/mongorestore may perform better at the cost of the the data file not being human readable.)

If the data.csv is not from a YCSB load then you won't be able to use it with YCSB. I know of no way to use externally derived data with YCSB. YCSB needs to be able to predict the names of the _id values and other fields.

I'll also caution that the script only works for the YCSB data sets. To properly presplit and balance you have to understand your shard key space and how to determine appropriate split points for that key space. To do that you have to understand the data and the query patterns you will be using. Again, look at the comments in the script for some discussion on the types of things to consider. It is not complete, but will get you pointed in the right direction.

Rob.

[annisaMS]

so, it means if I want to still use data.csv, I have to import it to ycsb (db) first followed by ycsb run phase (without load phase) ? and how to make sure that the load phase is success? db.count ?

[allanbank]

Yes. import and then run.

mongoimport will tell you if the load was successful. You can also do a db.usertable.count() to check the number of documents after the load.

[annisaMS]

Ok, after the run phase is done, I checked my usertable data and it's like being encrypted, and I count it too, it showed that I have 1000 documents, is that the default?

[allanbank]

The data YCSB uses is mostly random bytes converted to strings so it will look "encrypted" but isn't. Its just random.

The number of documents is dependant on the workload's configuration. As an example for workloada (in the repo) the record count is 1,000. I think most of the provided workloads default to 1,000. That is fine for initial tests to understand the process of running YCSB but for real benchmarks against MongoDB the values should be a few orders of magnitude higher but will be dependant on the hardware you have available. You want the benchmark to run for at least 10's of minutes to make sure you get reliable/consistent results. See #1014 for more discussions on that.

Rob

[annisaMS]

@allanbank " That is a good way to ensure that different runs use the same data set" can I said that I dont have to repeat load phase per each workload ? single load phase for all workloads run phase?

[annisaMS]

after importing data using mongoimport into ycsb , I do the run phase using = ./bin/ycsb run mongodb -P workloads/workloada > workloada.txt and I got this OVERALL], RunTime(ms), 2500.0 [OVERALL], Throughput(ops/sec), 400.0 [READ], Operations, 502.0 [READ], AverageLatency(us), 1851.5776892430279 [READ], MinLatency(us), 854.0 [READ], MaxLatency(us), 134911.0 [READ], 95thPercentileLatency(us), 1817.0 [READ], 99thPercentileLatency(us), 4903.0 [READ], Return=NOT_FOUND, 502 [CLEANUP], Operations, 1.0 [CLEANUP], AverageLatency(us), 1389.0 [CLEANUP], MinLatency(us), 1389.0 [CLEANUP], MaxLatency(us), 1389.0 [CLEANUP], 95thPercentileLatency(us), 1389.0 [CLEANUP], 99thPercentileLatency(us), 1389.0 [UPDATE], Operations, 498.0 [UPDATE], AverageLatency(us), 1609.562248995984 [UPDATE], MinLatency(us), 994.0 [UPDATE], MaxLatency(us), 43647.0 [UPDATE], 95thPercentileLatency(us), 1747.0 [UPDATE], 99thPercentileLatency(us), 2499.0 [UPDATE], Return=NOT_FOUND, 498

[allanbank]

@allanbank " That is a good way to ensure that different runs use the same data set" can I said that I dont have to repeat load phase per each workload ? single load phase for all workloads run phase?

That depends on the workload you are running. For the core workloads there is some guidance on when the data in the database needs to be dropped and the load stage repeated: Core-Workloads -> running-the-workloads. Generally a workload that adds or removes records will need the data dropped and reloaded.

after importing data using mongoimport into ycsb , I do the run phase using = ./bin/ycsb run mongodb -P workloads/workloada > workloada.txt and I got this OVERALL], RunTime(ms), 2500.0 [OVERALL], Throughput(ops/sec), 400.0 [READ], Operations, 502.0

[READ], Return=NOT_FOUND, 502 [UPDATE], Operations, 498.0

[UPDATE], Return=NOT_FOUND, 498

The fact that every operation (reads and updates) is NOT_FOUND means that the data you loaded and the data that workloada expects did not match. Are you sure that the data you imported was the results of a YCSB load using workloada?

[annisaMS]

Load,export and import? Actually ,where's ycsb data from? is ycsb has provided its own data?

[allanbank]

Have you read the Running-a-Workload wiki page?

YCSB generates its own random data. That is why you always have to do a 'ycsb load ...' first.

When I said:

That is a good way to ensure that different runs use the same data set.

I was referring to re-running a test repeatedly (which you should be doing to ensure you get consistent results). But again,

If the data.csv is not from a YCSB load then you won't be able to use it with YCSB.

If you want to save off the data YCSB created for multiple runs:

1. Prepare the cluster and split/balance the usertable collection.
2. Run a 'ycsb load ...' to load the data for a workload.
3. Do a mongoexport to save the generated data.
4. Run the 'ycsb run ...' to perform the test.
5. Run a db.usertable.remove({}) the data from the collection in MongoDB. Don't drop the collection or database or you will lose the splits created.
6. Load the data from step 2 again via mongoimport.
7. Goto 4.

Again you can use the guidance in Core-Workloads -> running-the-workloads for when you need to reload between test runs (e.g. when you need to do steps 4, 5, 6 vs just being able to run the next test via step 4).

[annisaMS]

ok , I've done the load process using this configuration:

recordcount=4000000 operationcount=1000 workload=com.yahoo.ycsb.workloads.CoreWorkload

readallfields=true

readproportion=0.5 updateproportion=0.5 scanproportion=0 insertproportion=0

fieldcount=10 fieldlength=100 requestdistribution=zipfian

then the data in my ycsb is 4GB,but after running run phase, I got this [READ], Return=OK, 491 [UPDATE], Return=OK, 509

why I only got 1000 ?

[allanbank]

The operatiocount is 1000.

[annisaMS]

thanks, my problems resolved