search

basho / basho_docs

Basho Products Documentation
http://docs.basho.com
Other
169 stars 191 forks source link

CS: Capture any resource constraints on scheduling usage [JIRA: PMT-112] #1597

Open macintux opened 9 years ago

macintux commented 9 years ago

https://github.com/basho/basho_docs/pull/1596#discussion_r27227959 (re source/languages/en/riakcs/cookbooks/Usage-and-Billing-Data.md)

Basho-JIRA commented 9 years ago

I have no idea what the resource constraints surrounding scheduling multiple storage calculations are. I don't think this should be my issue.

_[posted via JIRA by Drew Pirrone-Brusse]_

kuenishi commented 9 years ago

Storage calculation is actually a sequence of MapReduce jobs. It works as follows:

  1. run list_keys over all users and find out all bucket names of each user
  2. for each user, repeat this sequencially:
    • for each bucket, sequencially, run MapReduce job folding all manifests over a bucket resolving active manifests to find its sizes or sizes of parts
    • Merge results of all buckets and put back to Riak

So the number of MapReduce jobs are O(user) * O(bucket) . Although those MapReduces run sequencially and all manifests are on leveldb, their performance depends on leveldb's scan performance. It also bites IO bandwidth as well as garbage collection, bitcask merge, AAE tree build and normal users' requests. It may affect all of them, or they may affect each other, on performance.

So, it is constrained by IO bandwidth. Also, capacity to run fold in vnodes and leveldb's scan performance. But note that no multiple storage calculations run in parallel.