search

basho / yokozuna

Riak + Solr
245 stars 76 forks source link

Simultaneous querying and writing causes queries to be very slow and writes to fail [JIRA: RIAK-1860] #499

Open ondevboriso opened 9 years ago

ondevboriso commented 9 years ago

I'm having problems writing to the bucket configured with Riak Search while simultaneously querying it with Solr queries (through Riak). Both querying and writing work fine if done separately. However, even at the very moderate rate of writes, the same queries take an order or two of magnitude more time to complete, while a lot of puts are timing out. The cluster is 4 nodes on a decent hardware. Is it a known case of misconfiguration? Where should I look to figure out what causes that?

zeeshanlakhani commented 9 years ago

@ondevboriso can you use etop... via http://docs.basho.com/riak/latest/ops/running/tools/riak-admin/#top and list out the "top" processes during a query.

Are you having bottlenecks querying the keys in riak themselves? How large are the objects/documents?

ondevboriso commented 9 years ago

Are you having bottlenecks querying the keys in riak themselves?

No I didn't notice any bottlenecks when using 'get' on the bucket keys, if that's what you meant.

How large are the objects/documents?

Less than 10 fields, the biggest is about 100 bytes.

Here is the sample of top output during the query:

Pid Name or Initial Func Time Reds Memory MsgQ Current Function <6453.31275.1331> mochiweb_acceptor:init/3 '-' 223021 88616 0 gen:do_call/4
<6453.24244.1331> mochiweb_acceptor:init/3 '-' 214414 88616 0 gen:do_call/4
<6453.27699.1324> mochiweb_acceptor:init/3 '-' 207531 55128 0 gen:do_call/4
<6453.30583.1324> mochiweb_acceptor:init/3 '-' 198664 88616 0 gen:do_call/4
<6453.29918.1324> mochiweb_acceptor:init/3 '-' 191680 55128 0 gen:do_call/4
<6453.22742.1331> mochiweb_acceptor:init/3 '-' 190544 88616 0 gen:do_call/4
<6453.27374.1331> mochiweb_acceptor:init/3 '-' 189698 55128 0 gen:do_call/4
<6453.659.0> ibrowse_http_client:init/ '-' 189170 55776 0 gen_server:loop/6
<6453.15633.1331> ibrowse_http_client:init/ '-' 187882 55776 0 gen_server:loop/6
<6453.27072.1324> mochiweb_acceptor:init/3 '-' 187656 55128 0 gen:do_call/4