CPU usage increasing over time

[GoogleCodeExporter]

I have only the boot node up and running and see after a while (less then a 
day) a quite 
high CPU usage (40%) of the boot node, and it is monotonically increasing. When 
I do a 
process snapshot, I can see a lot of messages sent, it looks like between the 
dht_node and 
cyclon. I have two small screenshot attached.

<0.119.0> -> cyclon
<0.103.0> -> dht_node

So far it looks like the gen_component of cyclon is stuck in a recursion loop 
(Maybe no tail 
recursion?). I will try to monitor a bit more, e.g. using more than 1 node and 
see if the 
same effect will occur.
But maybe you have already seen a similar behavior and have a tip why cyclon is 
so active...

I'm running the latest SVN r790 and this is part of my configuration:

-define(RM, rm_tman).
-define(BUILTIN, true).
-define(LOG_MESSAGE(MESSAGE, SIZE), ok).
-define(DEBUG_REGISTER(PROCESS,PID),ok).
-define(LOG_CS_API(Timer, Time), ok).

Original issue reported on code.google.com by Uwe.Daue...@gmail.com on 21 May 2010 at 4:01

[GoogleCodeExporter]

Original comment by Uwe.Daue...@gmail.com on 21 May 2010 at 4:03

Attachments:

• cyclon_messages.png
• gperf_scalaris.png

[GoogleCodeExporter]

I attached another load plot, the first row showing the processes CPU load in 
blue. 
This time it is an out of the box SVN checkout installation, just started the 
boot 
node with no keys stored and no other nodes communicating to.
Also attached is a RedBug trace from the boot node tracing the 
gen_component:loop/4. 
There it looks cyclon and dht_node receive many messages. (Maybe this is just 
the phenomenon when the systems tries to load balance out as it is responsible 
for too 
many replications...?)
I will let a small set of nodes (4) run over the weekend to see if the same 
effect 
happens then. Also to see if the load trend is expected to level off at a 
certain 
amount.

Original comment by Uwe.Daue...@gmail.com on 22 May 2010 at 3:03

Attachments:

• boot_node_load_1h.png
• redbug_trace.txt

[GoogleCodeExporter]

Having >= nodes than replica amounts running, the CPU load stays low. So I 
assume the 
load balancing was the cause.
Is this behaviour intended?

Original comment by Uwe.Daue...@gmail.com on 22 May 2010 at 3:30

[GoogleCodeExporter]

Could you please run the following in your shell, when the problem happens?
[erlang:process_info(P, messages) || P <- 
process_dictionary:find_all_processes(cyclon)].
It will return the messages currently in cyclon's message queue.

I tried to verify your problem, but I don't get the high load. You are using an
umodified Scalaris from the SVN and run ./bin/boot.sh, right?

Original comment by schu...@gmail.com on 28 May 2010 at 8:04

[GoogleCodeExporter]

Yes, I run an unmodified Scalaris from the SVN and run ./bin/boot.sh. It also 
happens 
when I run a customized version.
I will let a set run during the day or weekend and then apply the process_info 
command.

Original comment by Uwe.Daue...@gmail.com on 28 May 2010 at 8:07

[GoogleCodeExporter]

I did run a single node over the weekend, but it looks like I can't reproduce 
this 
issue with the latest SVN revision anymore. The CPU stays idle and in the 
cyclon 
process the message queue is mostly empty; sometimes this:

[{messages,[{check_state},
            {check_state},
            {check_state},
            {check_state},
            {check_state},
            {check_state},
            {check_state},
            {check_state},
            {check_state}]}]

which I guess is fairly OK.
I guess this issue can be closed as I can't reproduce it right now.

Original comment by Uwe.Daue...@gmail.com on 30 May 2010 at 4:45

Attachments:

• 1_node_svn_r817_fair_cpu_usage.png

[GoogleCodeExporter]

Original comment by schin...@gmail.com on 31 May 2010 at 9:01

• Changed state: Invalid