xinbc / jdiameter

Automatically exported from code.google.com/p/jdiameter
0 stars 0 forks source link

Occasional 'Statistics already defined' during stack startup #70

Open GoogleCodeExporter opened 9 years ago

GoogleCodeExporter commented 9 years ago
What steps will reproduce the problem?
1.Create a diameter client 
2.Start it on a machine with a lot of cores
3.Start will fail occasionally 

What is the expected output? What do you see instead?
Diameter stack should start. Instead, follow exception is thrown: 
org.jdiameter.api.InternalException: java.lang.IllegalArgumentException: 
Statistic already defined: Statistic{ records=[0, 0.0] }
    at org.jdiameter.client.impl.controller.PeerImpl.connect(PeerImpl.java:457)
    at org.jdiameter.client.impl.controller.PeerTableImpl.start(PeerTableImpl.java:275)
    at org.jdiameter.client.impl.StackImpl.startPeerManager(StackImpl.java:246)
    at org.jdiameter.client.impl.StackImpl.start(StackImpl.java:221)
    at ..............................
Caused by: java.lang.IllegalArgumentException: Statistic already defined: 
Statistic{ records=[0, 0.0] }
    at org.jdiameter.common.impl.statistic.StatisticManagerImpl.newStatistic(StatisticManagerImpl.java:130)
    at org.jdiameter.client.impl.fsm.PeerFSMImpl.runQueueProcessing(PeerFSMImpl.java:177)
    at org.jdiameter.client.impl.fsm.PeerFSMImpl.handleEvent(PeerFSMImpl.java:294)
    at org.jdiameter.client.impl.controller.PeerImpl.connect(PeerImpl.java:454)
    ... 42 more

What version of the product are you using? On what operating system?
1.5.10.0, but the same problem exists in the latest source as well

Please provide any additional information below.
Root cause is the check for numberOfThreadsRunning in PeerFSMImpl. There is a 
lock to prevent concurrent access to the numberOfThreadsRunning, but the count 
is increased ouside of this lock (by another thread). So if an event happens 
(eg START event) before the numberOfThreadsRunning has been increased, the 
runQueueuProcessing is executed twice leading to the above error. 

This can be seen in the logs as well. The 'Starting FSM Thread' messages are 
printed first, but at that point numberOfThreadsRunning is not increased yet. 
That only happens when 'FSM threads are running' messages are shown (after the 
exception occured)

[main] DEBUG org.jdiameter.client.impl.controller.PeerTableImpl - Initializing 
Peer Table.
[main] DEBUG org.jdiameter.client.impl.controller.PeerTableImpl - Populating 
peerTable from configuration
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Maximum FSM Queue size 
is [10000]
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Initializing QueueStat 
@ Thread[main]
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Finished Initializing 
QueueStat @ Thread[main]
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Starting FSM Thread 1 
of 3
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Starting FSM Thread 2 
of 3
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Starting FSM Thread 3 
of 3
[main] DEBUG org.jdiameter.client.impl.transport.tcp.TCPTransportClient - 
Destination address is set to [127.0.0.1] : [3868]
[main] DEBUG org.jdiameter.client.impl.transport.tcp.TCPTransportClient - 
Origin address is set to [0.0.0.0] : [0]
[main] DEBUG org.jdiameter.client.impl.controller.PeerTableImpl - Appended peer 
[CPeer{Uri=aaa://127.0.0.1:3868; State=DOWN; 
Con=org.jdiameter.client.impl.transport.tcp.TCPClientConnection@11e70f62}] to 
peer table
[main] INFO org.jdiameter.client.impl.StackImpl - (-)(-)(-)(-)(-) Started  
Mobicents DIAMETER Stack v1.5.10.0-build639 (-)(-)(-)(-)(-)
[main] DEBUG org.jdiameter.client.impl.controller.PeerTableImpl - Starting 
PeerTable. Going to call connect on all peers in the peerTable
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Handling event with 
type [START_EVENT]
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - No FSM threads are 
running so calling runQueueProcessing()
[main] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl - Initializing QueueStat 
@ Thread[main]
[main] WARN org.jdiameter.client.impl.controller.PeerTableImpl - Can not start 
connect procedure to peer [CPeer{Uri=aaa://127.0.0.1:3868; State=DOWN; 
Con=org.jdiameter.client.impl.transport.tcp.TCPClientConnection@11e70f62}]
org.jdiameter.api.InternalException: java.lang.IllegalArgumentException: 
Statistic already defined: Statistic{ records=[0, 0.0] }
[main] DEBUG org.jdiameter.client.impl.controller.PeerTableImpl - Calling start 
on the router
[FSM-aaa://127.0.0.1:3868_2-1] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl 
- Starting ... [3] FSM threads are running
[FSM-aaa://127.0.0.1:3868_1-0] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl 
- Starting ... [2] FSM threads are running
[FSM-aaa://127.0.0.1:3868_3-2] DEBUG org.jdiameter.client.impl.fsm.PeerFSMImpl 
- Starting ... [1] FSM threads are running

Original issue reported on code.google.com by janss...@gmail.com on 27 Jan 2015 at 9:26