kabirkbr
commented
6 years ago Running stressTesting code on the basis of PR #67 seem to indicate memory leaks either in Agent.class or Simulation.class. Three indicative runs with the following parameters:
018-11-21 13:24:14,116 [Test worker] INFO Experiments.class :
method=stressTesting :
message=after storing agentId and vertexId in Agent.class as parameters; may solve the timeout of getVertexId(); :
experimentId=EXP11-21-01-24-VBMaha :
simulationId=SIM11-21-01-24-HcHRNr--ST :
parameters={
similarityConnectThreshold=0.99,
initialAgentNumber=50,
newAgentsPerSecond=1,
chainLenghts=[5, 10],
maxSimilaritySearchDistance=20,
maxCycleSearchDistance=15,
similaritySearchThreshold=0.99,
intialGraphType=smallWorld,
timeoutStressTesting=7200000,
randomPairsRatePerAgentPerDay=1.50,
chainCreationRatePerAgentPerDay=0.140,
randomCycleSearchRatePerAgentPerDay=3.0,
targetedCycleSearchRatePerAgentPerDay=0.120,
searchAndConnectRatePerAgentPerDay=1.0,
millisBetweenRandomPairsPerAgent=57600036.86402359,
millisBetweenChainCreationPerAgent=617142998.2041139,
millisBetweenRandomCycleSearchPerAgent=28800018.43201180,
millisBetweenTargetedCycleSearchPerAgent=719994240.046080,
millisBetweenSearchAndConnectPerAgent=86400000.5529600035}
2018-11-21 15:03:38,813 [Test worker] INFO Experiments.class :
method=stressTesting :
message=after storing agentId and vertexId in Agent.class as parameters; may solve the timeout of getVertexId(); :
experimentId=EXP11-21-03-03-NWmsEL :
simulationId=SIM11-21-03-03-udbAEI--ST :
parameters={
similarityConnectThreshold=0.99,
initialAgentNumber=50,
newAgentsPerSecond=1,
chainLenghts=[5, 10],
maxSimilaritySearchDistance=20,
maxCycleSearchDistance=15,
similaritySearchThreshold=0.99,
intialGraphType=smallWorld,
timeoutStressTesting=7200000,
randomPairsRatePerAgentPerDay=1.50,
chainCreationRatePerAgentPerDay=0.140,
randomCycleSearchRatePerAgentPerDay=3.0,
targetedCycleSearchRatePerAgentPerDay=0.120,
searchAndConnectRatePerAgentPerDay=1.0,
millisBetweenRandomPairsPerAgent=57600036.86402359,
millisBetweenChainCreationPerAgent=617142998.2041139,
millisBetweenRandomCycleSearchPerAgent=28800018.43201180,
millisBetweenTargetedCycleSearchPerAgent=719994240.046080,
millisBetweenSearchAndConnectPerAgent=86400000.5529600035}
2018-11-21 19:45:27,940 [Test worker] INFO Experiments.class :
method=stressTesting :
message=after storing agentId and vertexId in Agent.class as parameters; may solve the timeout of getVertexId(); :
experimentId=EXP11-21-07-45-sIfu52 :
simulationId=SIM11-21-07-45-6DijXy--ST :
parameters={
similarityConnectThreshold=0.99,
initialAgentNumber=50,
newAgentsPerSecond=1,
chainLenghts=[5, 10],
maxSimilaritySearchDistance=20,
maxCycleSearchDistance=15,
similaritySearchThreshold=0.99,
intialGraphType=smallWorld,
timeoutStressTesting=7200000,
randomPairsRatePerAgentPerDay=1.50,
chainCreationRatePerAgentPerDay=0.140,
randomCycleSearchRatePerAgentPerDay=3.0,
targetedCycleSearchRatePerAgentPerDay=0.120,
searchAndConnectRatePerAgentPerDay=1.0,
millisBetweenRandomPairsPerAgent=57600036.86402359,
millisBetweenChainCreationPerAgent=617142998.2041139,
millisBetweenRandomCycleSearchPerAgent=28800018.43201180,
millisBetweenTargetedCycleSearchPerAgent=719994240.046080,
millisBetweenSearchAndConnectPerAgent=86400000.5529600035}crash after reaching 1500 - 2000 agents:
cat OfferNet.log | grep '<init>' | grep SIM11-21-01-24-HcHRNr--ST | wc -l
2005
cat OfferNet.log | grep '<init>' | grep SIM11-21-03-03-udbAEI--ST | wc -l
1655
cat OfferNet.log | grep '<init>' | grep SIM11-21-07-45-6DijXy--ST | wc -l
1324all system resources reach the maximum (all three simulations are shown in the same graph, peak indicate times of crash):
First simulation crashed because of the timeout in actor system; the two last simulations crashed when java heap filled all availabe RAM. First thing to check was how many RAM is allocated to which processes; DSE uses half of system's memory (32GB); Elastic Stack infrastructure uses another 10GB, and offernet code fills all the rest when running. But this does not seem correct -- Agents should not take this amount of memory.
This is what JVisualVM says so far (for the third simulation):
It seams from the heap histogram that most of the memory is taken by LinkedHashMaps and all kinds of arrays, which are probably global structures. Will need to track them down and optimize...
To do next:
- Reduce DSE heap from 32 to 16 and see if it still handles the load;
- Profile offernet code and see what eats resources (starting from above debugging info);
Wish me luck.
Note, that this is the decrease from 32 to 16GB (!?).
Here, the the max_pc_pause_limit is set to maximum 
Stress testing of the current framework would involve persistently running new cycle generation and search processes on a moderately large network (1-5 million agents), based on the code for milestone 0.0.1. All processes will be decentralized. The goal is to push the scale of simulations (in terms of number of actors, number of links between them, number events per second etc) to somewhere that may resemble real setting.
The minimum scale of stress test simulation would be: number of agents: 1M; number of active daily agents: 100K; 'active daily agents' are those which put new offer-demand pairs or those who initiate search processes.
The program maximum would be reach the scale of the network when the framework cannot cope with the size and load anymore...