search

schrum2 / MM-NEAT

Modular Multiobjective (Hyper) Neuro-Evolution of Augmenting Topologies + MAP-Elites: Java code for evolving intelligent agents in Ms. Pac-Man, Tetris, and more, as well as code for Procedural Content Generation in Mario, Zelda, Minecraft, and more!
http://people.southwestern.edu/~schrum2/re/mm-neat.php
Other
50 stars 20 forks source link

Benchmark Parallel Init in Minecraft #787

Closed schrum2 closed 2 years ago

schrum2 commented 2 years ago

Try running some experiments with parallelMAPElitesInitialize set to true and set to false. Time each and compare. I want to know if we are getting a tangible benefit. You may need to increase either parallelMinecraftSlots and/or mu to notice a difference. Report results in this issue thread.

alejmedinajr commented 2 years ago

I ran both for the same amount of time ("roughly three minutes") just to see if there would be an immediate difference.

This first image is the result for when parallelism is set to True. filledBinsParallelismTrue

This next image is for when parallelism is set to False. filledBinsParallelismFalse

When parallelism is set to True, more bins were filled. However, this may be not be accurate of a decision just yet since the time both were run was only 3 minutes. I think I will run both on the two lab computers in the back for a longer duration of time.

alejmedinajr commented 2 years ago

I set up an actual timer to time the initialization, and the parallelism being true seems to make the initialization faster by ~1000 milliseconds.

Parallelism is False: TimingResultsNOPARALLEL

Parallelism is true: TimingResultsYESPARALLEL

schrum2 commented 2 years ago

Try with Change in Center of Mass since that takes a long time to compute. Parallelism could be more of a benefit there.

alejmedinajr commented 2 years ago

Timing with parallelism with center to mass: Parallelism with centertoMass

Timing with no parallelism with center to mass: No Parallelism with centertoMass

I ran both of these tests with both parallelism and no parallelism using center to mass fitness. It looks like both were faster than the other fitness test that was run, which seems a little confusing.

schrum2 commented 2 years ago

If the fitness function is literally the only thing that changed, then it is odd that the change in center of mass fitness would be faster. However, we might need to increase the delay time used to measure the fitness. We should look carefully at this together at some point

alejmedinajr commented 2 years ago

Did the same timing tests that I ran yesterday, below are the results. This test shows that parallelism being true is slower (although not significantly since a 1000 milliseconds is 1 second).

Timing for when Parallelism is False: 2022-06-14-ParallelismFalse

Timing for when Parallelism is True: 2022-06-14-ParallelismTrue

schrum2 commented 2 years ago

I think the Minecraft server might be a bottleneck that prevents us from seeing a large benefit from parallelism, but I'm not sure. I'm going to let this issue remain dormant for a while and revisit later.

schrum2 commented 2 years ago

You can return to this. Try using change in center of mass fitness, setting minecraftEndEvalNoMovement to false, and making the minecraftMandatoryWaitTime a much bigger number.

schrum2 commented 2 years ago

Since issue #774 was successfully closed, and parallelism definitely showed a benefit there, I'm going to close this issue. Focusing on getting a little extra speed at the start is less important than getting speed through the whole of evolution.