Level Generation is not consistent between certain runs using the same environment.

[GoogleCodeExporter]

I spent all day looking into this one, and I'm not sure of the cause but I have 
an isolated test case and some fixes.

What steps will reproduce the problem?
1. Download and run LevelGenBugTest.java
2. Remember the Fitness of the 2nd run.
3. Run the replay of the second run

What is the expected output? What do you see instead?
Expect the Fitnesses to be the same.  The fitnesses are different.

What version of the product are you using? On what operating system?
Newest from Subversion. Windows 7 etc.

Please provide any additional information below.
This is a bug that happens specifically when doing multiple runs in the same 
instance with different arguments.  Additionally, the specific trigger seems to 
be the '-lb on' argument.  If this argument is set then any following run which 
does not have that set will generate a different level than it should.

You can test that by switching to '-lb off' and noticing that the problem fixes 
itself.

I'm not sure exactly why the problem happens but it can be fixed by adding one 
line to LevelGenerator.createLevel() right after line 110 and before 111:
counters = new Level.objCounters();

It seems like old data from the previous runs is messing up subsequent runs but 
I can't see how it's happening exactly.

It's not a problem with the Recorder because I have my own replay system which 
shows the same problem and is fixed similarly.  (However, if the Recorder was 
working a bit better it wouldn't show the problem. see issue 10)

Original issue reported on code.google.com by melin...@gmail.com on 28 Oct 2010 at 9:30

Attachments:

• LevelGenBugTest.java

[GoogleCodeExporter]

Issue 8 has been merged into this issue.

Original comment by Sergey.K...@gmail.com on 28 Oct 2010 at 10:38

[GoogleCodeExporter]

all right! We now have our "Issue 9"! Let's GO for it! ;-)

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 9:50

• Changed state: Accepted

[GoogleCodeExporter]

Erik, Great job!

I would like to suggest you to join as a committer, so that you can include 
unit tests and suggestions easier. Write me an e-mail about that - we'll 
discuss!

You test contains important thing:
MarioEnvironment is a Singleton and it stores EvaluationInfo concerning only 
the latest run. Then it's fine that same task outputs different information 
after next run from another task. Benchmark now expects a user to store and 
utilize the evaluation info on his own, however now, with various tasks coming 
it looks like a nice idea to move this functionality up (to Tasks)

It is more a bit logical or design issue -- if we create two separate tasks, it 
could be more natural to store the latest evaluation info in corresponding 
task. Not in Environment. Agree! It will be redesigned.

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 11:20

• Added labels: Type-Enhancement
• Removed labels: Type-Defect

[GoogleCodeExporter]

So, fitnesses will remain different if you use

System.out.println(basicTask.getEnvironment().getEvaluationInfoAsString());

this is necessary due to cross-language usage: we propose using only 
environment as the sole mean of communication between agent and the Mario 
world. Tasks are build up upon Environments.

but I'll add basicTask.getEvaluationInfo() and this data will be Task-specific. 
In other languages, that use Mario AI through AmiCo, it will be necessary to 
wrap up with own tasks. 

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 12:05

[GoogleCodeExporter]

I'll send you an email about that.

Your comments about storage of the EvaluationInfo seem correct!  If many tasks 
are going to be used at once I think it makes sense.

I also want to clarify what I think is the bigger problem here though.

What I was trying to show is that with the same seed and same arguments that 
two different levels are created sometimes. (Of course, this will probably be 
fixed if the level generator is being redone in the future.)

Original comment by melin...@gmail.com on 29 Oct 2010 at 12:08

[GoogleCodeExporter]

I realized that my I was making the test a bit harder than I thought.

I created a much more isolated unittest which I added to 
LevelGeneratorTest.java.

Hopefully, it makes it easier to see specifically what I meant.

Original comment by melin...@gmail.com on 29 Oct 2010 at 12:23

Attachments:

• bug_unittest.patch

[GoogleCodeExporter]

That last test was broken. :)

Making a new one.

Original comment by melin...@gmail.com on 29 Oct 2010 at 12:24

[GoogleCodeExporter]

have a look at r615 -- that will fix this issue

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 12:25

[GoogleCodeExporter]

now you can put it directly to SVN!

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 12:28

[GoogleCodeExporter]

Yes r615 did fix the issue.  I added a regression test in r617.

Original comment by melin...@gmail.com on 29 Oct 2010 at 12:55

[GoogleCodeExporter]

This issue was closed by revision r618.

Original comment by Sergey.K...@gmail.com on 29 Oct 2010 at 1:08

• Changed state: Fixed