GBEA mario suite - Githubissues

CIGbalance commented 5 years ago

Things are working for me now, even for the simulated functions. Took a bit to figure out setting the paths correctly. Let me know if you have trouble.

CIGbalance commented 5 years ago

Btw, I tested with the python version

ttusar commented 5 years ago

I get an AttributeError: module 'torch' has no attribute 'no_grad' error. Which torch version do you have?

ttusar commented 5 years ago

pip freeze says I have

torch==0.3.1.post2
torchvision==0.2.0

ttusar commented 5 years ago

Also, I've noticed you changed sim=30 to 1. Was it intential to keep it to 1 or is this just for testing the communication? [I just wish to be careful we don't forget to set something back...]

CIGbalance commented 5 years ago

I get an AttributeError: module 'torch' has no attribute 'no_grad' error. Which torch version do you have?

That's probably because I've updated the code to work with the current versions of torch and torchvision. I have:

torch 1.2.0
torchvision 0.4.0

But it should work with whatever installation candidate pops up for you currently.

CIGbalance commented 5 years ago

Also, I've noticed you changed sim=30 to 1. Was it intential to keep it to 1 or is this just for testing the communication? [I just wish to be careful we don't forget to set something back...]

That's actually something I wanted to talk to you about. The number of simulations was set to 30 initially, because that was the best trade-off I could find. However, noise is varied across the landscape and I think it would be good to have the option to determine a lower number of simulations and let the algorithm handle the noise. Simon expressed a lot of interest in that.

Do you know a good way to expose this parameter? Could be a suite option?

However, that would require us to have a way of assessing the "true" value. One way would be of doing more runs during the experiments. Another (perhaps easier one) would be to just log everything during an experiment and compute the "true" value in post-processing.

Any thoughts on that?

But to answer your question: The main reason for switching to 1 now was to make testing faster :)

ttusar commented 5 years ago

After some effort (I forgot installing torch is non-trivial), and a bug fix on my side, I'm now able to run experiments with the mario-gan suite on Mac and Linux, but it still fails on Windows. The problem are the simulated functions. This is the error I get:

Received message: n mario-gan o 1 f 11 i 1 d 10 x 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at communication.GANProcess.launchGAN(GANProcess.java:49)
        at communication.GANProcess.start(GANProcess.java:73)
        at cmatest.MarioEvalFunction.<init>(MarioEvalFunction.java:60)
        at coco.evaluateLevel.main(evaluateLevel.java:100)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessImpl.create(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
        at java.base/java.lang.ProcessImpl.start(Unknown Source)
        ... 6 more
Exception in thread "main" java.lang.NullPointerException
        at communication.Comm.processCommRecv(Comm.java:54)
        at communication.Comm.commRecv(Comm.java:47)
        at cmatest.MarioEvalFunction.<init>(MarioEvalFunction.java:68)
        at coco.evaluateLevel.main(evaluateLevel.java:100)
Error within message evaluation: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.
Error: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.
Traceback (most recent call last):
  File "code-experiments\rw-problems\socket_server.py", line 109, in <module>
    socket_server_start()
  File "code-experiments\rw-problems\socket_server.py", line 102, in socket_server_start
    raise e
  File "code-experiments\rw-problems\socket_server.py", line 95, in socket_server_start
    response = evaluate_message(message)
  File "code-experiments\rw-problems\socket_server.py", line 56, in evaluate_message
    raise e
  File "code-experiments\rw-problems\socket_server.py", line 48, in evaluate_message
    y = evaluate(suite_name, num_objectives, func, instance, x)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 361, in evaluate_mario_gan
    out[i] = fun(x, netG, d)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 275, in basicFitnessSimAStar
    return executeSimulation(x, netG, dim, 1, 0)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 120, in executeSimulation
    java_output = subprocess.check_output('java -Djava.awt.headless=true -jar '+path+'/dist/MarioGAN.jar "' + str(x) +'" "' + netG + '" '+str(dim)+' '+str(fun)+' '+str(agent) +' ' +str(sim), shell=True);
  File "C:\Anaconda3\envs\py36coco\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "C:\Anaconda3\envs\py36coco\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.

So, the problem seems to be this command: java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1

I "fixed" the path to this (used backslashes consistently and added the quotes around the first path) java -Djava.awt.headless=true -jar "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\dist\MarioGAN.jar" "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\GAN\overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1 but it didn't help.

Can it be that the problem is that you are explicitly calling python3? I only have python...

(py36coco) D:\Delo\Github\ttusar-coco>python3
'python3' is not recognized as an internal or external command,
operable program or batch file.

(py36coco) D:\Delo\Github\ttusar-coco>python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 16:13:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

ttusar commented 5 years ago

Regarding the simulations. Would you allow to compare the results of algorithms that were ran on problems with a different number of simulations? If yes, then we could find a way to pass this parameter. But if this is not what you want (and I would imagine it's not), then we could "hide" this information in the instance number. And then you would have multiple sets of instances that correspond to different number of simulations (instances 1-10 have 1 simulation, instances 301-310 have 30). I think it shouldn't be too difficult to then use post-processing in such a way, that the instances of different algorithms would have to match to do the comparisons...

ttusar commented 5 years ago

Another thing - can you provide estimates for ideal and nadir points for all bi-objective problem instances? While I'm going to support logging with unknown optimum, the bi-objective logger will still need to know the ideal and nadir points, because it needs them for normalization.

The ideal should be set so that it's actually ideal (cannot be reached). And the nadir should denote a point beyond which the solutions are of a bad quality (you are not really interested in them).

CIGbalance commented 5 years ago

After some effort (I forgot installing torch is non-trivial), and a bug fix on my side, I'm now able to run experiments with the mario-gan suite on Mac and Linux, but it still fails on Windows. The problem are the simulated functions. This is the error I get:

Received message: n mario-gan o 1 f 11 i 1 d 10 x 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan
java.io.IOException: Cannot run program "python3": CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at java.base/java.lang.ProcessBuilder.start(Unknown Source)
        at communication.GANProcess.launchGAN(GANProcess.java:49)
        at communication.GANProcess.start(GANProcess.java:73)
        at cmatest.MarioEvalFunction.<init>(MarioEvalFunction.java:60)
        at coco.evaluateLevel.main(evaluateLevel.java:100)
Caused by: java.io.IOException: CreateProcess error=2, The system cannot find the file specified
        at java.base/java.lang.ProcessImpl.create(Native Method)
        at java.base/java.lang.ProcessImpl.<init>(Unknown Source)
        at java.base/java.lang.ProcessImpl.start(Unknown Source)
        ... 6 more
Exception in thread "main" java.lang.NullPointerException
        at communication.Comm.processCommRecv(Comm.java:54)
        at communication.Comm.commRecv(Comm.java:47)
        at cmatest.MarioEvalFunction.<init>(MarioEvalFunction.java:68)
        at coco.evaluateLevel.main(evaluateLevel.java:100)
Error within message evaluation: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.
Error: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.
Traceback (most recent call last):
  File "code-experiments\rw-problems\socket_server.py", line 109, in <module>
    socket_server_start()
  File "code-experiments\rw-problems\socket_server.py", line 102, in socket_server_start
    raise e
  File "code-experiments\rw-problems\socket_server.py", line 95, in socket_server_start
    response = evaluate_message(message)
  File "code-experiments\rw-problems\socket_server.py", line 56, in evaluate_message
    raise e
  File "code-experiments\rw-problems\socket_server.py", line 48, in evaluate_message
    y = evaluate(suite_name, num_objectives, func, instance, x)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 361, in evaluate_mario_gan
    out[i] = fun(x, netG, d)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 275, in basicFitnessSimAStar
    return executeSimulation(x, netG, dim, 1, 0)
  File "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\mario_gan_evaluator.py", line 120, in executeSimulation
    java_output = subprocess.check_output('java -Djava.awt.headless=true -jar '+path+'/dist/MarioGAN.jar "' + str(x) +'" "' + netG + '" '+str(dim)+' '+str(fun)+' '+str(agent) +' ' +str(sim), shell=True);
  File "C:\Anaconda3\envs\py36coco\lib\subprocess.py", line 336, in check_output
    **kwargs).stdout
  File "C:\Anaconda3\envs\py36coco\lib\subprocess.py", line 418, in run
    output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command 'java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1' returned non-zero exit status 1.

So, the problem seems to be this command: java -Djava.awt.headless=true -jar D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/dist/MarioGAN.jar "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan/GAN/overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1

I "fixed" the path to this (used backslashes consistently and added the quotes around the first path) java -Djava.awt.headless=true -jar "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\dist\MarioGAN.jar" "[0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]" "D:\Delo\Github\ttusar-coco\code-experiments\rw-problems\mario_gan\GAN\overworld-10-5000/netG_epoch_4999_5641.pth" 10 1 0 1 but it didn't help.

Can it be that the problem is that you are explicitly calling python3? I only have python...

(py36coco) D:\Delo\Github\ttusar-coco>python3
'python3' is not recognized as an internal or external command,
operable program or batch file.

(py36coco) D:\Delo\Github\ttusar-coco>python
Python 3.6.5 | packaged by conda-forge | (default, Apr  6 2018, 16:13:55) [MSC v.1900 64 bit (AMD64)] on win32
Type "help", "copyright", "credits" or "license" for more information.

Yes, the issue seems to be that I'm explicitly calling python3 now. I did that change because I couldn't figure out how to set an alias on CodeOcean. I'll change it back and find another workaround for CodeOcean.

CIGbalance commented 5 years ago

Regarding the simulations. Would you allow to compare the results of algorithms that were ran on problems with a different number of simulations? If yes, then we could find a way to pass this parameter. But if this is not what you want (and I would imagine it's not), then we could "hide" this information in the instance number. And then you would have multiple sets of instances that correspond to different number of simulations (instances 1-10 have 1 simulation, instances 301-310 have 30). I think it shouldn't be too difficult to then use post-processing in such a way, that the instances of different algorithms would have to match to do the comparisons...

I actually would want to allow the comparison. You'd just have count the number of function evaluations appropriately. So if sim=30, one evaluation corresponds to 30 evaluations.

CIGbalance commented 5 years ago

Another thing - can you provide estimates for ideal and nadir points for all bi-objective problem instances? While I'm going to support logging with unknown optimum, the bi-objective logger will still need to know the ideal and nadir points, because it needs them for normalization.

The ideal should be set so that it's actually ideal (cannot be reached). And the nadir should denote a point beyond which the solutions are of a bad quality (you are not really interested in them).

Yes, I can and I will. Just wanted to get both function suites running first. Also, I think I don't have the ideal points for the single-objective functions copied back in yet. Will try to look into that soon.

Btw, there is a new thing called "best parameter". Is that the optimum in search space? I have no idea what that is, should I set it to NULL?

ttusar commented 5 years ago

Regarding the simulations. Would you allow to compare the results of algorithms that were ran on problems with a different number of simulations? If yes, then we could find a way to pass this parameter. But if this is not what you want (and I would imagine it's not), then we could "hide" this information in the instance number. And then you would have multiple sets of instances that correspond to different number of simulations (instances 1-10 have 1 simulation, instances 301-310 have 30). I think it shouldn't be too difficult to then use post-processing in such a way, that the instances of different algorithms would have to match to do the comparisons...

I actually would want to allow the comparison. You'd just have count the number of function evaluations appropriately. So if sim=30, one evaluation corresponds to 30 evaluations.

Ah, okay, that makes sense. Let me think about the best way to interface this though...

ttusar commented 5 years ago

Another thing - can you provide estimates for ideal and nadir points for all bi-objective problem instances? While I'm going to support logging with unknown optimum, the bi-objective logger will still need to know the ideal and nadir points, because it needs them for normalization.

The ideal should be set so that it's actually ideal (cannot be reached). And the nadir should denote a point beyond which the solutions are of a bad quality (you are not really interested in them).

Yes, I can and I will. Just wanted to get both function suites running first. Also, I think I don't have the ideal points for the single-objective functions copied back in yet. Will try to look into that soon.

Btw, there is a new thing called "best parameter". Is that the optimum in search space? I have no idea what that is, should I set it to NULL?

best_parameter = the optimum in the search space best_value = the optimum in the objective space nadir_value = the nadir value in the objective space

In single-objective problems, set all to NULL.

In bi-objective problems, best_parameter = NULL, while the best_value and nadir_value need to be set.

CIGbalance commented 5 years ago

Another thing - can you provide estimates for ideal and nadir points for all bi-objective problem instances? While I'm going to support logging with unknown optimum, the bi-objective logger will still need to know the ideal and nadir points, because it needs them for normalization. The ideal should be set so that it's actually ideal (cannot be reached). And the nadir should denote a point beyond which the solutions are of a bad quality (you are not really interested in them).

Yes, I can and I will. Just wanted to get both function suites running first. Also, I think I don't have the ideal points for the single-objective functions copied back in yet. Will try to look into that soon. Btw, there is a new thing called "best parameter". Is that the optimum in search space? I have no idea what that is, should I set it to NULL?

best_parameter = the optimum in the search space best_value = the optimum in the objective space nadir_value = the nadir value in the objective space

In single-objective problems, set all to NULL.

In bi-objective problems, best_parameter = NULL, while the best_value and nadir_value need to be set.

I get an error trying to do this. error: incompatible types when assigning to type ‘double’ from type ‘void *’ Or should I just not set a value at all?

ttusar commented 5 years ago

Ah, it's probably a bit tricky to know which are already allocated and which not. You can just let me know the values and I will take care of it.

ttusar commented 5 years ago

Please see the gbea+logger branch for the combination of GBEA problems with socket communication and loggers (bbob-new and bbob-biobj) that support problems with unknown optima.

I tested it as much as I could, but mostly on Windows (I did run the example experiment in C also on Linux), so I hope it will work for you.

Note that you still have a TODO (search for Vanessa ;-)) in the file mario_gan.c, where you need to set the ideal and nadir points for the bi-objective suite. I have taken care of everything else.

Note that you should use bbob-new instead of bbob for logging single-objective problems.

Unfortunately, I didn't manage to do something about the increased number of simulations. I can do it when I'm back from vacations... I hope you can now perform the experiments you set out to do.

ttusar commented 5 years ago

Oh, on the Linux machine (where I'm able to test also the simulated Mario GAN objectives), the example experiment in C just failed with this error:

Received message: n mario-gan o 1 f 13 i 1 d 10 x 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00 0.00000000e+00
Error within message evaluation: MarioGAN.jar output not formatted as expected, got 0.08589678344726563 
Error: MarioGAN.jar output not formatted as expected, got 0.08589678344726563 
Closing socket
Traceback (most recent call last):
  File "code-experiments/rw-problems/socket_server.py", line 120, in <module>
    socket_server_start(silent=silent)
  File "code-experiments/rw-problems/socket_server.py", line 104, in socket_server_start
    raise e
  File "code-experiments/rw-problems/socket_server.py", line 97, in socket_server_start
    response = evaluate_message(message)
  File "code-experiments/rw-problems/socket_server.py", line 58, in evaluate_message
    raise e
  File "code-experiments/rw-problems/socket_server.py", line 50, in evaluate_message
    y = evaluate(suite_name, num_objectives, func, instance, x)
  File "/home/tea/Documents/ttusar-coco/code-experiments/rw-problems/mario_gan/mario_gan_evaluator.py", line 361, in evaluate_mario_gan
    out[i] = fun(x, netG, d)
  File "/home/tea/Documents/ttusar-coco/code-experiments/rw-problems/mario_gan/mario_gan_evaluator.py", line 275, in basicFitnessSimAStar
    return executeSimulation(x, netG, dim, 1, 0)
  File "/home/tea/Documents/ttusar-coco/code-experiments/rw-problems/mario_gan/mario_gan_evaluator.py", line 124, in executeSimulation
    raise ValueError('MarioGAN.jar output not formatted as expected, got {} '.format(result))
ValueError: MarioGAN.jar output not formatted as expected, got 0.08589678344726563

ttusar commented 5 years ago

Another thing: there is something called the problem_type in the C code, which is used to group functions (in post-processing). I have created two groups for Mario GAN functions like this (file mario_gan.c):

  if (((number_of_objectives == 1) && (function <= 10)) ||
      ((number_of_objectives == 2) && (function <= 2))) {
    coco_problem_set_type(problem, "mario_gan_computed");
  } else {
    coco_problem_set_type(problem, "mario_gan_simulated");
  }

You can change them according to your needs.

Also, if/when you add other suites, do not forget to set the problem type for their problems too.

CIGbalance commented 5 years ago

Thanks! I have a couple of things to look into, it seems, but the basic stuff is working, which is great. So I'll at least be able to demo everything at the tutorial :+1: So enjoy your holiday :)

I'll try to look into all the issues you listed. In order to keep track of them, I'll summarise below:

[ ] Vanessa: calling python3 directly from MarioGAN.jar fails on windows -> find a way to do an alias
[ ] Vanessa: specify value ranges for all existing GBEA suites (nadir + ideal)
[ ] both: Figure out a way to dynamically set the number of simulations
[ ] Vanessa: MarioGAN.jar output not formatted as expected, got 0.08589678344726563 (experiment in c)
[ ] Vanessa: set problem types according to function groups.
[ ] Tea?: Pull request regarding too many messages on sockets in toptrumps

Btw: regarding the specification of the values (like nadir etc) -> I will need to be able to do this dynamically eventually, so I can create new suites with my script. I've managed to adapt the script with the templates to the socket-evaluator, btw (only python version for now). I'll add templates for the post-processing as well, now that the logging works :)

ttusar / coco

GBEA mario suite #36