Evaluation on Test Worlds

[paischer101]

Hi!

Thank you for the great codebase! I ran into some errors when evaluating on the testworlds mentioned in the paper. I trained an agent using your codebase and want to evaluate it on the testworlds as used in the paper. Training went seamlessly on an A100. Then I followed the step-by-step guide in the README, but I am getting a shader-related error:

2023-01-09 18:19:07.696 | ERROR | avalon.datagen.godot_env.interactive_godot_process:_raise_godot_error:108 - Godot error: Cannot open file 'res://Shaders/BasicColor.material'.

I attached the full stack trace here: stacktrace.log Are there maybe some resource files missing in order to evaluate on the test worlds?

[micimize]

Hmm – looks like the main issue is that we renamed the Shaders dir to shaders at some point after generating these test worlds.

You can either try sed -i 's|res://Shaders|res://shaders|g' path/to/testworlds/*/*.tscn or just rename avalon/datagen/godot/shaders as a stop gap?

Am currently getting my env setup to help debug this particular issue better – thanks!

[micimize]

Based on my reproduction, the sed command should work, and there aren't any other changed paths.

Also looks like you already figured out but the path needs to be /tmp/avalon_worlds/2f788115-ea32-4041-8cae-6e7cd33091b7/ instead of what we had documented.

LMK if that works or if you have any other issues/questions

[paischer101]

Using the sed command works, thank you! Yes, I figured out the path quite quickly, but still it might be good to update it in the README section.

[micimize]

Great! Yes – I've made an internal issue to fix both the test worlds and the README.md :+1: