Problems on installing Avalon on intel macOS

[zangobot]

Following the guide described on the GitHub README leads to failing of tests (python -m avalon.common.check_install). I have created a python 3.10 conda environment, installed avalon-rl[train], and then python -m avalon.install_godot_binary. Executed code:

from avalon.agent.godot.godot_gym import AvalonEnv
from avalon.agent.godot.godot_gym import GodotEnvironmentParams
from avalon.agent.godot.godot_gym import TrainingProtocolChoice
from avalon.datagen.env_helper import display_video

env_params = GodotEnvironmentParams(
    resolution=256,
    training_protocol=TrainingProtocolChoice.SINGLE_TASK_MOVE,
    initial_difficulty=0,
)
env = AvalonEnv(env_params)
env.reset()

def random_env_step():
    action = env.action_space.sample()
    obs, reward, done, info = env.step(action)
    if done:
        env.reset()
    return obs

observations = [random_env_step() for _ in range(50)]
display_video(observations, fps=10)

Full error:

File "/Users/<USER>/opt/miniconda3/envs/avalon/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.

[micimize]

What is the specific context you are executing this code in? I've see this crop up in a different conda install while using vscode's jupyter integration specifically (but not the web UI). It might also be related to https://github.com/pytorch/pytorch/issues/5858#issuecomment-482421590.

[zangobot]

A python script, not a notebook, just to test the environment. Also, the python module for testing the installation fails. I will have a look at that issue (which seems to be still open), thanks!

[zplizzi]

@zangobot pretty sure you just need to put your code under if __name__ == '__main__': as that error message suggests. multiprocessing when using the spawn method requires this idiom (I think so that the child processes don't also run code they're not supposed to?). OSX defaults to spawn (but we also set it to spawn for some of the child processes explicitly), it's possible that test that's failing for you works on linux because linux defaults to fork.

[zplizzi]

Yep, check_install needed that guard too, fixed now and you should see the change soon. Thanks for pointing this out! I'll close this but let us know if it's not fixed for you and we can reopen.

[zangobot]

Thank you!

[mx781]

@zangobot we've synced over the changes, so the install should work if you install via git: pip install git+https://github.com/Avalon-Benchmark/avalon. Wwe'll push a pip release after consolidating a few more changes.

[zangobot]

I would need to reopen this. I tried a clean install with the pip install command you sent me, but some dependancies are missing: ModuleNotFoundError: No module named 'tree'

[zplizzi]

Sorry about that, for now that should be fixed by running pip install dm-tree==0.1.7 before retrying the installation. We'll fix the pip dependency list.

[mx781]

@zplizzi yes, looks like torch has slipped into a non-core dependency. @zangobot you can install all the train extras like so:

pip install "git+https://github.com/Avalon-Benchmark/avalon#egg=avalon-rl[train]"

[zangobot]

I think I have found another bug. When trying to install the Godot binaries, this happens:

2022-12-21 09:35:57.009 | INFO     | __main__:install_available_binaries_for_current_platform:130 - Installing editor build from https://github.com/Avalon-Benchmark/godot/releases/download
2022-12-21 09:35:57.014 | INFO     | __main__:fetch_binary:92 - Downloading 3.4.4.avalon.0.9.3/macos-editor.zip into /FOLDER/envs/avalon/lib/python3.10/site-packages/avalon/bin/godot_editor
2022-12-21 09:36:06.834 | INFO     | __main__:install_available_binaries_for_current_platform:144 - Note: No headless runner currently available for macos. Symlinking editor build in its place.

and running the avalon.common.check_install crashes in this way:

avalon.datagen.errors.GodotError: returncode=SIGHUP(1) 
Log: /var/folders/t9/9c25d13553s4q7t2v2c0b0z00000gn/T/science/tmp/4752303b-0a49-4c60-af70-86148b943079/godot.log
Artifacts: /var/folders/t9/9c25d13553s4q7t2v2c0b0z00000gn/T/science/godot/godot_env_artifacts__74530bad-1408-45a2-a47b-1ee630f60d17.tar.gz

[zplizzi]

Can you attach the contents of that log file?

[zangobot]

The content of the log is

realpath: illegal option -- e
usage: realpath [-q] [path ...]

[micimize]

@zangobot looks like you're using the /bin/realpath that ships with macos ventura which appears to be fairly non-standard – for now could you try with brew install coreutils?

We may be able to switch to something like -f $(realpath ...) instead to avoid the need for coreutils on ventura, or even just skip these checks in the usual usage from python as we can verify things before reaching the bash script.

[micimize]

Closing this as it's become a bit of a catch-all. Please open new issues for any other errors y'all encounter.

I've made an internal issue to improve the output of check_install such as outputting some of the godot log if it errors.

[zangobot]

Hello! So, yes, I have already installed the coreutils. I'll open a new issue soon about it then.