Go-Explore

[schrum2]

Go-Explore is an algorithm that showed amazing performance in Montezuma's Revenge and Pitfall on the Atari. It is a bit unusual ... it plans out sequences of deterministic actions and saves promising sequences for the sake of later exploration. It is sort of like Brute and JERK in this way, but better because of how it maintains an archive of multiple trajectories. Later, in a robustification stage, a deep net is trained to imitate the best trajectory using imitation learning.

We should see if we can recreate the results on Pitfall and Montezuma's Revenge, and then also see if we can extend the algorithm to Sonic.

Blog post here: https://eng.uber.com/go-explore/

Source code here: https://github.com/uber-research/go-explore

Research article here: https://arxiv.org/pdf/1901.10995.pdf

[nazaruka]

Copied Go-Explore and tried running the original call (./phase1.sh --help) to no avail. After examining phase1.sh further, I realized that all it was doing was calling the main.py file in the goexplore_py directory:

PYTHONPATH="${PYTHONPATH}:." python3 goexplore_py/main.py "$@"

All the other shell files merely called to phase1.sh before setting the values of some optional arguments. Running Montezuma with domain knowledge, for instance, is enabled through the phase1_montezuma_domain.sh file, which contains:

./phase1.sh \
    --high_score_weight=10.0 \
    --horiz_weight=0.3 \
    --vert_weight=0.1 \
    --batch_size 1000 \
    --max_hours=240 \
    --max_compute_steps=150000000 \
    --remember_rooms

Instead of following through with this simple call, I opted for another form of calling in Anaconda. I first made sure that it worked at all and called python -m goexplore_py.main --help. Sure enough, I received the list of appropriate command-line arguments and their descriptions, many of which were taken by the aforementioned shell files. However, when I ran python -m goexplore_py.main --high_score_weight=10.0 --horiz_weight=0.3 --vert_weight=0.1 --batch_size=1000 --max_hours=240 --max_compute_steps=150000000 --remember_rooms, I received the following error:

Code hash: 575219b1e5e93361be2d2249f0d6d91d810a36942a1fa3b65302f73a255b61c5
Experiment running in ./results//0000_a0a2c7e68e984212b251ff1e2d6c9b77/
Traceback (most recent call last):
  File "C:\Users\Admin\Anaconda3\lib\multiprocessing\context.py", line 190, in get_context
    ctx = _concrete_contexts[method]
KeyError: 'fork'

followed by

Traceback (most recent call last):
  File "C:\Users\Admin\Anaconda3\lib\runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "C:\Users\Admin\Anaconda3\lib\runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "C:\Users\Admin\Desktop\Southwestern\SCOPE\Files\Repo\go-explore\goexplore_py\main.py", line 409, in <module>
    reset_cell_on_update=args.reset_cell_on_update)
  File "C:\Users\Admin\Desktop\Southwestern\SCOPE\Files\Repo\go-explore\goexplore_py\main.py", line 288, in run
    _run(base_path=base_path, **kwargs)
  File "C:\Users\Admin\Desktop\Southwestern\SCOPE\Files\Repo\go-explore\goexplore_py\main.py", line 100, in _run
    pool_cls = multiprocessing.get_context(start_method).Pool
  File "C:\Users\Admin\Anaconda3\lib\multiprocessing\context.py", line 238, in get_context
    return super().get_context(method)
  File "C:\Users\Admin\Anaconda3\lib\multiprocessing\context.py", line 192, in get_context
    raise ValueError('cannot find context for %r' % method)
ValueError: cannot find context for 'fork'

I'm tempted to believe this issue has to do with Windows-UNIX incompatibility, since fork is a popular UNIX call.

[schrum2]

We're going to give up on this issue for now due to Windows compatibility issues. Closing ...