TensorKart and Universe openAI

erotavlasme commented 7 years ago

Hi, I just found your project on nvidia blog. Why do not integrate it in https://github.com/openai/universe?

kevinhughes27 commented 7 years ago

I haven't had a chance to look too deep into universe yet but it looks really cool. If its possible to integrate I would love for that to happen!

bzier commented 7 years ago

I have started working on wrapping mupen64plus (and Mario Kart specifically) in a Gym environment. This should make it compatible with Universe as well (if I understand things correctly). When I get a little further, I'll update this with more info.

kevinhughes27 commented 7 years ago

That's super cool. Please keep me posted 😄

erotavlasme commented 7 years ago

Great. I'm looking forward to see the results...

bzier commented 7 years ago

Just wanted to provide an update on this. I have made really good progress and am getting close to putting something out there. I just have a few more things I want to clean up and organize a bit before I make it available. I'll may be able get around to it this weekend. I haven't forgotten, and didn't want to leave you guys hanging, but I usually only get a chance to work on this stuff over the weekend.

bzier commented 7 years ago

When you guys have some time, I just published what I've got so far. Still plenty of work to do, but it is a start: https://github.com/bzier/gym-mupen64plus

@kevinhughes27, I added your copyright to the LICENSE.txt file and a shout-out to you in the README with a link back to this repository. Please let me know if you'd like to see any more/different attribution.

kevinhughes27 commented 7 years ago

Looks great! this is so cool :smiley:

Let me know when you're ready for a release and I'll tweet about it - hopefully tap into some of the original TensorKart viralness.

bzier commented 7 years ago

Will do, thanks. I'd like to see you, or someone else, get it up and running before 'announcing' it, just to make sure I didn't leave anything important out. I also just pushed my branch adapting your Tensorflow agent to work with the gym environment. I still need to update the README in that branch to reflect the differences, but it works with the already trained model, which is pretty cool. https://github.com/bzier/TensorKart/tree/gym-agent

kevinhughes27 commented 7 years ago

Awesome! I will for sure pull it down and run it (probably not until next weekend though)

bzier commented 7 years ago

Fair enough. Just let me know when you do, and of course, if you bump into anything along the way. And whenever you get it running, you can probably close this issue. I'm not sure if @erotavlas85 wants true universe integration, or if gym is good enough. My understanding is that universe simply wraps gym envs and a few other envs (e.g. browser-based flash games) to further abstract away env details from agents. It also allows you to run envs over VNC, as well as control multiple envs with a single agent (or set of agents). I think having the emulator wrapped up in a gym environment is probably sufficient. It at least allows training RL agents, which is a step in the right direction.

erotavlasme commented 7 years ago

@bzier thank you for the effort. I'm going to take a look on it as soon as possible.

kevinhughes27 commented 7 years ago

I checked out the branch and setup gym-mupen64plus on my machine. After some fiddling I got it to run but the screenshot didn't line up with my emulator by default.

test0

I noticed you had made some changes to the screenshot code in your branch. Although if I understand correctly the gym code now handles screenshots while playing. It would be nice to be able to control where this window goes so people don't need to be aware of this. It would also be good to provide some sort of debugging or test to show where the AI is looking.

Here are a few other notes I made while playing around with this:

I found it ran a lot slower on my machine for some reason and I get the audio stuttering mentioned in #14
I think the ROMs directory should be a top level configuration or we should make a more opinionated choice about where ROMs are stored. It was confusing to have to go a few folders deep in the gym repo in order to configure where my ROM was.

Additionally the error message that is thrown if the ROM file is not found could be a lot better.

This is what I get - I think we could make this a one liner.

I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcublas.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcudnn.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcufft.so locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcuda.so.1 locally
I tensorflow/stream_executor/dso_loader.cc:128] successfully opened CUDA library libcurand.so locally
[2017-03-18 14:29:30,416] Making new env: Mario-Kart-Luigi-Raceway-v0
('ControllerHTTPServer started on port ', 8082)
Starting emulator with comand: mupen64plus --resolution 640x480 --input /home/kevin/mupen64plus/mupen64plus-input-bot/mupen64plus-input-bot.so /home/kevin/gym-mupen64plus/ROMS/marioKart64.n64
Traceback (most recent call last):
File "play.py", line 68, in <module>
  env = gym.make('Mario-Kart-Luigi-Raceway-v0')
File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 161, in make
  return registry.make(id)
File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 119, in make
  env = spec.make()
File "/usr/local/lib/python2.7/dist-packages/gym/envs/registration.py", line 86, in make
  env = cls(**self._kwargs)
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/MarioKart64/mario_kart_env.py", line 32, in __init__
  super(MarioKartEnv, self).__init__(mk_config['ROM_PATH'])
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 61, in __init__
  self.emulator_process = self._start_emulator(rom_path=rom_path)
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 161, in _start_emulator
  stderr=subprocess.STDOUT)
File "/usr/lib/python2.7/subprocess.py", line 711, in __init__
  errread, errwrite)
File "/usr/lib/python2.7/subprocess.py", line 1343, in _execute_child
  raise child_exception
OSError: [Errno 2] No such file or directory
Close called!
Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
  func(*targs, **kargs)
File "/usr/local/lib/python2.7/dist-packages/gym/utils/closer.py", line 67, in close
  closeable.close()
File "/usr/local/lib/python2.7/dist-packages/gym/core.py", line 173, in close
  self._close()
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 126, in _close
  self._kill_emulator()
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 173, in _kill_emulator
  if self.emulator_process is not None:
AttributeError: 'MarioKartLuigiRacewayEnv' object has no attribute 'emulator_process'
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.7/atexit.py", line 24, in _run_exitfuncs
  func(*targs, **kargs)
File "/usr/local/lib/python2.7/dist-packages/gym/utils/closer.py", line 67, in close
  closeable.close()
File "/usr/local/lib/python2.7/dist-packages/gym/core.py", line 173, in close
  self._close()
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 126, in _close
  self._kill_emulator()
File "/home/kevin/src/gym-mupen64plus/gym_mupen64plus/envs/mupen64plus_env.py", line 173, in _kill_emulator
  if self.emulator_process is not None:
AttributeError: 'MarioKartLuigiRacewayEnv' object has no attribute 'emulator_process'

The other main piece of configuration is the input driver. We could simplify this / remove the need for it by adding an install command for mupen64plus-input-bot and then assuming a working default in the gym.
It might be worth noting that relative paths don't work in the config files
is conda really necessary? I'd never heard of it before and the minimalist in me thinks its better to avoid extra dependencies or tools. If someone uses a tool for their environment management that should be their choice.

Overall I really like the idea of being able to run TensorKart with one command. If we can eliminate the config steps with sensible defaults I would be down to merge this :heart:!

bzier commented 7 years ago

Awesome, thank you for all of this feedback. I will take a closer look at this soon and see if there's anything I can do to incorporate some of the suggested improvements. Several of your notes were things I bumped into and thought were annoying too. I may write some of them up as issues as well to make it easier to track as they get resolved.

To answer the question regarding conda: No, it is not truly necessary, but it can be convenient. I had not heard of or used it before I started playing with the universe-starter-agent. The thing I liked about it was that it can take care of installing all of your dependencies for you and ensure you have the proper versions of everything. Of course, the same can be done with pip, however, conda can be especially useful if you have multiple projects that depend on different versions of the same package. Installing directly into your system would potentially break other projects. I noted in the README the alternative of installing directly, but now I'm thinking I may update that to mention both ways as options, still providing the conda file for those that prefer to use it.

kevinhughes27 commented 7 years ago

Sounds good! I'll try and jump on the install for the input plugin :smile:

conda definitely sounds neat. I'm used to reading things really fast so I missed the optional instructions at first - that's why I recommended to put the simplest thing first and then the more advanced way. Thats my 2c at least.

bzier commented 7 years ago

Updated the installation instructions in this commit. I provided both options more explicitly and put the simpler option first.

bzier commented 7 years ago

@kevinhughes27, I added issues for each of the bullet points in your feedback. I've also closed three of them relating to the configuration paths (setting defaults, better error messages, etc). Take a look and feel free to reopen them if you have any concerns or suggestions.

kevinhughes27 commented 7 years ago

Changes look great!

bzier commented 7 years ago

Added a few more improvements today. Performance should be a bit better now. Also, I added support for xvfb to handle running the emulator behind-the-scenes. This simplifies setup quite a bit because the emulator will match the resolution of the xvfb virtual 'screen' and essentially be run in full-screen. This means there is no need for offsets. I implemented it as a feature that can be toggled on/off in the config file, so you can still run it the old way (if necessary). I also implemented the render() call so you can still see what the AI is doing, even (especially) when the emulator is running in xvfb. At some point, I'll add some more details around this setup in the README too.

Let me know when you get the chance to try it out and if you have any issues, or if there are any new requirements. I didn't need to add anything, but I may have already had the necessary components installed.

kevinhughes27 commented 7 years ago

I haven't tried this yet so this might be a stupid question - how can you see whats happening when running with xvfb then? Obviously this lets us run on servers etc for reinforcement learning which is super cool but it would also be nice to leverage some of the advantages of this setup for people who just want to run and watch it on their machine. Thoughts?

bzier commented 7 years ago

Yeah, there are a couple easy ways to do it. The easiest is in the agent itself with the render() call. See the implementation of the render call here. I updated the gym-agent branch of my fork to do this, so it does display when running. See here.

The other way is setting up a vnc server pointed to the xvfb display and then connecting with a vnc client. This approach should work for a server scenario too. I plan to write this up as a part of the README too.

bzier commented 7 years ago

I have updated the README with some info about using XVFB (or not): https://github.com/bzier/gym-mupen64plus#xvfb Hopefully this is helpful. Feel free to let me know if there is anything that jumps out as incomplete or confusing.

kevinhughes27 commented 7 years ago

Sweet! If the gym is done do you want to open a PR for your branch on TensorKart?

bzier commented 7 years ago

Well, software is never really done, but I suppose it is as ready as it will ever be. I also just updated the README in my TensorKart branch to briefly mention that play.py now uses the gym environment. Yeah, I guess I'll go ahead and submit the PR.

kevinhughes27 / TensorKart

TensorKart and Universe openAI #12