architecture for extensive list of game environments

[Kallin]

I'm interested in creating a framework that would maximize the number of games that could be represented and allow for as much code re-use as possible. Gym is already providing a few classes to help with this such as Box and Discrete in the spaces package. However, I think this could be expanded to eventually support nearly any game, from simple board games such as tic-tac-toe, to chess, monopoly, citadels, poker, and even extremely complex games such as hearthstone, magic, netrunner etc.

Such a system has already been attempted with some success in the Game Description Language, and implemented in the General Game Playing Organization. Perhaps simply integrating GDL into Gym is the most sensible approach.

Regardless of the underlying representation, I wonder if such a project would be a good fit for the Gym ecosystem. Personally I think it would be great to have this community contributing AIs that work across an ever growing library of games. Additionally, it would encourage developers of proprietary gaming libraries (look at a projects like Fireplace or Jinteki ) to collaborate in a unified space , as they're all solving very similar problems.

I'm curious if the community thinks such an effort belongs inside Gym, or if it might be better off as a separate project using Gym as a dependency, or something different altogether. Any feedback would be appreciated!

[EndingCredits]

I've actually done something relatively similar to what you're proposing by (crudely) porting V-GDL (video-game description language) to gym: github.com/EndingCredits/gym_vgdl. Of course, VGDL is not a great engine if you want to create a serious game, so it's mainly just a testing ground, with some improvements over just using something like the ALE.

As for enabling plug-and play agents for many different types of game, the main problem is representing games in a unified way. Currently that format is video input, but learning from video isn't really relevant to most practical applications. Once you've settled on a standard then it's just the legwork of tying that into the game engine.

I'd love to see support for new observation types. I needed a list-style of observation for my own purposes, and had to make a very crude one myself. A List and Dict style of observation types, which can nest other observations would definitely be helpful. Perhaps there could also be a universally defined ontology. an agent that could learn from an object heirarchy would be very useful.

[Kallin]

Thanks for the feedback @EndingCredits.

That's the first first that I've heard of VGDL. I think learning from video is definitely an interesting problem in it's own right, and it looks like the openai/Universe project is setting out to solve that by rendering arbitrary programs out of a docker container over VNC.

What I'm looking into is something more akin to the 'universally defined ontology' that you allude to. There would be some definite benefits vs. video:

• speed of training would be much faster. not limited to 60fps.
• direct access to the game model. In theory an agent could intuit the game model with enough training, but this seems like overhead vs. just saying 'these are the cards in your hand' etc.
• an AI utilizing a model can explore the game-tree using MCTS or what have you. I'm imagining a model trained to play a game like 'Hearthstone', and then an AI using that model to evaluate all of it's options at any point in the game.
• niceties could be included on top, such as a generic renderer/input system, allowing humans to play against AIs in competitive games etc.

[EndingCredits]

This is something I've been looking into. As you point out there are a lot of issues with learning from video, another example being with RTS's you'd have to add some sort of camera control which complicates the agent. There's also a bunch of other advantages of having a more direct representation such as easily being able to ignore objects, and there's a large body of research on Object-Oriented Reinforcement Learning (see OO-MDPs).

As I understand it, ultimately the goal is to make a pluggable agent that works with any game (so long as the interface is set up). In terms of designing such an agent VGDL is definitely a good test-bed for looking into this problem as you get:

• Open Source, so can access anything you want.
• Forward model available (although I ripped this out of my implementation, it's a lot more fleshed out in the GVG-AI competition version).
• Unified description of the game available in a relatively compact format.
• Already has a basic ontology of sorts.
• Numerous games from the GVG-AI competiiton

The only issues are the lack of flexibility of games; you can replicate quite a lot, but I wouldn't call it anything like a universal game engine.

Then you also need to start specifying how your unified representation works. There are some tricky issues here, for example, how do you represent continuous terrain if you're doing stuff object based? And do you present the information in relative terms to a player avatar (and what if there is no avatar). It'd be great to hear more thoughts on this point.

[Kallin]

Thanks @EndingCredits , I'm going to take a closer look at OO-MDP and that GVG-AI competition. I wasn't even aware of it's existence! Could you recommend any sites that keep on top of these sorts of projects?

Regarding your question on continuous terrain, and avatar relativism (if that's a term), I think perhaps I'm looking to solve a slightly different problem. Correct me if I'm wrong, but you seem focused on video games, and while that's really interesting to me I'm looking to focus more on turn based games here with a discrete environment, ie. (most) board games, card games, etc. Basically, I'd like to have a system that could efficiently implement >90% of what's represented here.

I'd love to find out which games could be played well by a general AI, and which ones are a struggle. A generalized AI for such games could also be really useful for game designers, as they could see if there are any obvious imbalances in their game that an AI ought ought to be able to find and exploit.

[EndingCredits]

As it happens I'm working on the same thing for video-games. Obviously, there do tend to be differences between the two, but broadly it's the same problem.

I've created a gitter community to discuss this problem (although if anyone is aware of an already existent one, let me know) here: https://gitter.im/General-Game-AI/Lobby

[christopherhesse]

I think this would be best off as a project separate from the gym interface. A good example is unity ml agents https://github.com/Unity-Technologies/ml-agents which has a gym interface, but lets you build your RL environments using Unity.