search

google-deepmind / open_spiel

OpenSpiel is a collection of environments and algorithms for research in general reinforcement learning and search/planning in games.
Apache License 2.0
4.18k stars 922 forks source link

Can't evaluate mcts on tic tac toe with Julia. #530

Closed Kesanov closed 3 years ago

Kesanov commented 3 years ago

I installed openspiel with ] add OpenSpiel on julia 1.5.3 and copy pasted code from one of your test:

using OpenSpiel

UCT_C = 2.

init_bot(game, max_simulations, evaluator) = MCTSBot(game, evaluator, UCT_C, max_simulations, 5, true, 42, false, UCT, 0., 0.)

game = load_game("tic_tac_toe")
max_simulations = 100
evaluator = random_rollout_evaluator_factory(20, 42)
bot0 = init_bot(game, max_simulations, evaluator)
bot1 = init_bot(game, max_simulations, evaluator)
results = evaluate_bots(new_initial_state(game), [bot0, bot1], 42)

If I run this code I get:

LoadError: MethodError: no method matching evaluate_bots(::CxxWrap.StdLib.UniquePtrAllocated{State}, ::Array{MCTSBotAllocated,1}, ::Int64)
Closest candidates are:
  evaluate_bots(::Union{Ptr{Nothing}, CxxWrap.CxxWrapCore.CxxPtr{var"#s36"} where var"#s36"<:State}, ::Union{CxxWrap.CxxWrapCore.ConstCxxRef{var"#s36"} where var"#s36"<:CxxWrap.StdLib.StdVector{CxxWrap.CxxWrapCore.CxxPtr{Bot}}, CxxWrap.CxxWrapCore.CxxRef{var"#s35"} where var"#s35"<:CxxWrap.StdLib.StdVector{CxxWrap.CxxWrapCore.CxxPtr{Bot}}, Union{CxxWrap.CxxWrapCore.SmartPointer{T2}, T2} where T2<:CxxWrap.StdLib.StdVector{CxxWrap.CxxWrapCore.CxxPtr{Bot}}}, ::Integer) at /home/unix/.julia/packages/CxxWrap/94t40/src/CxxWrap.jl:590

Btw I am trying to rewrite my Python project in Julia to improve performance. But the Julia API is much more rough than the Python one (complex types, no comments, harder to discover methods for given type). Are there some tricks / tooling to list methods, simplify types or inspect comments / jump to C++ source definition? I would be very thankful for any advice on this.

lanctot commented 3 years ago

@findmyway can you take a look?

findmyway commented 3 years ago

Hi @Josef-Vonasek ,

Thanks for trying out the Julia API.

Is it possible to fix this error?

Yes, but it requires a little knowledge of CxxWrap. (I'll skip the details here for now) So why the test cases are passed but the same code throws an error here? Because the evaluate_bots function is not the original one in the c++ backend. I added a customized one here 😞 :

https://github.com/deepmind/open_spiel/blob/f7c562588389b75dd0076fe10c26ce1a57063d5a/open_spiel/julia/test/runtests.jl#L7-L35

Note that this method is not added in the OpenSpiel.jl package. It is for tests only.

I am also getting a little bit lost in the types. Is there some way to get more helpful type signatures? For example by truncating the prefixes CxxWrap.CxxWrapCore.

Hmm... I agree it might be difficult to understand the type signatures without knowing CxxWrap.jl. But since the whole APIs are built on CxxWrap.jl, there's nothing more we can do here.

Finally, how can I pass custom child_selection_fn in MCTSBot? I assume using ordinary Julia function won't work.

Correct! In theory, it's possible to pass in an ordinary Julia function. But that needs some extra work and I'm afraid it doesn't worth the efforts.

Btw I am trying to rewrite my Python project in Julia to improve performance.

That's great! Welcome to the Julia community.

But the Julia API is much more rough than the Python one (complex types, no comments, harder to discover methods for given type). Are there some tricks / tooling to list methods, simplify types or inspect comments / jump to C++ source definition? I would be very thankful for any advice on this.

  1. It's not fair to compare the Julia APIs with the Python APIs. The Julia APIs are mainly to provide a wrapper for the games in OpenSpiel. For the algorithms part, only some of them are provided and they are mainly for tests/benchmarks. If a new game is written in Julia, it's not possible to apply the algorithms written in C++ to it currently (I guess the same goes for the Python wrapper). However, many algorithms are (re-)implemented in Python so it is more feature-rich.
  2. It's impossible to jump to C++ source definition AFAIK. But it is possible to extract doc strings from the spiel.h with some simple regex and add them into OpenSpiel.jl. I'll give it a try later. Currently the only choice is to manually search in the source code. Note that all the names are transformed from CamelCase into camel_case in OpenSpiel.jl.

I'm not sure how you would like to use the Julia APIs of OpenSpiel. If you focus on the games only. I'd suggest you give ReinforcementLearningEnvironments.jl a try. It provides a set of APIs similar to OpenAI/gym and is more user-friendly.

Example usage:

] add ReinforcementLearningBase, ReinforcementLearningEnvironments, OpenSpiel

using ReinforcementLearningBase, ReinforcementLearningEnvironments, OpenSpiel

env = OpenSpielEnv("tic_tac_toe")
while !is_terminated(env)
    env(rand(legal_action_space(env)))
end

And then you can use most algorithms provided in ReinforcementLearningZoo.jl written in pure Julia, which covers many algorithms in OpenSpiel. (MCTS related are not added yet but it's on my todo list to integrate AlphaZero.jl .)

Let me know if anything is still unclear to you.

Kesanov commented 3 years ago

Thank you for quick response.