A2C attempt

[eubinecto]

game engine complete. it works like a charm!

from kalah_python.utils.env import KalahEnv
... from kalah_python.utils.board import Board
... from kalah_python.utils.agents import *
... b = Board()
... agent_1 = RandomAgent(board=b)
... agent_2 = RandomAgent(board=b)
... env = KalahEnv(board=b, agent_s=agent_1, agent_n=agent_2)
env.play_game()
------decide_on_action----
It is your turn:
N: 0 -- 7 7 7 7 7 7 7
S: 7 7 7 7 7 7 7 -- 0
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 0 -- 7 7 8 8 8 8 8
S: 7 7 7 7 7 0 8 -- 1
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 1 -- 8 0 8 8 8 8 8
S: 8 8 8 8 8 0 8 -- 1
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 1 -- 9 1 9 9 9 9 9
S: 8 8 8 8 8 0 0 -- 2
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 2 -- 10 2 0 9 9 9 9
S: 9 9 9 9 9 1 0 -- 2
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 2 -- 10 2 0 9 10 10 10
S: 9 0 10 10 10 2 1 -- 3
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 3 -- 0 2 0 9 10 11 11
S: 10 1 11 11 11 3 2 -- 3
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 3 -- 0 2 0 9 10 11 12
S: 10 1 11 11 11 3 0 -- 4
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 4 -- 1 0 0 9 10 11 12
S: 10 1 11 11 11 3 0 -- 4
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 4 -- 1 0 0 9 10 11 13
S: 10 1 11 11 11 0 1 -- 5
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 5 -- 2 1 1 10 11 12 0
S: 11 2 12 12 12 1 1 -- 5
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 5 -- 2 1 1 10 11 12 0
S: 11 0 13 13 12 1 1 -- 5
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 5 -- 3 0 1 10 11 12 0
S: 11 0 13 13 12 1 1 -- 5
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 5 -- 4 1 2 11 12 0 1
S: 12 0 13 0 13 2 2 -- 8
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 8 -- 4 1 2 11 12 0 0
S: 12 0 13 0 13 2 2 -- 8
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 8 -- 5 2 3 12 13 1 1
S: 13 0 0 1 14 3 3 -- 9
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 8 -- 5 2 3 12 13 2 0
S: 13 0 0 1 14 3 3 -- 9
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 8 -- 6 3 4 13 14 3 1
S: 14 1 1 2 0 4 4 -- 10
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 9 -- 7 4 5 0 14 4 2
S: 15 2 2 3 1 5 5 -- 10
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 9 -- 7 4 5 1 15 5 3
S: 15 2 2 3 1 5 0 -- 11
your side: Side.NORTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 9 -- 8 5 6 2 16 0 3
S: 15 2 2 3 1 5 0 -- 11
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 9 -- 8 5 6 2 16 0 3
S: 15 2 2 3 0 6 0 -- 11
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 10 -- 0 5 6 2 16 0 3
S: 16 3 3 4 1 7 1 -- 11
your side: Side.SOUTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 10 -- 1 6 7 3 17 1 4
S: 1 4 4 5 2 8 2 -- 13
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 11 -- 2 7 0 3 17 1 4
S: 2 5 5 6 2 8 2 -- 13
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 11 -- 2 7 0 3 17 1 4
S: 2 5 0 7 3 9 3 -- 14
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 11 -- 2 7 1 4 18 2 0
S: 2 5 0 7 3 9 3 -- 14
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 11 -- 2 7 1 4 18 2 0
S: 2 5 0 7 0 10 4 -- 15
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 12 -- 0 7 1 4 18 2 0
S: 3 5 0 7 0 10 4 -- 15
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 12 -- 0 7 1 4 19 3 1
S: 3 5 0 7 0 10 0 -- 16
your side: Side.NORTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 14 -- 1 8 2 5 1 4 2
S: 5 7 2 8 1 11 1 -- 16
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 14 -- 1 8 2 5 1 4 3
S: 5 0 3 9 2 12 2 -- 17
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 15 -- 2 0 2 5 1 4 3
S: 6 1 4 10 3 13 2 -- 17
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 15 -- 2 0 2 5 1 4 3
S: 6 1 0 11 4 14 3 -- 17
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 16 -- 0 0 2 5 1 4 3
S: 7 1 0 11 4 14 3 -- 17
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 16 -- 1 1 3 6 2 5 4
S: 7 1 0 0 5 15 4 -- 18
your side: Side.NORTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 16 -- 2 2 4 7 3 0 4
S: 7 1 0 0 5 15 4 -- 18
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 16 -- 3 3 5 8 4 1 5
S: 8 2 1 1 6 1 5 -- 19
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 17 -- 4 4 0 8 4 1 5
S: 9 3 1 1 6 1 5 -- 19
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 17 -- 4 4 0 9 5 2 6
S: 9 3 1 1 6 1 0 -- 20
your side: Side.NORTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 18 -- 5 5 1 10 0 2 6
S: 9 3 1 1 6 1 0 -- 20
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 18 -- 5 5 1 10 1 3 7
S: 9 3 1 1 0 2 1 -- 21
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 19 -- 6 0 1 10 1 3 7
S: 10 4 2 1 0 2 1 -- 21
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 19 -- 6 0 1 10 1 3 7
S: 10 4 2 1 0 0 2 -- 22
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 20 -- 7 1 2 0 1 3 7
S: 11 5 3 2 1 1 2 -- 22
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 20 -- 7 1 2 0 1 3 7
S: 11 0 4 3 2 2 3 -- 22
your side: Side.NORTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 20 -- 7 1 3 1 2 0 7
S: 11 0 4 3 2 2 3 -- 22
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 20 -- 7 1 3 1 2 1 8
S: 11 0 4 3 2 2 0 -- 23
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 20 -- 7 1 4 0 2 1 8
S: 11 0 4 3 2 2 0 -- 23
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 20 -- 0 1 4 0 2 1 8
S: 11 0 4 3 0 3 0 -- 32
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 32 -- 0 0 4 0 2 1 8
S: 11 0 4 3 0 3 0 -- 32
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 32 -- 0 0 4 0 2 1 8
S: 11 0 4 0 1 4 0 -- 41
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 33 -- 1 1 0 0 2 1 8
S: 12 0 4 0 1 4 0 -- 41
your side: Side.SOUTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 33 -- 1 1 1 1 3 2 9
S: 0 1 5 1 2 5 1 -- 42
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 33 -- 1 1 2 0 3 2 9
S: 0 1 5 1 2 5 1 -- 42
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 33 -- 1 1 2 0 3 2 9
S: 0 1 0 2 3 6 2 -- 43
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 34 -- 2 2 3 1 4 3 0
S: 1 2 0 2 3 6 2 -- 43
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 34 -- 2 2 3 1 4 3 0
S: 1 2 0 0 4 7 2 -- 43
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 35 -- 0 2 3 1 4 3 0
S: 2 2 0 0 4 7 2 -- 43
your side: Side.SOUTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 35 -- 0 2 3 1 0 3 0
S: 0 3 0 0 4 7 2 -- 47
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 35 -- 0 2 4 0 0 3 0
S: 0 3 0 0 4 7 2 -- 47
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 35 -- 0 2 4 0 0 3 1
S: 0 3 0 0 4 7 0 -- 48
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 36 -- 1 3 0 0 0 3 1
S: 1 3 0 0 4 7 0 -- 48
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 36 -- 1 3 0 0 0 3 1
S: 1 0 1 1 5 7 0 -- 48
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 37 -- 0 3 0 0 0 3 1
S: 1 0 1 1 5 7 0 -- 48
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 37 -- 0 3 0 0 0 4 2
S: 1 0 1 1 0 8 1 -- 49
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 38 -- 1 0 0 0 0 4 2
S: 2 0 1 1 0 8 1 -- 49
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 38 -- 1 1 1 1 1 5 3
S: 2 0 1 1 0 0 2 -- 50
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 38 -- 2 0 1 1 1 5 3
S: 2 0 1 1 0 0 2 -- 50
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 38 -- 2 0 0 1 1 5 3
S: 2 0 1 0 0 0 2 -- 52
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 38 -- 2 0 0 2 2 6 0
S: 2 0 1 0 0 0 2 -- 52
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 38 -- 2 0 0 0 2 6 0
S: 2 0 0 0 0 0 2 -- 55
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 39 -- 0 0 0 0 2 6 0
S: 3 0 0 0 0 0 2 -- 55
your side: Side.SOUTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 39 -- 0 0 0 0 2 6 0
S: 0 1 1 1 0 0 2 -- 55
your side: Side.NORTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 40 -- 1 1 1 1 3 0 0
S: 0 1 1 1 0 0 2 -- 55
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 40 -- 1 1 0 1 3 0 0
S: 0 1 1 0 0 0 2 -- 59
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 40 -- 2 0 0 1 3 0 0
S: 0 1 1 0 0 0 2 -- 59
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 40 -- 2 0 0 1 3 0 0
S: 0 0 2 0 0 0 2 -- 59
your side: Side.NORTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 41 -- 0 0 0 1 3 0 0
S: 1 0 2 0 0 0 2 -- 59
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 41 -- 0 0 0 1 3 0 1
S: 1 0 2 0 0 0 0 -- 60
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 44 -- 0 0 0 0 3 0 1
S: 1 0 2 0 0 0 0 -- 60
your side: Side.SOUTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 44 -- 0 0 0 0 3 0 1
S: 0 1 2 0 0 0 0 -- 60
your side: Side.NORTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 46 -- 0 0 1 1 0 0 1
S: 0 1 2 0 0 0 0 -- 60
your side: Side.SOUTH
random action: MOVE;2
------decide_on_action----
It is your turn:
N: 46 -- 0 0 1 1 0 0 1
S: 0 0 3 0 0 0 0 -- 60
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 46 -- 0 1 0 1 0 0 1
S: 0 0 3 0 0 0 0 -- 60
your side: Side.SOUTH
random action: MOVE;3
------decide_on_action----
It is your turn:
N: 46 -- 0 1 0 1 0 0 1
S: 0 0 0 1 1 1 0 -- 60
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 46 -- 1 0 0 1 0 0 1
S: 0 0 0 1 1 1 0 -- 60
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 46 -- 0 0 0 1 0 0 1
S: 0 0 0 1 1 0 0 -- 62
your side: Side.NORTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 46 -- 0 0 1 0 0 0 1
S: 0 0 0 1 1 0 0 -- 62
your side: Side.SOUTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 46 -- 0 0 1 0 0 0 1
S: 0 0 0 1 0 1 0 -- 62
your side: Side.NORTH
random action: MOVE;1
------decide_on_action----
It is your turn:
N: 48 -- 0 0 1 0 0 0 0
S: 0 0 0 1 0 1 0 -- 62
your side: Side.SOUTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 48 -- 0 0 1 0 0 0 0
S: 0 0 0 1 0 0 1 -- 62
your side: Side.NORTH
random action: MOVE;5
------decide_on_action----
It is your turn:
N: 48 -- 0 1 0 0 0 0 0
S: 0 0 0 1 0 0 1 -- 62
your side: Side.SOUTH
random action: MOVE;4
------decide_on_action----
It is your turn:
N: 48 -- 0 1 0 0 0 0 0
S: 0 0 0 0 1 0 1 -- 62
your side: Side.NORTH
random action: MOVE;6
------decide_on_action----
It is your turn:
N: 48 -- 1 0 0 0 0 0 0
S: 0 0 0 0 1 0 1 -- 62
your side: Side.SOUTH
random action: MOVE;7
------decide_on_action----
It is your turn:
N: 48 -- 1 0 0 0 0 0 0
S: 0 0 0 0 1 0 0 -- 63
your side: Side.NORTH
random action: MOVE;7
------- game is finished --------
N: 49 -- 0 0 0 0 0 0 0
S: 0 0 0 0 0 0 0 -- 64
------- game is finished --------
N: 49 -- 0 0 0 0 0 0 0
S: 0 0 0 0 0 0 0 -- 64
Ellipsis

[eubinecto]

How do I do masking for impossible actions correctly?

the problem

• tried setting the logit for impossible actions to neg_inf, but it would result in nan values.
• tried setting it to pseudo-neg_inf, but this would also occasionally result in nan values on loss.backward().

any solution?

• could there be a way I could... just ignore the impossible actions on sample()?

this is how I'm doing it

y_2 = y_1 * action_mask
        neg_inf = torch.scalar_tensor(float('-inf'))
        y_1_masked = torch.where(y_2 == 0, neg_inf, y_2)
        # https://discuss.pytorch.org/t/recommended-way-to-replace-a-partcular-value-in-a-tensor/25424
        y_3 = F.softmax(y_1_masked, dim=0)  # logits -> probability distributions.
        return y_3.clone() 

this seems to work?