Implementing Player-Zero Training (2/2)

[Klazkin]

Explain the context of the issue, what is being addressed in detail.

The goal

• [ ] Board Input normalization

• [ ] Look into weight standardization https://arxiv.org/pdf/1903.10520v1.pdf

• [ ] ConvNext https://github.com/facebookresearch/ConvNeXt

• [x] Data generation during tournament

• [x] BUG same node sub-tree has actions for different casters

• [x] Single training instance

• [x] Running training server

• [x] increase number of simulations due to large branch factor

• [x] verify that winner is relative to current unit

• [x] policy outputs should be softmax'ed'

• [x] normalize the node scores, normalize policy outputs

• normalize value unnecessary?

• [x] add mask to policy outputs

• [x] Implement removal of old files

• [ ] Configure optimizer

• [x] Configure Regularization

• [x] Configure loading of old model weights for training

• [ ] There is probably no reason to constantly save and load models if the training server is running, but only save for checkpoints

• [x] #84

• BLOODDRAWING child scores do not account for the possible random outcomes - currently score is calculated only once for one of the two outcomes, but realistically it should be done twice (or more, depending on number of branches).

Time tracking

Time Estimate: undefined Time spent: 7 hours 48 minutes

Resources

https://realpython.com/python-sockets/ https://docs.godotengine.org/en/stable/classes/class_packedbytearray.html https://docs.godotengine.org/en/stable/classes/class_streampeertcp.html#class-streampeertcp https://docs.godotengine.org/en/stable/classes/class_streampeer.html#class-streampeer-method-get-string https://keras.io/guides/serialization_and_saving/ https://rl-vs.github.io/rlvs2021/class-material/regularized_mdp/Regularization_RL_RLVS.pdf

[Klazkin]

Investigate

Simulating generated/sim_0_745970196_1712312526.077.txt...
New unit: 17,9,-1,2,1
[13, 27, 3, 14, 9, 8, 4, 7]
[13, 14, 9, 8, 27]
New unit: 13,8,0,-1,-2
[5, 13, 27, 3, 23, 2, 9, 6]
[13, 9, 6, 27]
Vector2i(10, 5)
Vector2i(3, 8)
Not a proper blooddrawing node

issue: casting Blooddrawing when there are no actions that can be replenished. (later in TAB it is expected to search for a branch)

[Klazkin]

for normal node:
    Node -> predict -> get policy get value
    expand children
        child score is value
        child policy added

for BLOODDRAWING node:
    Node -> predict -> get policy get value
    (policy is ignored)
    expand random branches
        branch score is 
        branch policy is 0 ```