Closed GergelyMagyar closed 6 months ago
There are 2 directions we can go
Pros:
Cons:
Pros:
Cons:
After experimenting a bit with the codebase I managed to modify stable baselines so that we can have observations with changing sizes. This was non-trivial, since originally the observations were stored as large pytorch tensors or numpy arrays, with extra dimensions for the steps and environments. But if we want to save an observation vector (or matrix, if we want spatial data information in the future) for every factory and unit, we run into the problem that (if let's say we have a 100 features) sometimes we have a 4 entities and have a (4, 100) matrix, and sometimes we have 13 entities and a (13, 100) matrix. And the difference can happen in 1 timestep too, since we are running multiple environments simultaneously to speed up training.
We decided to not pursue direction (2.), since this would be a computational nightmare. We would have to keep track of the trajectory of every entity separately. It is much easier to treat 1 timestep as a single action, by summing up the log probabilities of the different entity actions. We saw other teams from previous seasons doing the thing with the probabilities in their implementations, so this approach should work.
Another important thing is that with this setup we can have both local (entity-level) and global observations. It is probably best to keep the actor and critic nets separate. The actor net should work with local information and the value net with global information. The motivation for this is simply the fact that is would be hard to deal with the changing size of local information when we want to create 1 value for the whole. Maybe we could do it by averaging a value output for each entity, or something similar, but for now it's easier to keep them separate.
With a very few simple observations and aggressive masking the agent managed to learn to mine ice. The next goal is to learn more complex behaviors.
Update: Transfer action is working now. Units can mine ice and then give it to a factory. Unfortunately right now this doesn't happen frequently, this could be because a couple of reasons.
A bit of investigation is needed.