KoehlerM173
commented
4 months ago To achieve the Batch-Solving proposed above, GEM is to be vectorized. The first in implementing said vectorization is to facilitated a running example for the PMSM (environment ID: "Finite-CC-PMSM-v0") to get an impression of the performance. In order for that to work a rather large amount of GEM has to be vectorized already. That is:
- All components of the environment have to atleast be handed over the desired batch size. These components are:
- reference generator
- supply
- converter
- motor
- load
- ode solver
- calculation of rewards
- visualization
For the extent off the vectorization the following is proposed / implemented:
Reference Generator:
For the reference generator, in a first step, it is planned to only vectorize a single reference generator according to the desired batch dimension, so that all environments within one batch get the same type of reference. For the above chosen environment therefore the wiener_process_reference_generator.py, the subepisoded_reference_generator.py, the multiple_reference_generator.py as well as the base class ReferenceGenerator in the core.py have to be modified accordingly.
Supply:
_Currently the ideal voltage supply is supported. It can be passed the desired nominal voltage, as is the case for batch size = 1. It puts out a vector of voltages according to the given number of environments. To achieve that, the base class as well as the class for the ideal voltage source have been modified (voltage_supplies.py)._ For other voltage supplies to be added, the changes made to the ideal voltage supply have to be adapted.
Converter:
For the converter similar things hold as for the supply. To this date, the base class, the FiniteTwoQuadrantConverter have been changed to set the switching pattern for each environment inside the batch. The vectorization of the output current and the FiniteB6BridgeConverter have yet to be implemented.
Motor:
The PMSM is modified, so that it can be vectorized and with that initialized with different motor parameters for each motor of the batch. The desired motor parameters have to be handed over as a dict of numpy-arrays.
Load:
There shall be a number of loads according to the batchsize. The loads will be of the same type, but there will be one for every individual motor within the batch.
Ode Solver & Rewardfunction:
The solver and the reward function are to be modified so that they can handle vectors. For the ode-solvers this should involve no extended modifications, since the used scipy solvers are abled to handle vectors by default.
visualization:
In a first step the live visualization shall be limited to a single freely choseable environment within the batch, if it at all will be included. Besides the plots for all environments shall be available after at the end of an episode.
The way a recurrent NN in most deep learning toolboxes is trained is via mini-batch training on several trajectories where the input tensor has the shape
(#trajectories, #timesteps, #features).
In order for a RNN to work with GEM, GEM should be able to process the current observation/input of several scenarios/trajectories/profiles at once, preferrably vectorized as opposed to sequential processing. Such a feature would open up GEM for the use with agents that do not retrieve observations from a memory/experience replay buffer, but rather learn directly on the go of decision making yet on multiple scenarios at once. Practically, the amount of trajectories is equal to the batch size that fits into memory (couple thousands/millions in case of <10 features).
A short look on SciPy's odeint reveals that SciPy's integrators do provide such an interface, so GEM could be updated accordingly.