Setting up RL code

[maxweissenbacher]

This issue is used to track progress in setting up the RL code initially

TO-DO list:

• Decide on two RL algorithms, one on-policy and one off-policy
• Implement both in torchrl
• Implement the enhancement with the trained autoencoder
• Set up logging and code to compare the augmented / un-augmented model
• Run initial baseline runs (without any augmentation)

[maxweissenbacher]

created a new branch to track this

[maxweissenbacher]

Let's use PPO for on-policy and SAC for off-policy. Can change this later if desired.

[maxweissenbacher]

I've added a working PPO code for KS now. Need to check hyperparams and add logging and evaluation. Also: need to inject the trained model from @eliseoe into the RL agent

[maxweissenbacher]

Preliminary analysis of runs: I finished a few PPO runs (no auto encoder) and it turns out that performance is very very good. I suspect this is because we use multiple parallel training environments (previously I only used one).

One finding: using dt=0.05 works significantly worse than dt=0.005. With the latter choice, we can get fast convergence even for nu=0.01!