v2.0

[Armandpl]

This is a tracking issue for the second iteration of this project.

CAD/Mechanical Assembly:

• [x] #29
• [x] #22
• [x] #30
• [x] #31
• [ ] finish designing the part to mount the slip ring to the boom arm and add it to the repo

Software:

• [x] #25
• [x] #24
• [x] use hydra to configure pendulum
• [x] add tests? robot self test like 3D printer?
• [x] separate package for gym wrapper and actual pendulum?
• [x] clean up wandb
  • [x] delete all models except for the working one (v204) started by deleting replay buffers first
  • [x] log robot config to training runs
  • [ ] delete failed and killed runs?
• [ ] #23
• [x] better env reset. previously we used hardcoded commands to reset the robot to its starting position. using a pid should be easier, cleaner and should transfer more easily should we have different robot hardware configurations. this should be part of the gym wrapper. the robot class/API should only be about the robot and should be usable outside of the gym context just wait for the pendulum to be below an angle threshold for a number of steps then reset both encoders. this way we don't have to move motor back. drawback is the pendulum isn't facing the same way every episode, but this time we're not making a video so that's alright
• [ ] control frequency wrapper: catch up delays
• [x] update pre-commit setup

Electronics:

• add a current sensor
• use more precise encoders for the motor + pendulum and evaluate their impact
• use a motor that has helical gear vs. spur gear. see if it reduces play and allows for better/smoother control

RL:

• [ ] faster training. previous iteration took 4-5h to train. can we go faster? can we train under <10 min? can we do it without giving more info/constraints.
• [x] upgrade gym to make upgrade to gymnasium easier
• [ ] offline RL?
  • datasets could be from an energy based swing up + pid?
  • use MCAP files? or replay buffers from SAC? MCAP feels like the good choice here as the replay buffer contains the transforms applied by the wrappers such as the HistoryWrapper
• [ ] measure how much of the time is spent running the policy and how much of the time is spent doing matrix multiplication while the pendulum is idle. is it possible to parallelize?? outsource matrix multiplication to a cloud machine?
• [ ] try using torque as the output of the policy (torque control on arduino??)
• [ ] try training with/without current sensor data in the observation
• [ ] remove the two different control frequencies and use a SkipFrame wrapper instead.
• [ ] #26

Control Theory:

• pid? mpc? system id?

Documentation:

• [ ] an assembly video tutorial would be nice
• [ ] doc should show the angles origings and directions
• [ ] changelog?
• [ ] atomic/concise wandb reports for RL experiments would be nice

Validate reproducibility:

• [x] Can we source the parts 1 year later?

Ressources: Offline RL:

• https://arxiv.org/abs/2005.01643

RL:

• Planning with Diffusion for Flexible Behavior Synthesis
• Soft Actor-Critic: a beginner-friendly introduction
• L5 DDPG and SAC (Foundations of Deep RL Series)

newnew:

• https://pytorch.org/rl/reference/generated/knowledge_base/DEBUGGING_RL.html
• https://github.com/miyamotok0105/unity-ml-agents/blob/master/docs/Training-PPO.md

[Armandpl]

Goals as of 7 jan: Pierre:

• broadly compare different approaches for upright control: PID, MPC, RL
• code should allow easily swapping controllers and comparing them
• code up MPC
• first pass: tune the MPC + PID on the physical robot
• second pass: make the sim closer to reality to tune the controller and transfer
• RL: sim2real

Armand:

• broadly make training faster. 5h is too slow given deepmind trains robot dogs in 4 min
• faster = wall time
• parallelize simulation
  • easy first pass: use vec env, paralelize on cpu with the current sim
  • try sim2real, maybe fine tune
• try other algos
  • now that we can use not onboard compute maybe we can use on policy algos e.g PPO
• try and remove as much code as possible
  • try and remove the velocity filter by using PPO-LSTM and having the agent figure out the filter. see PPO vs RecurrentPPO (aka PPO LSTM) on environments with masked velocity (SB3 Contrib)
• depending on sim2real transfer, make sim more accurate
  • simulate sensor noise and resolution
  • simulate motor
  • system id from logs?
• Depending on sim speed on cpu + sim2real success
  • sim in isaac gym

[Armandpl]

New list of todos:

• [ ] #40
• [x] #41
• [x] #44
• [ ] #46
• [ ] group dependencies? rl, control, dev?
• [ ] #54
• [ ] unify hardware
  • choose a single motor and motor driver
  • anything else?
• [ ] #48
• [ ] #51
• [ ] clean up todos above
• [ ] if necessary, handle the safety on arduino
• [x] order components
  • new logic level shifter - TXS0108E
  • big red button
  • [ ] Improve documentation
  • assembly video
  • update the readme to reflect current state of the project
    • new motivations
    • new features
    • include the price