plan

[Armandpl]

Motivations:

• I want to train my furuta pendulum FASTER in terms of wall clock time and I am tired of tuning the hyper-parameters of classic algos
• I may be interested in trying to claim the tinygrad dreamerv3 bounty
• wonder if we could remove the VelocityFilter and have the agent learn from RAW sensors measurements only? that'd be cool. seems easier to do if we have a world model

I feel like there might be an opportunity to start by ripping out stuff i don't need for this problem from the dreamer code and then building it back up? for example I don't think I need to encode the state since it is already very small? I don't need any image stuff, don't need to rescale the reward, what else can I remove at first?

[Armandpl]

maybe i should start from https://github.com/kc-ml2/SimpleDreamer and upgrade it to dreamer v3?

[Armandpl]

even before that I should take a look at hardware acceleration, what is even tinygrad, what's the different between it and torch. Few leads:

• cornell ece 5545
• All about AI Accelerators: GPU, TPU, Dataflow, Near-Memory, Optical, Neuromorphic & more (w/ Author)
• hw for dl

[Armandpl]

Why re-implement Dreamer even though there already exist implementations on GitHub:

• NM512 implementation is hard too read: no comments, no typing
• kc-ml2 is easy to read but doesn't include benchmarks and only implements dreamer v1
• electicsheep implementation is drowned in their library, can't use it on its own
• some of those use gym instead of gymnasium
• none of those also implement gSDE, which is necessary for exploration on real robots without low level controllers

[Armandpl]

drowned in their library was a bit harsh lmao implementation seems to be really clear and well commented. They do use a Fabric Module and idk what it is but whatever. seems like a nice ressource. really like that the dreamer.py file starts with the interesting stuff!!

[Armandpl]

another one: https://github.com/pytorch/rl/tree/main/examples/dreamer