Migrate to CleanRL - Githubissues

SulRash / envenc

Repository for environment encoder, an attempt at improving reinforcement learning agents' generalisability through learning how to act on universal multimodal embeddings generated by a vision-language model.

MIT License

2 stars 0 forks source link

Migrate to CleanRL #1

Closed SulRash closed 5 months ago

SulRash commented 5 months ago

Migrating to cleanrl would make my life a lot easier.

There is currently a branch for this and it's on-going

SulRash commented 5 months ago

Since cleanrl uses envpool I found out that the following preprocessing happens to the image:

Image gets grayscaled
The Atari frame is now 84,84
The frames are stacked

Need to take this into account during dev

SulRash commented 5 months ago

Slight issue... I can't figure out how to modify envpool such that each environment gets its frames processed by the VLM. To avoid this and accelerate development, I've limited the number of environments to 1 but I can see this causing a lot of issues down the line. Maybe I should switch to the non-envpool version for now?

SulRash commented 5 months ago

Reading the docs of cleanrl they have these 2 hacks they did for Atari games:

Shared Nature-CNN network for the policy and value functions ( common/policies.py#L157, common/models.py#L15-L26)
Scaling the Images to Range [0, 1] ( common/models.py#L19)

A CNN won't make sense for the embeddings and scaling the image won't make sense either so I need to disable both options.