Does OpenVLA baseline work in the simulated environments?

mbodiai / embodied-agents

Seamlessly integrate state-of-the-art transformer models into robotics stacks

https://mbodi.ai

Apache License 2.0

165 stars 21 forks source link

Does OpenVLA baseline work in the simulated environments? #75

Open StoneT2000 opened 3 months ago

StoneT2000 commented 3 months ago

Was taking a look at the OpenVLA baseline in colab but it seems the final video is a failure. For some context I'm in the middle of porting the simpler environments to the latest ManiSkill (runs faster + easy support of batched evaluation via GPU parallel rendering). Was planning on using this repository probably for testing/evaluation (it's very clean code! and only pytorch, jax dependencies of octo eval is causing some problems)

sebbyjp commented 3 months ago

Hi sounds very cool and thank you! It wasn't able to do any of the tasks when we tried but maybe with more fine-tuning or co-training in maniskill. If your project involves running robotics transformers seamlessly in simulation environments please let me know if you're looking for contributions :)