fishbotics
commented
2 weeks ago To be honest, I wrote this code in 2022 and as I recall, the Isaac Sim/Gym split was different at that time. If I remember correctly, you're right and Lula was only supported in Isaac Sim. It's possible that that expert is in Isaac Gym now, but I haven't investigated.
If you're building off of this work, I'd recommend you check out our recent follow-up called Avoid Everything (avoid-everything.github.io), which we're showing at CoRL this year. That codebase does not rely on any heavyweight simulator because the expert pipeline uses FrankaAITStarPlanner (as you described). In that paper, we use a newer version of that planner that has some fast collision checking methods I wrote in Numba (although these collision checking methods only work with primitives). If you want to use mesh-based collision checking, you'd probably want to integrate your own collision checker.
To give a quick summary of that paper: after a lot of investigation, the issue we found with the global planner is that it tends to move extremely close to obstacles which makes the learning problem very challenging (if the robot can be nearly touching obstacles, the policy struggles to learn collision avoidant behavior). This is why the Hybrid (Lula) planner worked so much better in MPiNets. To get around this, we added a fine tuning step that tries to find failure points in the pre-trained policy and explicitly correct them (similar to hard negative mining).
If you're trying to learn a collision avoidant policy, I think this technique will really help. I haven't tried it with other planners, but I suspect it'll help improve collision avoidance with whatever expert you want to use.
Would it be possible to explain why the repository is built upon Isaac Sim and not Isaac Gym? Is it because Lula is only supported in Isaac Sim? Could we use
FrankaAITStarPlannerand therobofinpackages with Isaac Gym instead?Thank you so much!