How to deploy pre-trained/trained habitat_baseline to a real differential robot

[mhaboali]

❓ Questions and Help

I've trained and validated a PPO model using that example in the Readme file and I'm looking forward to deploying this model to a real differential robot that has RealSense cameras(D435 and T265) and Jetson Nano board.

Also, I found habitat/sims/pyrobot/pyrobot.py but I don't know how to use it for such an application.

[mhaboali]

I'm wondering why is it unclear how to use PyRobot to deploy the models on physical robots as it's claimed on the ReadMe file here?

I'm trying to deploy one of your pre-trained models and test it on a real robot, then I'm planning to train my model.

Please let the community know the steps of how to achieve that -if it's supported- I spent much time getting it but I didn't form complete procedure.

Thanks! @erikwijmans @abhiskk @mathfac

[dhruvbatra]

Supporting general hardware platforms is beyond the scope of Habitat. You should read our paper to understand the goal of Habitat-PyRobot Bridge, what we experimented with, and why we open-sourced it: https://abhiskk.github.io/sim2real

As a helpful feedback, you may not intend to, but your comment risks coming across as snarky. An open-source community made up of volunteers does not owe you anything.

[erikwijmans]

Just to expand on what Dhruv said, we currently support deployment to a robot through https://github.com/facebookresearch/pyrobot -- if the platform you have isn't supported by pyrobot, you can open an issue there and see if someone is willing to work with you on supporting that platform.

[mhaboali]

@dhruvbatra Ok, I'll do that.

Thank you so much for your comment and your feedback. Actually, I didn't intend to make the comment look snarky but when I read it after your comment, yeah it looks bad. I'm sorry about that. Please accept my apologies.

[mhaboali]

@erikwijmans

Actually, I'm willing to buy the LocoBot(Kobuki base) for this deployment, but it's not figured to me yet how to deploy a trained model on LocoBot using PyRobot.

I looked at PyRobot repo but the integration with Habitat AI is only a future feature. Is there any documentation about this process?

Thanks for your continuous help :)

[erikwijmans]

Once you have pyrobot setup on a locobot, you need to change the simulator type in habitat-api to "PyRobot-v0" and follow the additional instruction about the "ROS_PATH" here: https://github.com/facebookresearch/habitat-api/blob/master/habitat/sims/pyrobot/pyrobot.py#L152

CC @abhiskk

[mhaboali]

I read pyrobot.py well and I got how it works roughly, but I didn't get how and where to use a Habitat RL baseline to achieve some navigation tasks autonomously. Do you have any ideas? @erikwijmans @abhiskk

[mhaboali]

Hi, The conclusion of the sim2real paper starts with:

We introduce the Habitat-PyRobot Bridge (HaPy) library which allows for seamless deployment of visual navigation models across simulation (Habitat) and reality (LoCoBot).

And under section 8 which talks about HaPy architecture:

In this work we develop the Habitat-PyRobot bridge (HaPy) extension which allows for seamless sim2real transfer of simulation-trained models to reality. Figure 10 shows the integration of our extension with Habitat-API [1]. This integration allows us to reuse the PointNav task and RL baseline already implemented in Habitat-API. We have added our codebase which contains the implementation of HaPy to the supplemental zip. Please refer to habitat-pyrobot-bridge-source/README.md for details.

I got that HaPy was developed for my exact purpose and it would help me to deploy/reuse the Habitat-API RL baseline with LocoBot, but I searched online about it but didn't find it.

Where can I find habitat-pyrobot-bridge-source?

Thanks!

[abhiskk]

Hi @mhaboali, apologies for the late reply. Below I have laid out the steps on how to use the HaPy for your task:

1. Setup your robot and install pyrobot following instructions at the pyrobot repository: https://github.com/facebookresearch/pyrobot

2. Setup training of habitat-API RL training, for this you can follow instructions under habitat_baselines.

3. Once you have habitat-API setup I would suggest doing a walkthrough of code by stepping around inside the codebase, the things you should understand is how the simulator gets created, how are episodes defined and how RL leverages these. An important part of the code is creation of the simulator: https://github.com/facebookresearch/habitat-api/blob/master/habitat/core/env.py#L95 inside env. I would also suggest using this workflow to create the pyrobot simulation that we have. Using this workflow I would suggest you create the pyrobot simulation and execute some actions as defined here.

4. Once that is done you will have to write some logic on mapping the actions from habitat-API RL simulation env to reality env. The actions defined inside sims/pyrobot/pyrobot.py are basic and you should be able to adapt them to your needs, for eg if for your task you want to define a left, right, forward action then you can use the actions we have defined, (docstring).

Feel free to reach out to us on the habitat-dev slack channel. Would be happy to help on this.