Model inference on the real arm

[Soha18]

Hi Thanks for this repo, I have a model trained on robosuite it performs perfectly there. But when trying to deploy it on the real robot the movement was realy bad. As a test I recorded the observations for a whole episode in the simulator then just read it and inputed it to the model, the outputted actions are sent to the controller through deoxys as explained in the example. Here the performance is just similar to the simulator smooth and successfull. But when collecting the observations from the real arm via libfranka msgs and camera the problem comes out. Please consider that I adjusted all the values of eef pose similar to robosuite ( the eef rotation in robosuite is the hand rotation which is eef rotated by -90 degrees. And also all values are in world frame not the robot frame. I tried also using interpolators but did not solve the problem. I'm using realsense webcam and Aruco marker detector from cv2 Would you please give any idea that may help. Also how to make use frm the file osc_benchmark.py as I am using osc controller

Thanks in advance ..

[zhuyifengzju]

Based on my understanding of your message, is it about sim-to-real transfer? If so, please note that Deoxys is designed for algorithmic transfer, but not really sim-to-real policy transfer --- the distribution gap is inherently large and cannot be solved easily unless you've pre-trained a robust model that can really operate in systems with different physical parameters.