How apply it in realsense?

[yulincho]

尊敬的wen，

您好！感谢您在FoundationPose项目上的辛勤工作和贡献。我希望能够使用Realsense相机进行6D姿态估计和跟踪的实验。具体来说，我有以下几个问题：

1. 如何在FoundationPose项目中集成Realsense相机？
2. 如何处理从Realsense相机获得的彩色图像和深度图像？
3. 如何提取并处理纹理信息？
4. 如何生成物体的mask，以便在姿态估计过程中使用？

非常感谢您的时间和帮助！

此致

敬礼

YulinZhou

[wenbowen123]

Hi, for getting RGBD from realsense, this is a question more of realsense. I'd suggest to check with their tech support or some public repo.

For building a textured mesh, you can try BundleSDF

For getting the mask, you can try things such as SAM.

[yulincho]

尊敬的温学长： 很感谢您之前的回复！我目前已经按照您前面的要求和建议使用realsense获得了rgb数据和depth数据，并且使用YOLOV8-seg生成了物体的mask，也获得了相机内参K，按照demo_data的数据类型，就只差mesh数据了。想再请教一些生成mesh的细节，例如您谈到生成mesh可以使用Bundlesdf，也需要rgb,depth,mask,好像还额外需要相机的pose？但是我看你bundlesdf中论文说不需要pose，所以具体是怎么回事呢，如果需要的话每一帧的pose都需要吗，这个怎么获得呢？另外run_demo.py中的模型应该怎么训练和更新呢，它是非model-based的吧。 我是华中科技大学机械学院的一名研究生，做的机器人视觉方向，想要复现并应用你的算法，并fellow你，在这个基础上改进，最终希望应用到我们的机器人上。以后还希望能多向您请教和交流，感谢！

此致

祝身体健康，事业进步

YulinZhou

------------------ 原始邮件 ------------------ 发件人: "NVlabs/FoundationPose" @.>; 发送时间: 2024年6月3日(星期一) 上午8:30 @.>; @.**@.>; 主题: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147)

Hi, for getting RGBD from realsense, this is a question more of realsense. I'd suggest to check with their tech support or some public repo.

For building a textured mesh, you can try BundleSDF

For getting the mask, you can try things such as SAM.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[wenbowen123]

Hi, if you use BundleSDF you do NOT need pose, it will compute for you. If you do the model-free which only inherits the neural object field part from BundleSDF (not the full version of BundleSDF), it requires pose from you.

[yulincho]

好的，感谢您的回复！ 所以您的意思是BundleSDF有model的话不需要Pose，而model-free的需要Pose是么？而FoundationPose工程里面里面自带的BundleSDF是非完整版的，需要pose，而如果没有pose是不是只能运行完整版的BundleSDF？

具体来讲是哪个文件呢？我看到FoundationPose里面有run-nerf.py和nerf-runner.py，这个是需要pose的版本吧？而在BundleSDF里面有run-custom.py，具体而言我该如何使用呢，如果您能提供一点指导我将感激不尽！

还有Pose是指物体相对于相机的位姿么？还是只是相机在世界坐标的位姿？这个应该怎么获得比较好？ 非常感谢您 YulinZhou

---Original--- From: "Bowen @.> Date: Sun, Jun 9, 2024 10:07 AM To: @.>; Cc: @.**@.>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147)

Hi, if you use BundleSDF you do NOT need pose, it will compute for you. If you do the model-free which only inherits the neural object field part from BundleSDF (not the full version of BundleSDF), it requires pose from you.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[wenbowen123]

The files you mentioned are correct. About how to run BundleSDF, you can follow the readme there and open issues in that repo if you run into questions.

Either the ob-to-cam or cam-to-ob pose is fine. One is just the inverse of the other. For BundleSDF, you dont need to worry about its pose (they are intermediate outputs), you can just use its mesh output to run model-based FoundationPose.

[yulincho]

好的，我懂您的意思了，有mesh的相当于model-based的，不需要pose，foundationpose中的就是model-based的版本。而如果跑model-free的则需要pose，是这样么？

我在docker中努力配了BundleSDF的环境，经过两天终于配好，但目前还存在一个小问题，run-custom中似乎不止需要第一帧的mask，有没有只需要第一帧的版本？或者我理解错了也希望您能指出！感激不尽 YulinZhou

---Original--- From: "Bowen @.> Date: Wed, Jun 12, 2024 12:27 PM To: @.>; Cc: @.**@.>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147)

The files you mentioned are correct. About how to run BundleSDF, you can follow the readme there and open issues in that repo if you run into questions.

Either the ob-to-cam or cam-to-ob pose is fine. One is just the inverse of the other. For BundleSDF, you dont need to worry about its pose (they are intermediate outputs), you can just use its mesh output to run model-based FoundationPose.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[wenbowen123]

好的，我懂您的意思了，有mesh的相当于model-based的，不需要pose，foundationpose中的就是model-based的版本。而如果跑model-free的则需要pose，是这样么？ 我在docker中努力配了BundleSDF的环境，经过两天终于配好，但目前还存在一个小问题，run-custom中似乎不止需要第一帧的mask，有没有只需要第一帧的版本？或者我理解错了也希望您能指出！感激不尽 YulinZhou … ---Original--- From: "Bowen @.> Date: Wed, Jun 12, 2024 12:27 PM To: @.>; Cc: @.**@.>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147) The files you mentioned are correct. About how to run BundleSDF, you can follow the readme there and open issues in that repo if you run into questions. Either the ob-to-cam or cam-to-ob pose is fine. One is just the inverse of the other. For BundleSDF, you dont need to worry about its pose (they are intermediate outputs), you can just use its mesh output to run model-based FoundationPose. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

1) yes 2) you can use tools such as XMem to pre-compute the per-frame mask for BundleSDF.

[yulincho]

非常感谢！我确实尝试了XMem，但是效果不是很好，下面是我的第一帧图像，第一帧mask，XMem得到的结果，运行BundleSDF得到的texture，obj文件以及tracking结果视频，能否帮我分析一下原因，非常感谢！ 另外，我在自己的笔记本上跑BunldeSDF算法显存不够，在服务器上配置环境，docker国内的源总是失效，要么就是挂载不上，docker比较难配置环境，我尝试了很多次，花了大量时间才成功，但是后面又崩了。有没有其他方式配置BunldeSDF？如果能用pip安装，那么在其他项目中用会很方便！ best wishes！

------------------ 原始邮件 ------------------ 发件人: "NVlabs/FoundationPose" @.>; 发送时间: 2024年6月26日(星期三) 下午2:56 @.>; @.**@.>; 主题: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147)

好的，我懂您的意思了，有mesh的相当于model-based的，不需要pose，foundationpose中的就是model-based的版本。而如果跑model-free的则需要pose，是这样么？ 我在docker中努力配了BundleSDF的环境，经过两天终于配好，但目前还存在一个小问题，run-custom中似乎不止需要第一帧的mask，有没有只需要第一帧的版本？或者我理解错了也希望您能指出！感激不尽 YulinZhou … ---Original--- From: "Bowen @.> Date: Wed, Jun 12, 2024 12:27 PM To: @.>; Cc: @.@.>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147) The files you mentioned are correct. About how to run BundleSDF, you can follow the readme there and open issues in that repo if you run into questions. Either the ob-to-cam or cam-to-ob pose is fine. One is just the inverse of the other. For BundleSDF, you dont need to worry about its pose (they are intermediate outputs), you can just use its mesh output to run model-based FoundationPose. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

yes

you can use tools such as XMem to pre-compute the per-frame mask for BundleSDF.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

[yulincho]

好的，我懂您的意思了，有mesh的相当于model-based的，不需要pose，foundationpose中的就是model-based的版本。而如果跑model-free的则需要pose，是这样么？ 我在docker中努力配了BundleSDF的环境，经过两天终于配好，但目前还存在一个小问题，run-custom中似乎不止需要第一帧的mask，有没有只需要第一帧的版本？或者我理解错了也希望您能指出！感激不尽 YulinZhou … ---Original--- From: "Bowen @.**> Date: Wed, Jun 12, 2024 12:27 PM To: @.**>; Cc: @.**@.**>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147) The files you mentioned are correct. About how to run BundleSDF, you can follow the readme there and open issues in that repo if you run into questions. Either the ob-to-cam or cam-to-ob pose is fine. One is just the inverse of the other. For BundleSDF, you dont need to worry about its pose (they are intermediate outputs), you can just use its mesh output to run model-based FoundationPose. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>

1. yes
2. you can use tools such as XMem to pre-compute the per-frame mask for BundleSDF.

第一帧我采用YOLOv8来分割，可以看到XMem在后续帧把手也做成了mask，算法不是很鲁棒。BundleSDF有考虑过这种情况么，该怎么办？别的图是结果

[wenbowen123]

if the segmentation error is huge (like in this one, the hand is included), then it will cause issues to BundleSDF. You might want to try some more recent segmentation methods like SAM, or others.

[yulincho]

明白了，很感谢！但是我的笔记本跑bundlesdf存在困难，因此在实际机器人应用上model-based不太方便。请问如何将model-free的foundationpose实际应用呢？具体跑哪个代码呢，需不需要自己改写？我需要的是不是仅仅是mask，rgb和depth？ best wishes！

---Original--- From: "Bowen @.> Date: Sat, Jun 29, 2024 04:57 AM To: @.>; Cc: @.**@.>; Subject: Re: [NVlabs/FoundationPose] How apply it in realsense? (Issue #147)

if the segmentation error is huge (like in this one, the hand is included), then it will cause issues to BundleSDF. You might want to try some more recent segmentation methods like SAM, or others.

— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: @.***>