Open Sjey-Lyn opened 11 months ago
Hi, thank you for your interest in our work! Yes, it is possible to optimize without dynamic models. If the action is only pick and place, you can assume the dynamics model is just a 3D rigid transformation. For your second question, yes it is correct.
Thank you for your answer, is it convenient to provide the code for the robotic arm simulation?
I may consider that if many people want it, but it may take a while. (pls leave a thumbup if you want this feature).
Okay, thanks.
I will leave this issue open so that people could comment if they want this feature.
I have another question about how to choose a reference camera to project 3D keypoints into 2D images?
Currently, we are manually setting the reference camera pose. But it is possible to do it automatically.
Thank you for your excellent work! May I ask how to train my own model?
Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.
Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.
Thank you for your timely reply! What are the GPU requirements for running this model? And how can I get the robotic simulation in the paper?
We use Nvidia 3090 to run the model. For the simulation. I may need to find some time to organize the code. We use OmniGibson for simulation
Get Outlook for iOShttps://aka.ms/o0ukef
From: Gloryseven @.> Sent: Thursday, October 26, 2023 1:14:46 AM To: WangYixuan12/d3fields @.> Cc: Wang, Yixuan @.>; State change @.> Subject: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3)
Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.
Thank you for your timely reply! What are the GPU requirements for running this model? And how can I get the robotic simulation in the paper?
— Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/WangYixuan12/d3fields/issues/3*issuecomment-1780476724__;Iw!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPkmcbKmKzg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHWV3D5G7SLAS3RVFPAZ2PDYBH5VNAVCNFSM6AAAAAA6ECQZUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGQ3TMNZSGQ__;!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPknX4UGVrA$. You are receiving this because you modified the open/close state.Message ID: @.***>
May I ask where these datasets come from? How do I create my own dataset?
I collect the dataset on my own using four RGBD cameras. You could use RGBD cameras and create data using a similar file structure.
We use Nvidia 3090 to run the model. For the simulation. I may need to find some time to organize the code. We use OmniGibson for simulation Get Outlook for iOShttps://aka.ms/o0ukef … ____ From: Gloryseven @.> Sent: Thursday, October 26, 2023 1:14:46 AM To: WangYixuan12/d3fields @.> Cc: Wang, Yixuan @.>; State change @.> Subject: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3) Hi, actually our work does not need to train the model. You only need the off-shelf foundation models. Thank you for your timely reply! What are the GPU requirements for running this model? And how can I get the robotic simulation in the paper? — Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/WangYixuan12/d3fields/issues/3*issuecomment-1780476724__;Iw!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPkmcbKmKzg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHWV3D5G7SLAS3RVFPAZ2PDYBH5VNAVCNFSM6AAAAAA6ECQZUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGQ3TMNZSGQ__;!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPknX4UGVrA$. You are receiving this because you modified the open/close state.Message ID: @.***>
We use Nvidia 3090 to run the model. For the simulation. I may need to find some time to organize the code. We use OmniGibson for simulation Get Outlook for iOShttps://aka.ms/o0ukef … ____ From: Gloryseven @.> Sent: Thursday, October 26, 2023 1:14:46 AM To: WangYixuan12/d3fields @.> Cc: Wang, Yixuan @.>; State change @.> Subject: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3) Hi, actually our work does not need to train the model. You only need the off-shelf foundation models. Thank you for your timely reply! What are the GPU requirements for running this model? And how can I get the robotic simulation in the paper? — Reply to this email directly, view it on GitHubhttps://urldefense.com/v3/__https://github.com/WangYixuan12/d3fields/issues/3*issuecomment-1780476724__;Iw!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPkmcbKmKzg$, or unsubscribehttps://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/AHWV3D5G7SLAS3RVFPAZ2PDYBH5VNAVCNFSM6AAAAAA6ECQZUSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBQGQ3TMNZSGQ__;!!DZ3fjg!__R3UdWfHVnF3yC0eCtCy0aVdIeLMtoIHf3cWoAzMXqIdk9jwV9J0VCiGMlEOteIIOy6UjTW-vXl1Hxw44kTPknX4UGVrA$. You are receiving this because you modified the open/close state.Message ID: @.***>
Hello! I'm very interested in this work. I want ask some question:
Thank you for your reply! If I use simulation robot experiments, will the dataset (RGBD images) be captured and generated by the OmniGibson simulation platform?
该邮件从移动设备发送
------------------ 原始邮件 ------------------ 发件人: "WangYixuan12/d3fields" @.>; 发送时间: 2023年11月10日(星期五) 上午10:43 @.>; @.**@.>; 主题: Re: [WangYixuan12/d3fields] Confusing about optimize (Issue #3)
Yes, the sim code corresponds to the planning part of the paper. But we also have real robot experiments.
Yes (+ planning code for real-world experiments)
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you commented.Message ID: @.***>
Yep
Hi, actually our work does not need to train the model. You only need the off-shelf foundation models.
What about the pretrained model in the readme 'bash scripts/download_ckpts.sh'. Does it mean the Optimization process in the OmniGibson? Is the pretrained model conducted by OmniGibson?
Omnigibson is a simulation platform, while the pre-trained model is to construct our representation
What do you see as the limitations of your work?
I think the control part could be more advanced.
How many sets of matching points are needed to calculate the transformation matrix?
More matching points lead to a more stable transformation matrix. A typical choice is 100 points
the knife blade is not match the goal image
Since DINOv2 cannot distinguish the sides of blades, it is expected that these two cannot match
I found the similar using DINO to find correspond feature in this repo.
but he need the goal image's depth to find the transformation. When we use the goal image using AI to generate image, it does have depth information, what is you method?
I just find in the code compare two images difference, https://github.com/WangYixuan12/d3fields/blob/5158f48ac6314bd9fbfea532b4c0e40a11493c17/fusion.py#L1729 , but i have no idea how to implement to get the transformation? Do you use the learned MPC to directly get the action?
https://gist.github.com/normandipalo/fbc21f23606fbe3d407e22c363cb134e
Actually, our method does not need a depth image to obtain the goal image. We assume that there is a floating reference camera in the workspace. The projected 2D image will be compared with goal image without the need for depth image
Thank you for your excellent work! I am confused about one part, Is it feasible to optimize without dynamic model? And whether the cost function can be interpreted as the pixel difference between key points?