stuck after selecting object mesh

jessiezixinWang commented 3 weeks ago

Hi, thank you for your nice work. I use the docker image provided by FoundationPose.

At first, when I run foundationpose_ros_multi.py, I got error message:

Traceback (most recent call last):
  File "foundationpose_ros_multi.py", line 377, in <module>
    main()
  File "foundationpose_ros_multi.py", line 371, in main
    node = PoseEstimationNode(new_file_paths)
  File "foundationpose_ros_multi.py", line 130, in __init__
    self.seg_model = SAM("sam2_b.pt")
  File "/opt/conda/envs/my/lib/python3.8/site-packages/ultralytics/vit/sam/model.py", line 19, in __init__
    self.model = build_sam(model)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/ultralytics/vit/sam/build.py", line 125, in build_sam
    raise FileNotFoundError(f'{ckpt} is not a supported sam model. Available models are: \n {sam_model_map.keys()}')
FileNotFoundError: sam2_b.pt is not a supported sam model. Available models are: 
 dict_keys(['sam_h.pt', 'sam_l.pt', 'sam_b.pt'])

Then I change the script from self.seg_model = SAM("sam2_b.pt") to self.seg_model = SAM("sam_b.pt") After selecting the object mesh, it stuck with the sentence "[init()] init done". May I ask how to solve this problem?

(my) root@jessie-HP-Z4-G4-Workstation:/home/jessie/FoundationPoseROS2# python foundationpose_ros_multi.py
Warp 1.0.2 initialized:
   CUDA Toolkit 11.5, Driver 12.2
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.0.2
[__init__()] self.cfg: 
 lr: 0.0001
c_in: 6
zfar: 'Infinity'
debug: null
n_view: 1
run_id: 3wy8qqex
use_BN: true
exp_name: 2024-01-11-20-02-45
n_epochs: 62
save_dir: /home/bowenw/debug/2024-01-11-20-02-45/
use_mask: false
loss_type: pairwise_valid
optimizer: adam
batch_size: 64
crop_ratio: 1.1
enable_amp: true
use_normal: false
max_num_key: null
warmup_step: -1
input_resize:
- 160
- 160
max_step_val: 1000
vis_interval: 1000
weight_decay: 0
normalize_xyz: true
resume_run_id: null
clip_grad_norm: 'Infinity'
lr_epoch_decay: 500
render_backend: nvdiffrast
train_num_pair: 5
lr_decay_epochs:
- 50
n_epochs_warmup: 1
make_pair_online: false
gradient_max_norm: 'Infinity'
max_step_per_epoch: 10000
n_rendering_workers: 1
save_epoch_interval: 100
n_dataloader_workers: 100
split_objects_across_gpus: true
ckpt_dir: /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2024-01-11-20-02-45/model_best.pth

[__init__()] self.h5_file:None
[__init__()] Using pretrained model from /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2024-01-11-20-02-45/model_best.pth
[__init__()] init done
[__init__()] welcome
[__init__()] self.cfg: 
 lr: 0.0001
c_in: 6
zfar: .inf
debug: null
w_rot: 0.1
n_view: 1
run_id: null
use_BN: true
rot_rep: axis_angle
ckpt_dir: /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
exp_name: 2023-10-28-18-33-37
save_dir: /tmp/2023-10-28-18-33-37/
loss_type: l2
optimizer: adam
trans_rep: tracknet
batch_size: 64
crop_ratio: 1.2
use_normal: false
BN_momentum: 0.1
max_num_key: null
warmup_step: -1
input_resize:
- 160
- 160
max_step_val: 1000
normal_uint8: false
vis_interval: 1000
weight_decay: 0
n_max_objects: null
normalize_xyz: true
clip_grad_norm: 'Infinity'
rot_normalizer: 0.3490658503988659
trans_normalizer:
- 0.019999999552965164
- 0.019999999552965164
- 0.05000000074505806
max_step_per_epoch: 25000
val_epoch_interval: 10
n_dataloader_workers: 60
enable_amp: true
use_mask: false

[__init__()] self.h5_file:
[__init__()] Using pretrained model from /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
[__init__()] init done

ammar-n-abbas commented 3 weeks ago

@jessiezixinWang Thank you.

Can you share the snapshot of the mesh selection window? Normally, when you click on the object it should display on the terminal a message of "Object 0 selected..."

Once you select the required object you can press enter to start tracking.

jessiezixinWang commented 3 weeks ago

ammar-n-abbas commented 3 weeks ago

@jessiezixinWang can you share the snapshot after this GUI when you press done and select a object

jessiezixinWang commented 3 weeks ago

This is the snapshot after I press done

ammar-n-abbas commented 3 weeks ago

Can you check the camera topic and change that accordingly in the code.

Sometimes it is /camera/camera/...

jessiezixinWang commented 3 weeks ago

thank you for your reply, I am new to ros and I have checked my camera topic and now it is correct.

(my) root@jessie-HP-Z4-G4-Workstation:/home/jessie/FoundationPoseROS2# ros2 topic list
/camera/aligned_depth_to_color/camera_info
/camera/aligned_depth_to_color/image_raw
/camera/color/camera_info
/camera/color/image_raw
/camera/color/metadata
/camera/depth/camera_info
/camera/depth/color/points
/camera/depth/image_rect_raw
/camera/depth/metadata
/camera/extrinsics/depth_to_color
/camera/imu
/parameter_events
/rosout
/tf_static

however, there is a new error:

(my) root@jessie-HP-Z4-G4-Workstation:/home/jessie/FoundationPoseROS2# python foundationpose_ros_multi.py
Warp 1.0.2 initialized:
   CUDA Toolkit 11.5, Driver 12.2
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 4090" (24 GiB, sm_89, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.0.2
[__init__()] self.cfg: 
 lr: 0.0001
c_in: 6
zfar: 'Infinity'
debug: null
n_view: 1
run_id: 3wy8qqex
use_BN: true
exp_name: 2024-01-11-20-02-45
n_epochs: 62
save_dir: /home/bowenw/debug/2024-01-11-20-02-45/
use_mask: false
loss_type: pairwise_valid
optimizer: adam
batch_size: 64
crop_ratio: 1.1
enable_amp: true
use_normal: false
max_num_key: null
warmup_step: -1
input_resize:
- 160
- 160
max_step_val: 1000
vis_interval: 1000
weight_decay: 0
normalize_xyz: true
resume_run_id: null
clip_grad_norm: 'Infinity'
lr_epoch_decay: 500
render_backend: nvdiffrast
train_num_pair: 5
lr_decay_epochs:
- 50
n_epochs_warmup: 1
make_pair_online: false
gradient_max_norm: 'Infinity'
max_step_per_epoch: 10000
n_rendering_workers: 1
save_epoch_interval: 100
n_dataloader_workers: 100
split_objects_across_gpus: true
ckpt_dir: /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2024-01-11-20-02-45/model_best.pth

[__init__()] self.h5_file:None
[__init__()] Using pretrained model from /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2024-01-11-20-02-45/model_best.pth
[__init__()] init done
[__init__()] welcome
[__init__()] self.cfg: 
 lr: 0.0001
c_in: 6
zfar: .inf
debug: null
w_rot: 0.1
n_view: 1
run_id: null
use_BN: true
rot_rep: axis_angle
ckpt_dir: /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
exp_name: 2023-10-28-18-33-37
save_dir: /tmp/2023-10-28-18-33-37/
loss_type: l2
optimizer: adam
trans_rep: tracknet
batch_size: 64
crop_ratio: 1.2
use_normal: false
BN_momentum: 0.1
max_num_key: null
warmup_step: -1
input_resize:
- 160
- 160
max_step_val: 1000
normal_uint8: false
vis_interval: 1000
weight_decay: 0
n_max_objects: null
normalize_xyz: true
clip_grad_norm: 'Infinity'
rot_normalizer: 0.3490658503988659
trans_normalizer:
- 0.019999999552965164
- 0.019999999552965164
- 0.05000000074505806
max_step_per_epoch: 25000
val_epoch_interval: 10
n_dataloader_workers: 60
enable_amp: true
use_mask: false

[__init__()] self.h5_file:
[__init__()] Using pretrained model from /home/jessie/FoundationPoseROS2/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
[__init__()] init done
Traceback (most recent call last):
  File "foundationpose_ros_multi.py", line 378, in <module>
    main()
  File "foundationpose_ros_multi.py", line 373, in main
    rclpy.spin(node)
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/__init__.py", line 191, in spin
    executor.spin_once()
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 719, in spin_once
    raise handler.exception()
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/task.py", line 239, in __call__
    self._handler.send(None)
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 429, in handler
    await call_coroutine(entity, arg)
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 354, in _execute_subscription
    await await_or_execute(sub.callback, msg)
  File "/opt/ros/foxy/lib/python3.8/site-packages/rclpy/executors.py", line 118, in await_or_execute
    return callback(*args)
  File "foundationpose_ros_multi.py", line 144, in image_callback
    self.color_image = self.bridge.imgmsg_to_cv2(msg, "rgb8")
  File "/opt/ros/foxy/lib/python3.8/site-packages/cv_bridge/core.py", line 169, in imgmsg_to_cv2
    dtype, n_channels = self.encoding_to_dtype_with_channels(img_msg.encoding)
  File "/opt/ros/foxy/lib/python3.8/site-packages/cv_bridge/core.py", line 104, in encoding_to_dtype_with_channels
    return self.cvtype2_to_dtype_with_channels(self.encoding_to_cvtype2(encoding))
  File "/opt/ros/foxy/lib/python3.8/site-packages/cv_bridge/core.py", line 96, in encoding_to_cvtype2
    from cv_bridge.boost.cv_bridge_boost import getCvType
ImportError: /usr/lib/x86_64-linux-gnu/libp11-kit.so.0: undefined symbol: ffi_type_pointer, version LIBFFI_BASE_7.0

I would be greateful if you could give me some suggestions on how to deal with this problem. thank you.

ammar-n-abbas commented 3 weeks ago

It seems more like a dependency issue, @jainaayush2006 have you encountered this?

mrtnbm commented 1 week ago

@jessiezixinWang Did you make sure to run all commands of conda_build_all.sh inside a new Conda environment with the right Python version installed for either ROS2 Humble (Python 3.10) or Foxy (3.8)?

ammar-n-abbas / FoundationPoseROS2

stuck after selecting object mesh #1