running the inference on a single image or video of a custom object

[monajalal]

Hi @wenbowen123 I checked the README.md and looked through issues but couldn't find an instruction for running the demo on a custom object of my own either in a single RGB(or RGBD) photo or a video. Could you please clarify on that?

I understand that mustard and driller are both famous objects in 6D object pose estimation. Could you please guide how to run this for a custom object that has not been present in any of famous 6D object pose estimation datasets?

Thanks a lot.

[wenbowen123]

First of all, the driller is a custom one, not the common one you saw.

run_demo.py exactly provides an example. Did you have issue mimicking that?

[monajalal]

Unfortunately, I got very bad results. I will share with you via GDrive tomorrow.

I initially used the object mesh in ply and results were bad. Decided to convert the mesh in Blender from ply to obj and now I get error.

[__init__()] self.h5_file:
[__init__()] Using pretrained model from /home/mona/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
[__init__()] init done
[reset_object()] self.diameter:724.4480657714533, vox_size:36.222403288572664
[reset_object()] self.pts:torch.Size([105, 3])
Traceback (most recent call last):
  File "run_demo.py", line 43, in <module>
    est = FoundationPose(model_pts=mesh.vertices, model_normals=mesh.vertex_normals, mesh=mesh, scorer=scorer, refiner=refiner, debug_dir=debug_dir, debug=debug, glctx=glctx)
  File "/home/mona/FoundationPose/estimater.py", line 26, in __init__
    self.reset_object(model_pts, model_normals, symmetry_tfs=symmetry_tfs, mesh=mesh)
  File "/home/mona/FoundationPose/estimater.py", line 71, in reset_object
    self.mesh_tensors = make_mesh_tensors(self.mesh)
  File "/home/mona/FoundationPose/Utils.py", line 107, in make_mesh_tensors
    img = np.array(mesh.visual.material.image.convert('RGB'))
AttributeError: 'NoneType' object has no attribute 'convert'
(my) root@DOS:/home/mona/FoundationPose# vi run_demo.py 

[abhishekmonogram]

Unfortunately, I got very bad results. I will share with you via GDrive tomorrow.

I initially used the object mesh in ply and results were bad. Decided to convert the mesh in Blender from ply to obj and now I get error.

[__init__()] self.h5_file:
[__init__()] Using pretrained model from /home/mona/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
[__init__()] init done
[reset_object()] self.diameter:724.4480657714533, vox_size:36.222403288572664
[reset_object()] self.pts:torch.Size([105, 3])
Traceback (most recent call last):
  File "run_demo.py", line 43, in <module>
    est = FoundationPose(model_pts=mesh.vertices, model_normals=mesh.vertex_normals, mesh=mesh, scorer=scorer, refiner=refiner, debug_dir=debug_dir, debug=debug, glctx=glctx)
  File "/home/mona/FoundationPose/estimater.py", line 26, in __init__
    self.reset_object(model_pts, model_normals, symmetry_tfs=symmetry_tfs, mesh=mesh)
  File "/home/mona/FoundationPose/estimater.py", line 71, in reset_object
    self.mesh_tensors = make_mesh_tensors(self.mesh)
  File "/home/mona/FoundationPose/Utils.py", line 107, in make_mesh_tensors
    img = np.array(mesh.visual.material.image.convert('RGB'))
AttributeError: 'NoneType' object has no attribute 'convert'
(my) root@DOS:/home/mona/FoundationPose# vi run_demo.py 

Make sure the obj file you have also has a texture file or corresponding mtl file. A simple color would do. Also change the line where you load the mesh to include the argument force='mesh' as shown below

mesh = trimesh.load(args.mesh_file, force='mesh')

See if this helps

[monajalal]

@abhishekmonogram

Thank you I colored my mesh (even added a texture). However, I get same error

max_step_per_epoch: 25000
val_epoch_interval: 10
n_dataloader_workers: 60
enable_amp: true
use_mask: false

[__init__()] self.h5_file:
[__init__()] Using pretrained model from /home/mona/FoundationPose/learning/training/../../weights/2023-10-28-18-33-37/model_best.pth
[__init__()] init done
[reset_object()] self.diameter:724.4480657714533, vox_size:36.222403288572664
[reset_object()] self.pts:torch.Size([105, 3])
Traceback (most recent call last):
  File "run_demo.py", line 43, in <module>
    est = FoundationPose(model_pts=mesh.vertices, model_normals=mesh.vertex_normals, mesh=mesh, scorer=scorer, refiner=refiner, debug_dir=debug_dir, debug=debug, glctx=glctx)
  File "/home/mona/FoundationPose/estimater.py", line 26, in __init__
    self.reset_object(model_pts, model_normals, symmetry_tfs=symmetry_tfs, mesh=mesh)
  File "/home/mona/FoundationPose/estimater.py", line 71, in reset_object
    self.mesh_tensors = make_mesh_tensors(self.mesh)
  File "/home/mona/FoundationPose/Utils.py", line 116, in make_mesh_tensors
    uv = torch.as_tensor(mesh.visual.uv, device=device, dtype=torch.float)
TypeError: must be real number, not NoneType

I have the followings:

textured_simple.mtl  textured_simple.obj  textured_simple.ply

and this error happens when I use textured_simple.ply

Here is mtl file. Do you think something is wrong with it?

mesh# cat textured_simple.mtl 
# Blender MTL File: 'None'
# Material Count: 1

newmtl Material.001
Ns 225.000000
Ka 1.000000 1.000000 1.000000
Kd 0.303482 0.072549 0.020460
Ks 0.500000 0.500000 0.500000
Ke 0.000000 0.000000 0.000000
Ni 1.450000
d 1.000000
illum 2
map_Kd /home/mona/Downloads/pandav-tank-88-KbSZbuEc-unsplash.jpg

Here is the texture file

Here is the beginning of my obj file:

# cat textured_simple.obj 
# Blender v3.0.1 OBJ File: ''
# www.blender.org
mtllib textured_simple.mtl
o pallet
v -302.500000 20.000000 -190.000000
v -302.500000 20.000000 -147.000000
v 0.000000 20.000000 -190.000000
v 302.500000 20.000000 -190.000000
v 0.000000 20.000000 -147.000000
v 302.500000 20.000000 -147.000000
v -302.500000 0.000000 -190.000000
v 0.000000 0.000000 -190.000000
v -302.500000 0.000000 -147.000000
v -302.500000 0.000000 -104.000000
v 0.000000 0.000000 -147.000000
v 0.000000 0.000000 -104.000000
v 302.500000 20.000000 -190.000000
v 302.500000 20.000000 -147.000000
v 302.500000 10.000000 -190.000000
v 302.500000 0.000000 -190.000000
v 302.500000 10.000000 -147.000000
v 302.500000 0.000000 -147.000000
v 302.500000 20.000000 -104.000000
v 0.000000 20.000000 -104.000000
v 302.500000 10.000000 -104.000000
v 302.500000 0.000000 -104.000000

[abhishekmonogram]

@monajalal Can you share the entire dataset? I can check and let you know

[monajalal]

@abhishekmonogram could you please share your email?

[monajalal]

The issue was my CAD model needed to be scaled down by a factor of 1000. Thanks a lot @abhishekmonogram for your guidance and help.