search

NVlabs / FoundationPose

[CVPR 2024 Highlight] FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects
https://nvlabs.github.io/FoundationPose/
Other
955 stars 99 forks source link

Could not create GL context #100

Closed shzcuber closed 1 week ago

shzcuber commented 2 weeks ago

Hi! I'm in docker/Ubuntu 20.04 and I ran the following:

python bundlesdf/run_nerf.py --ref_view_dir /home/donghun/Documents/FoundationPose/ref_views_16 --dataset ycbv

And I'm getting the below error:

Warp 1.0.0 initialized:
   CUDA Toolkit 11.5, Driver 12.2
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 3070" (8 GiB, sm_86, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.0.0
bundlesdf/run_nerf.py:61: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning disappear) use `import imageio.v2 as imageio` or call `imageio.v2.imread` directly.
  rgb = imageio.imread(color_file)
[compute_scene_bounds()] compute_scene_bounds_worker start
[compute_scene_bounds()] compute_scene_bounds_worker done
[compute_scene_bounds()] merge pcd
[compute_scene_bounds()] compute_translation_scales done
translation_cvcam=[ 0.00114478  0.00392944 -0.00655522], sc_factor=11.946805208886905
[build_octree()] Octree voxel dilate_radius:1
[__init__()] level:0, vox_pts:torch.Size([1, 3]), corner_pts:torch.Size([8, 3])
[__init__()] level:1, vox_pts:torch.Size([8, 3]), corner_pts:torch.Size([27, 3])
[__init__()] level:2, vox_pts:torch.Size([64, 3]), corner_pts:torch.Size([125, 3])
[__init__()] level:3, vox_pts:torch.Size([480, 3]), corner_pts:torch.Size([693, 3])
[draw()] level:3
[draw()] level:3
level 0, resolution: 32
level 1, resolution: 39
level 2, resolution: 47
level 3, resolution: 56
level 4, resolution: 68
level 5, resolution: 81
level 6, resolution: 98
level 7, resolution: 117
level 8, resolution: 141
level 9, resolution: 169
level 10, resolution: 204
level 11, resolution: 245
level 12, resolution: 295
level 13, resolution: 354
level 14, resolution: 426
level 15, resolution: 512
GridEncoder: input_dim=3 n_levels=16 level_dim=2 resolution=32 -> 512 per_level_scale=1.2030 params=(36112368, 2) gridtype=hash align_corners=False
sc_factor 11.946805208886905
translation [ 0.00114478  0.00392944 -0.00655522]
[__init__()] denoise cloud
[__init__()] Denoising rays based on octree cloud
[__init__()] bad_mask#=0
rays torch.Size([815338, 12])
[train()] train progress 0/1001
[train_loop()] Iter: 0, valid_samples: 524288/524288, valid_rays: 2048/2048, loss: 31.9766159, rgb_loss: 6.7816586, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 19.8721867, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 5.0755720, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0995187,

[train()] train progress 100/1001
[train()] train progress 200/1001
[train()] train progress 300/1001
[train()] train progress 400/1001
[train()] train progress 500/1001
Saved checkpoints at /home/donghun/Documents/FoundationPose/ref_views_16/ob_0000001/nerf/model_latest.pth
[train_loop()] Iter: 500, valid_samples: 524030/524288, valid_rays: 2047/2048, loss: 1.6955070, rgb_loss: 0.5792584, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 0.2583808, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 0.8330671, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0225349,

[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(12984, 3), F:(25964, 3)
[train()] train progress 600/1001
[train()] train progress 700/1001
[train()] train progress 800/1001
[train()] train progress 900/1001
[train()] train progress 1000/1001
Saved checkpoints at /home/donghun/Documents/FoundationPose/ref_views_16/ob_0000001/nerf/model_latest.pth
[train_loop()] Iter: 1000, valid_samples: 523773/524288, valid_rays: 2046/2048, loss: 1.5242170, rgb_loss: 0.3446009, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 0.3535821, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 0.8060742, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0174493,

[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(12926, 3), F:(25848, 3)
[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(12926, 3), F:(25848, 3)
[<module>()] No OpenGL_accelerate module loaded: No module named 'OpenGL_accelerate'
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Traceback (most recent call last):
  File "bundlesdf/run_nerf.py", line 113, in <module>
    run_ycbv()
  File "bundlesdf/run_nerf.py", line 85, in run_ycbv
    mesh = run_one_ob(base_dir=base_dir, cfg=cfg)
  File "bundlesdf/run_nerf.py", line 73, in run_one_ob
    mesh = run_neural_object_field(cfg, K, rgbs, depths, masks, cam_in_obs, save_dir=save_dir, debug=0)
  File "bundlesdf/run_nerf.py", line 43, in run_neural_object_field
    mesh = nerf.mesh_texture_from_train_images(mesh, rgbs_raw=rgbs, tex_res=1028)
  File "/home/donghun/Documents/FoundationPose/bundlesdf/nerf_runner.py", line 1143, in mesh_texture_from_train_images
    renderer = ModelRendererOffscreen(cam_K=self.K, H=self.H, W=self.W, zfar=self.cfg['far']*self.cfg['sc_factor'])
  File "/home/donghun/Documents/FoundationPose/bundlesdf/../offscreen_renderer.py", line 47, in __init__
    self.r = pyrender.OffscreenRenderer(self.W, self.H)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/platforms/pyglet_platform.py", line 50, in init_context
    self._window = pyglet.window.Window(config=conf, visible=False,
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/xlib/__init__.py", line 138, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/__init__.py", line 538, in __init__
    context = config.create_context(gl.current_context)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 105, in create_context
    return XlibContext(self, share)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 127, in __init__
    raise gl.ContextException('Could not create GL context')
pyglet.gl.ContextException: Could not create GL context
free(): invalid pointer
Aborted (core dumped)
(my) root@donghun:/home/donghun/Documents/FoundationPose# pip install PyOpenGl_Accelerate
Collecting PyOpenGl_Accelerate
  Downloading PyOpenGL_accelerate-3.1.7-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (789 bytes)
Downloading PyOpenGL_accelerate-3.1.7-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 2.5/2.5 MB 11.2 MB/s eta 0:00:00
Installing collected packages: PyOpenGl_Accelerate
Successfully installed PyOpenGl_Accelerate-3.1.7
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
(my) root@donghun:/home/donghun/Documents/FoundationPose# python bundlesdf/run_nerf.py --ref_view_dir /home/donghun/Documents/FoundationPose/ref_views_16 --dataset ycbv
Warp 1.0.0 initialized:
   CUDA Toolkit 11.5, Driver 12.2
   Devices:
     "cpu"      : "x86_64"
     "cuda:0"   : "NVIDIA GeForce RTX 3070" (8 GiB, sm_86, mempool enabled)
   Kernel cache:
     /root/.cache/warp/1.0.0
bundlesdf/run_nerf.py:61: DeprecationWarning: Starting with ImageIO v3 the behavior of this function will switch to that of iio.v3.imread. To keep the current behavior (and make this warning disappear) use `import imageio.v2 as imageio` or call `imageio.v2.imread` directly.
  rgb = imageio.imread(color_file)
[compute_scene_bounds()] compute_scene_bounds_worker start
[compute_scene_bounds()] compute_scene_bounds_worker done
[compute_scene_bounds()] merge pcd
[compute_scene_bounds()] compute_translation_scales done
translation_cvcam=[ 0.00114478  0.00392944 -0.00655522], sc_factor=11.946805208886905
[build_octree()] Octree voxel dilate_radius:1
[__init__()] level:0, vox_pts:torch.Size([1, 3]), corner_pts:torch.Size([8, 3])
[__init__()] level:1, vox_pts:torch.Size([8, 3]), corner_pts:torch.Size([27, 3])
[__init__()] level:2, vox_pts:torch.Size([64, 3]), corner_pts:torch.Size([125, 3])
[__init__()] level:3, vox_pts:torch.Size([480, 3]), corner_pts:torch.Size([693, 3])
[draw()] level:3
[draw()] level:3
level 0, resolution: 32
level 1, resolution: 39
level 2, resolution: 47
level 3, resolution: 56
level 4, resolution: 68
level 5, resolution: 81
level 6, resolution: 98
level 7, resolution: 117
level 8, resolution: 141
level 9, resolution: 169
level 10, resolution: 204
level 11, resolution: 245
level 12, resolution: 295
level 13, resolution: 354
level 14, resolution: 426
level 15, resolution: 512
GridEncoder: input_dim=3 n_levels=16 level_dim=2 resolution=32 -> 512 per_level_scale=1.2030 params=(36112368, 2) gridtype=hash align_corners=False
sc_factor 11.946805208886905
translation [ 0.00114478  0.00392944 -0.00655522]
[__init__()] denoise cloud
[__init__()] Denoising rays based on octree cloud
[__init__()] bad_mask#=0
rays torch.Size([815338, 12])
[train()] train progress 0/1001
[train_loop()] Iter: 0, valid_samples: 524288/524288, valid_rays: 2048/2048, loss: 31.9766159, rgb_loss: 6.7816586, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 19.8721867, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 5.0755720, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0995187,

[train()] train progress 100/1001
[train()] train progress 200/1001
[train()] train progress 300/1001
[train()] train progress 400/1001
[train()] train progress 500/1001
Saved checkpoints at /home/donghun/Documents/FoundationPose/ref_views_16/ob_0000001/nerf/model_latest.pth
[train_loop()] Iter: 500, valid_samples: 524032/524288, valid_rays: 2047/2048, loss: 1.7143134, rgb_loss: 0.5991633, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 0.2567559, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 0.8336604, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0224731,

[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(13010, 3), F:(26016, 3)
[train()] train progress 600/1001
[train()] train progress 700/1001
[train()] train progress 800/1001
[train()] train progress 900/1001
[train()] train progress 1000/1001
Saved checkpoints at /home/donghun/Documents/FoundationPose/ref_views_16/ob_0000001/nerf/model_latest.pth
[train_loop()] Iter: 1000, valid_samples: 523776/524288, valid_rays: 2046/2048, loss: 1.5218006, rgb_loss: 0.3439563, rgb0_loss: 0.0000000, fs_rgb_loss: 0.0000000, depth_loss: 0.0000000, depth_loss0: 0.0000000, fs_loss: 0.3536648, point_cloud_loss: 0.0000000, point_cloud_normal_loss: 0.0000000, sdf_loss: 0.8043075, eikonal_loss: 0.0000000, variation_loss: 0.0000000, truncation(meter): 0.0100000, pose_reg: 0.0000000, reg_features: 0.0173703,

[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(12922, 3), F:(25840, 3)
[extract_mesh()] query_pts:torch.Size([175616, 3]), valid:164640
[extract_mesh()] Running Marching Cubes
[extract_mesh()] done V:(12922, 3), F:(25840, 3)
[<module>()] OpenGL_accelerate module loaded
[<module>()] Using accelerated ArrayDatatype
libGL error: No matching fbConfigs or visuals found
libGL error: failed to load driver: swrast
Traceback (most recent call last):
  File "bundlesdf/run_nerf.py", line 113, in <module>
    run_ycbv()
  File "bundlesdf/run_nerf.py", line 85, in run_ycbv
    mesh = run_one_ob(base_dir=base_dir, cfg=cfg)
  File "bundlesdf/run_nerf.py", line 73, in run_one_ob
    mesh = run_neural_object_field(cfg, K, rgbs, depths, masks, cam_in_obs, save_dir=save_dir, debug=0)
  File "bundlesdf/run_nerf.py", line 43, in run_neural_object_field
    mesh = nerf.mesh_texture_from_train_images(mesh, rgbs_raw=rgbs, tex_res=1028)
  File "/home/donghun/Documents/FoundationPose/bundlesdf/nerf_runner.py", line 1143, in mesh_texture_from_train_images
    renderer = ModelRendererOffscreen(cam_K=self.K, H=self.H, W=self.W, zfar=self.cfg['far']*self.cfg['sc_factor'])
  File "/home/donghun/Documents/FoundationPose/bundlesdf/../offscreen_renderer.py", line 47, in __init__
    self.r = pyrender.OffscreenRenderer(self.W, self.H)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/platforms/pyglet_platform.py", line 50, in init_context
    self._window = pyglet.window.Window(config=conf, visible=False,
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/xlib/__init__.py", line 138, in __init__
    super(XlibWindow, self).__init__(*args, **kwargs)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/__init__.py", line 538, in __init__
    context = config.create_context(gl.current_context)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 105, in create_context
    return XlibContext(self, share)
  File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/gl/xlib.py", line 127, in __init__
    raise gl.ContextException('Could not create GL context')
pyglet.gl.ContextException: Could not create GL context
free(): invalid pointer
Aborted (core dumped)

Any ideas why?

wenbowen123 commented 1 week ago

Hi, are you running on a server without display?

EricYang3721 commented 1 week ago

Yes, I'm running on a server without display. For the issue above, I solved it by adding PyOpenGL-accelerate in the Dockerfile. However, I run into display issues

`[()] OpenGL_accelerate module loaded

[()] Using accelerated ArrayDatatype Traceback (most recent call last): File "/work/Code/bundlesdf/run_nerf_compute_grid.py", line 128, in run_ycbv() File "/work/Code/bundlesdf/run_nerf_compute_grid.py", line 98, in run_ycbv mesh = run_one_ob(base_dir=base_dir, cfg=cfg, output_dir=output_dir) File "/work/Code/bundlesdf/run_nerf_compute_grid.py", line 85, in run_one_ob mesh = run_neural_object_field(cfg, K, rgbs, depths, masks, cam_in_obs, save_dir=save_dir, debug=0) File "/work/Code/bundlesdf/run_nerf_compute_grid.py", line 54, in run_neural_object_field mesh = nerf.mesh_texture_from_train_images(mesh, rgbs_raw=rgbs, tex_res=1028) File "/work/Code/bundlesdf/nerf_runner.py", line 1143, in mesh_texture_from_train_images renderer = ModelRendererOffscreen(cam_K=self.K, H=self.H, W=self.W, zfar=self.cfg['far']self.cfg['sc_factor']) File "/work/Code/offscreen_renderer.py", line 47, in init self.r = pyrender.OffscreenRenderer(self.W, self.H) File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 31, in init self._create() File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/offscreen.py", line 149, in _create self._platform.init_context() File "/opt/conda/envs/my/lib/python3.8/site-packages/pyrender/platforms/pyglet_platform.py", line 50, in init_context self._window = pyglet.window.Window(config=conf, visible=False, File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/xlib/init.py", line 138, in init super(XlibWindow, self).init(args, **kwargs) File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/window/init.py", line 514, in init display = pyglet.canvas.get_display() File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/canvas/init.py", line 59, in get_display return Display() File "/opt/conda/envs/my/lib/python3.8/site-packages/pyglet/canvas/xlib.py", line 88, in init raise NoSuchDisplayException(f'Cannot connect to "{name}"') pyglet.canvas.xlib.NoSuchDisplayException: Cannot connect to "None" free(): invalid pointer`

EricYang3721 commented 1 week ago

Following this https://github.com/nkolot/SPIN/issues/41#issuecomment-575761078, add os.environ['PYOPENGL_PLATFORM'] = 'egl' into offscreen_renderer.py solves the issue. The code can run in the remote server.