chenhsuanlin / photometric-mesh-optim

Photometric Mesh Optimization for Video-Aligned 3D Object Reconstruction :globe_with_meridians: (CVPR 2019)
MIT License
208 stars 25 forks source link

failing inside meshrender library, gpu related #3

Closed bmauchly closed 5 years ago

bmauchly commented 5 years ago

I guess the meshrender library source is not part of the project.

I'm running in a Colab notebook, with versions backed out to torch 0.4 and cuda 9.0 I'm getting this message: torch.FatalError: abort... at /home/chenhsuan/adobe-scenemeshing/meshrender-forwardonly/src/gpu.c:33

command: python3 main.py --load={model} --code=5e-2 --scale=2e-2 --lr-pmo=3e-3 --noise=0.1 --video --eval

full output:

/root/photometric-mesh-optim

main.py (photometric mesh optimization)

setting configurations... H : 224 W : 224 aug_transl : None avg_frame : False batch_size : 32 batch_size_pmo : -1 category : None code : 0.05 cpu : False device : cuda:0 eval : True from_epoch : 0 gpu : 0 group : 0 imagenet_enc : False init_idx : 27 load : /root/photometric-mesh-optim/pretrained/02958343_atl25.npz log_tb : False log_visdom : False lr_decay : 1.0 lr_pmo : 0.003 lr_pretrain : 0.0001 lr_step : 100 name : debug_seed0 noise : 0.1 num_meshgrid : 5 num_points : 100 num_points_all : 2500 num_prim : 25 num_workers : 8 pointcloud_path : data/customShapeNet pretrained_dec : None rendering_path : data/rendering scale : 0.02 seed : 0 seq_path : data/sequences sfm : False size : 224x224 sphere : False sphere_densify : 3 sun360_path : data/background to_epoch : 500 to_it : 100 video : True vis_port : 8097 vis_server : http://localhost

reading list of sequences... number of sequences: 1 building AtlasNet... loading checkpoint /root/photometric-mesh-optim/pretrained/02958343_atl25.npz... ======= OPTIMIZATION START ======= loading sequence... reading RGB .npy file... reading ground-truth camera .npz file... noise -- scale: -0.1378, rot: [-0.0205,0.0343,-0.0529], trans: [0.0852,0.0939,-0.1320] error: invalid device function Traceback (most recent call last): File "main.py", line 31, in pmo.optimize(opt) File "/root/photometric-mesh-optim/model.py", line 185, in optimize loss = self.compute_loss(opt,var) File "/root/photometric-mesh-optim/model.py", line 75, in compute_loss loss.photom += self.compute_photometric_loss_batch(opt,var,idx_a,idx_b) File "/root/photometric-mesh-optim/model.py", line 96, in compute_photometric_loss_batch indexa,,,,vertices_a = render.rasterize_3D_mesh(opt,var.vertices_clone,self.faces,cam_extr_a,cam_intr) File "/root/photometric-mesh-optim/render.py", line 17, in rasterize_3D_mesh index_map,baryc_map,mask_map,inv_depth_map = Rasterize().apply(opt,B,cam_intr,face_vertices,batch_face_index) File "/root/photometric-mesh-optim/render.py", line 76, in forward meshrender.forward_cuda(cam_intr,face_vertices_trans,batch_face_index,index_map,baryc_map,inv_depth_map,lock_map) File "/usr/local/lib/python3.6/dist-packages/torch/utils/ffi/init.py", line 197, in safe_call result = torch._C._safe_call(*args, **kwargs) torch.FatalError: abort... at /home/chenhsuan/adobe-scenemeshing/meshrender-forwardonly/src/gpu.c:33

chenhsuanlin commented 5 years ago

The meshrender library source is part of the project. It's currently compiled with compute compatibility 6.1 due to some of the atomic operations that's required, so it might not be able to run smoothly if you have older GPUs. I'm working on optimizing the cuda code so it could compiled and run with lower compute compatibilities, when it's done I'll also release this part of the source code. Regarding the specific reported error, however, I don't have a very good idea of what might be the cause, since it's happening on a return statement. I'll leave this issue open for now.

chenhsuanlin commented 5 years ago

I've just released the newly compiled meshrender library as well as the C source code. If your issue still persists with this version, please let me know!

bmauchly commented 5 years ago

Thank you, it is now running through the entire eval without error, with the new library. I didn't need to build it. (and yes, with an older GPU.)

bmauchly commented 5 years ago

The new version of the meshrender library fixes this.