RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

[rohit7044]

Hello everyone, I am trying to run this code on single GPU and after following the parameters mentioned #14 , I am getting this error.

                    align: True                             [default: False]
             aspect_ratio: 1.0                           
      cache_filelist_read: False                         
     cache_filelist_write: False                         
          checkpoints_dir: ./checkpoints                 
               chunk_size: [1]                              [default: None]
   contain_dontcare_label: False                         
                crop_size: 256                           
                  dataset: example                          [default: ms1m,casia]
             dataset_mode: allface                       
             device_count: 1                                [default: 8]
          display_winsize: 256                           
             erode_kernel: 21                            
                  gpu_ids: 0,1                              [default: 0]
             heatmap_size: 2.5                              [default: 3]
                 how_many: inf                           
                init_type: xavier                        
            init_variance: 0.02                          
                  isTrain: False                            [default: None]
               label_mask: True                             [default: False]
                 label_nc: 5                             
           landmark_align: False                         
                 list_end: 10                               [default: inf]
                 list_num: 0                             
               list_start: 0                             
       load_from_opt_file: False                         
                load_size: 256                           
         max_dataset_size: 9223372036854775807           
                    model: rotatespade                      [default: rotate]
                multi_gpu: True                             [default: False]
                 nThreads: 1                             
                     name: mesh2face                     
                    names: rs_model                         [default: rs_ijba3]
                      nef: 16                            
                     netG: rotatespade                      [default: rotate]
                      ngf: 64                            
                  no_flip: True                          
     no_gaussian_landmark: True                             [default: False]
              no_instance: True                          
         no_pairing_check: False                         
                   norm_D: spectralinstance              
                   norm_E: spectralinstance              
                   norm_G: spectralsyncbatch                [default: spectralinstance]
                output_nc: 3                             
                    phase: test                          
              pitch_poses: None                          
              posesrandom: False                         
          preprocess_mode: scale_width_and_crop          
            render_thread: 1                                [default: 2]
resnet_initial_kernel_size: 7                             
       resnet_kernel_size: 3                             
          resnet_n_blocks: 9                             
      resnet_n_downsample: 4                             
              results_dir: ./results/                    
                save_path: ./results/                    
           serial_batches: True                          
                  trainer: rotate                        
              which_epoch: latest                        
                yaw_poses: [0.0, 90.0]                      [default: None]
----------------- End -------------------
dataset [AllFaceDataset] of size 8 was created
Testing gpu  [0]
Network [RotateSPADEGenerator] was created. Total number of parameters: 225.1 million. To see the architecture, do print(network).
start prefetching data...
Process Process-1:
Traceback (most recent call last):
  File "C:\ProgramData\Anaconda3\envs\thesis\lib\multiprocessing\process.py", line 315, in _bootstrap
    self.run()
  File "C:\ProgramData\Anaconda3\envs\thesis\lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\data\data_utils.py", line 146, in prefetch_data
    prefetcher = data_prefetcher(dataloader, opt, render_layer)
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\data\data_utils.py", line 99, in __init__
    self.preload()
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\data\data_utils.py", line 124, in preload
    self.next_input = get_multipose_test_input(data, self.render_layer, self.opt.yaw_poses, self.opt.pitch_poses)
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\data\data_utils.py", line 65, in get_multipose_test_input
    = render.rotate_render(data['param_path'], real_image, data['M'], yaw_pose=pose)
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\models\networks\rotate_render.py", line 103, in rotate_render
    tex_a_crop = self.get_render_from_vertices(rendered_images_erode[n], vertices_in_ori_img[n])
  File "D:\Lab Works\3D Face Reconstruction\Rotate-and-Render\models\networks\render.py", line 340, in get_render_from_vertices
    textures = img_ori[vertices_in_ori_img[1, :].round().clamp(0, h - 1).long(), \
RuntimeError: indices should be either on cpu or on the same device as the indexed tensor (cpu)

[rohit7044]

So, I thought about changing to rendered_images_erode to cuda but the results are blank images and CUDA shutdown. Here is the error

test_multipose.py:165: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  rotated_landmarks = data['rotated_landmarks'][:, :, :2].cpu().numpy().astype(np.float)
test_multipose.py:166: DeprecationWarning: `np.float` is a deprecated alias for the builtin `float`. To silence this warning, use `float` by itself. Doing this will not modify any behavior and is safe. If you specifically wanted the numpy scalar type, use `np.float64` here.
Deprecated in NumPy 1.20; for more details and guidance: https://numpy.org/devdocs/release/1.20.0-notes.html#deprecations
  rotated_landmarks_106 = data['rotated_landmarks_106'][:, :, :2].cpu().numpy().astype(np.float)
process image..../results/rs_model\example\orig\yaw_0.0_Images\Ann_Veneman_0010.jpg
processed num 1
process image..../results/rs_model\example\orig\yaw_1.57_Images\Ann_Veneman_0010.jpg
processed num 2
(************* each image time total: 71.395 *****************)
(************* each image render time: 22.617 *****************)
C:\ProgramData\Anaconda3\envs\thesis\lib\site-packages\skimage\transform\_geometric.py:158: RuntimeWarning: divide by zero encountered in divide
  scale = 1.0 / src_demean.var(axis=0).sum() * (S @ d)
process image..../results/rs_model\example\orig\yaw_0.0_Images\Benjamin_Netanyahu_0005.jpg
processed num 3
process image..../results/rs_model\example\orig\yaw_1.57_Images\Benjamin_Netanyahu_0005.jpg
processed num 4
(************* each image time total: 22.912 *****************)
(************* each image render time: 28.006 *****************)
cannot convert float NaN to integer
[W C:\actions-runner\_work\pytorch\pytorch\builder\windows\pytorch\torch\csrc\CudaIPCTypes.cpp:15] Producer process has been terminated before all shared CUDA tensors released. See Note [Sharing CUDA tensors]
[W CUDAGuardImpl.h:46] Warning: CUDA warning: driver shutting down (function uncheckedGetDevice)
[W CUDAGuardImpl.h:62] Warning: CUDA warning: invalid device ordinal (function uncheckedSetDevice) ```

[rohit7044]

I tried converting vertices_in_ori_img or rendered_images_erode to CPU and cuda. But unfortunately the problem still persists. Anybody faced the same problem?

[Elaine0]

I have the same issue.

[rohit7044]

I have the same issue.

I have created a blog on medium. You can follow the instructions and see if you can work it out.