TypeError: 'NoneType' object is not subscriptable

[YiX98]

HI! I got an TypeError as follows, already tried delete cache pkl file but still got this error. My configuration is python3 train_3d.py --outdir ./outputs/motorbike/ --data ShapeNetCore.v1.rendered/motorbike/img/03790512/ --camera_path /mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/camera/ --gpus 1 --batch 4 --gamma 40 --use_shapenet_split 0 --data_camera_mode shapenet_motorbike --one_3d_generator 1 --dmtet_scale 1.0 --fp32 0

2023-03-23 18:46:05 ==> start 2023-03-23 18:46:05 ==> use shapenet dataset 2023-03-23 18:46:05 ==> use ts shapenet motorbike split all 2023-03-23 18:46:05 ==> use shapenet folder number 337 2023-03-23 18:46:05 ==> use image path: /mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/, num images: 33700 2023-03-23 18:46:05 ==> launch training 2023-03-23 18:46:05 2023-03-23 18:46:05 Training options: 2023-03-23 18:46:05 { 2023-03-23 18:46:05 "G_kwargs": { 2023-03-23 18:46:05 "class_name": "training.networks_get3d.GeneratorDMTETMesh", 2023-03-23 18:46:05 "z_dim": 512, 2023-03-23 18:46:05 "w_dim": 512, 2023-03-23 18:46:05 "mapping_kwargs": { 2023-03-23 18:46:05 "num_layers": 8 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "one_3d_generator": true, 2023-03-23 18:46:05 "n_implicit_layer": 1, 2023-03-23 18:46:05 "deformation_multiplier": 1.0, 2023-03-23 18:46:05 "use_style_mixing": true, 2023-03-23 18:46:05 "dmtet_scale": 1.0, 2023-03-23 18:46:05 "feat_channel": 16, 2023-03-23 18:46:05 "mlp_latent_channel": 32, 2023-03-23 18:46:05 "tri_plane_resolution": 256, 2023-03-23 18:46:05 "n_views": 1, 2023-03-23 18:46:05 "render_type": "neural_render", 2023-03-23 18:46:05 "use_tri_plane": true, 2023-03-23 18:46:05 "tet_res": 90, 2023-03-23 18:46:05 "geometry_type": "conv3d", 2023-03-23 18:46:05 "data_camera_mode": "shapenet_motorbike", 2023-03-23 18:46:05 "channel_base": 32768, 2023-03-23 18:46:05 "channel_max": 512, 2023-03-23 18:46:05 "fused_modconv_default": "inference_only" 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "D_kwargs": { 2023-03-23 18:46:05 "class_name": "training.networks_get3d.Discriminator", 2023-03-23 18:46:05 "block_kwargs": { 2023-03-23 18:46:05 "freeze_layers": 0 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "mapping_kwargs": {}, 2023-03-23 18:46:05 "epilogue_kwargs": { 2023-03-23 18:46:05 "mbstd_group_size": 4 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "data_camera_mode": "shapenet_motorbike", 2023-03-23 18:46:05 "add_camera_cond": true, 2023-03-23 18:46:05 "channel_base": 32768, 2023-03-23 18:46:05 "channel_max": 512, 2023-03-23 18:46:05 "architecture": "skip" 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "G_opt_kwargs": { 2023-03-23 18:46:05 "class_name": "torch.optim.Adam", 2023-03-23 18:46:05 "betas": [ 2023-03-23 18:46:05 0, 2023-03-23 18:46:05 0.99 2023-03-23 18:46:05 ], 2023-03-23 18:46:05 "eps": 1e-08, 2023-03-23 18:46:05 "lr": 0.002 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "D_opt_kwargs": { 2023-03-23 18:46:05 "class_name": "torch.optim.Adam", 2023-03-23 18:46:05 "betas": [ 2023-03-23 18:46:05 0, 2023-03-23 18:46:05 0.99 2023-03-23 18:46:05 ], 2023-03-23 18:46:05 "eps": 1e-08, 2023-03-23 18:46:05 "lr": 0.002 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "loss_kwargs": { 2023-03-23 18:46:05 "class_name": "training.loss.StyleGAN2Loss", 2023-03-23 18:46:05 "gamma_mask": 40.0, 2023-03-23 18:46:05 "r1_gamma": 40.0, 2023-03-23 18:46:05 "style_mixing_prob": 0.9, 2023-03-23 18:46:05 "pl_weight": 0.0 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "data_loader_kwargs": { 2023-03-23 18:46:05 "pin_memory": true, 2023-03-23 18:46:05 "prefetch_factor": 2, 2023-03-23 18:46:05 "num_workers": 3 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "inference_vis": false, 2023-03-23 18:46:05 "training_set_kwargs": { 2023-03-23 18:46:05 "class_name": "training.dataset.ImageFolderDataset", 2023-03-23 18:46:05 "path": "/mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/", 2023-03-23 18:46:05 "use_labels": false, 2023-03-23 18:46:05 "max_size": 33700, 2023-03-23 18:46:05 "xflip": false, 2023-03-23 18:46:05 "resolution": 1024, 2023-03-23 18:46:05 "data_camera_mode": "shapenet_motorbike", 2023-03-23 18:46:05 "add_camera_cond": true, 2023-03-23 18:46:05 "camera_path": "/mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/camera/", 2023-03-23 18:46:05 "split": "all", 2023-03-23 18:46:05 "random_seed": 0 2023-03-23 18:46:05 }, 2023-03-23 18:46:05 "resume_pretrain": null, 2023-03-23 18:46:05 "D_reg_interval": 16, 2023-03-23 18:46:05 "num_gpus": 1, 2023-03-23 18:46:05 "batch_size": 4, 2023-03-23 18:46:05 "batch_gpu": 4, 2023-03-23 18:46:05 "metrics": [ 2023-03-23 18:46:05 "fid50k" 2023-03-23 18:46:05 ], 2023-03-23 18:46:05 "total_kimg": 20000, 2023-03-23 18:46:05 "kimg_per_tick": 1, 2023-03-23 18:46:05 "image_snapshot_ticks": 50, 2023-03-23 18:46:05 "network_snapshot_ticks": 200, 2023-03-23 18:46:05 "random_seed": 0, 2023-03-23 18:46:05 "ema_kimg": 1.25, 2023-03-23 18:46:05 "G_reg_interval": 4, 2023-03-23 18:46:05 "run_dir": "/mnt/MountVolume/username/code/GET3D/outputs/motorbike/00012-stylegan2--gpus1-batch4-gamma40" 2023-03-23 18:46:05 } 2023-03-23 18:46:05 2023-03-23 18:46:05 Output directory: /mnt/MountVolume/username/code/GET3D/outputs/motorbike/00012-stylegan2--gpus1-batch4-gamma40 2023-03-23 18:46:05 Number of GPUs: 1 2023-03-23 18:46:05 Batch size: 4 images 2023-03-23 18:46:05 Training duration: 20000 kimg 2023-03-23 18:46:05 Dataset path: /mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/ 2023-03-23 18:46:05 Dataset size: 33700 images 2023-03-23 18:46:05 Dataset resolution: 1024 2023-03-23 18:46:05 Dataset labels: False 2023-03-23 18:46:05 Dataset x-flips: False 2023-03-23 18:46:05 2023-03-23 18:46:05 Creating output directory... 2023-03-23 18:46:06 Launching processes... 2023-03-23 18:47:08 Setting up PyTorch plugin "upfirdn2d_plugin"... Done. 2023-03-23 18:47:18 Setting up PyTorch plugin "bias_act_plugin"... Done. 2023-03-23 18:47:40 Setting up PyTorch plugin "filtered_lrelu_plugin"... Done. 2023-03-23 18:47:40 Loading training set... 2023-03-23 18:47:40 ==> use shapenet dataset 2023-03-23 18:47:42 ==> use ts shapenet motorbike split all 2023-03-23 18:47:42 ==> use shapenet folder number 337 2023-03-23 18:49:43 ==> use image path: /mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/, num images: 33700 2023-03-23 18:50:02 2023-03-23 18:50:02 Num images: 33700 2023-03-23 18:50:02 Image shape: [3, 1024, 1024] 2023-03-23 18:50:02 Label shape: [0] 2023-03-23 18:50:02 2023-03-23 18:50:02 Constructing networks... 2023-03-23 18:50:13 Setting up augmentation... 2023-03-23 18:50:13 Distributing across 1 GPUs... 2023-03-23 18:50:13 Setting up training phases... 2023-03-23 18:50:14 Exporting sample images... 2023-03-23 18:50:37 Initializing logs... 2023-03-23 18:50:37 Skipping tfevents export: No module named 'tensorboard' 2023-03-23 18:50:37 Training for 20000 kimg... 2023-03-23 18:50:37 2023-03-23 18:50:55 tick 0 kimg 0.0 time 3m 15s sec/tick 17.8 sec/kimg 4439.29 maintenance 177.5 2023-03-23 18:50:55 ==> start visualization 2023-03-23 18:50:55 /mnt/MountVolume/username/code/GET3D/training/networks_get3d.py:430: UserWarning: torch.range is deprecated and will be removed in a future release because its behavior is inconsistent with Python's range builtin. Instead, use torch.arange, which produces values in [start, end). 2023-03-23 18:50:55 camera_theta = torch.range(0, n_camera - 1, device=self.device).unsqueeze(dim=-1) / n_camera math.pi 2.0 2023-03-23 18:51:09 ==> saved visualization 2023-03-23 18:51:11 Evaluating metrics... 2023-03-23 18:51:11 ==> use shapenet dataset 2023-03-23 18:51:14 ==> use ts shapenet motorbike split all 2023-03-23 18:51:14 ==> use shapenet folder number 337 2023-03-23 18:52:32 ==> use image path: /mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/, num images: 33700 2023-03-23 18:52:32 ==> preparing the cache for fid scores 2023-03-23 18:52:32 {'class_name': 'training.dataset.ImageFolderDataset', 'path': '/mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/img/03790512/', 'use_labels': False, 'max_size': None, 'xflip': False, 'resolution': 1024, 'data_camera_mode': 'shapenet_motorbike', 'add_camera_cond': True, 'camera_path': '/mnt/MountVolume/username/data/ShapeNetCore.v1.rendered/motorbike/camera/', 'split': 'all', 'random_seed': 0} 2023-03-23 18:53:31 0%| | 0/527 [00:00<?, ?it/s]/opt/conda/lib/python3.9/site-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at /pytorch/c10/core/TensorImpl.h:1156.) 2023-03-23 18:53:31 return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode) 2023-03-23 19:14:22 31%|###1 | 166/527 [21:48<47:25, 7.88s/it]
2023-03-23 19:14:23 Traceback (most recent call last): 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/train_3d.py", line 330, in 2023-03-23 19:14:23 main() # pylint: disable=no-value-for-parameter 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1130, in call 2023-03-23 19:14:23 return self.main(args, kwargs) 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1055, in main 2023-03-23 19:14:23 rv = self.invoke(ctx) 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 1404, in invoke 2023-03-23 19:14:23 return ctx.invoke(self.callback, ctx.params) 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/click/core.py", line 760, in invoke 2023-03-23 19:14:23 return __callback(args, kwargs) 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/train_3d.py", line 324, in main 2023-03-23 19:14:23 launch_training(c=c, desc=desc, outdir=opts.outdir, dry_run=opts.dry_run) 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/train_3d.py", line 103, in launch_training 2023-03-23 19:14:23 subprocess_fn(rank=0, c=c, temp_dir=temp_dir) 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/train_3d.py", line 49, in subprocess_fn 2023-03-23 19:14:23 training_loop_3d.training_loop(rank=rank, c) 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/training/training_loop_3d.py", line 407, in training_loop 2023-03-23 19:14:23 result_dict = metric_main.calc_metric( 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/metrics/metric_main.py", line 52, in calc_metric 2023-03-23 19:14:23 results = _metric_dictmetric 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/metrics/metric_main.py", line 145, in fid50k 2023-03-23 19:14:23 fid = frechet_inception_distance.compute_fid(opts, max_real=50000, num_gen=50000) 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/metrics/frechet_inception_distance.py", line 30, in compute_fid 2023-03-23 19:14:23 mu_real, sigma_real = metric_utils.compute_feature_stats_for_dataset( 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/metrics/metric_utils.py", line 265, in compute_feature_stats_for_dataset 2023-03-23 19:14:23 for images, _labels, _masks in tqdm( 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/tqdm/std.py", line 1195, in iter 2023-03-23 19:14:23 for obj in iterable: 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 521, in next 2023-03-23 19:14:23 data = self._next_data() 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1183, in _next_data 2023-03-23 19:14:23 return self._process_data(data) 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/dataloader.py", line 1229, in _process_data 2023-03-23 19:14:23 data.reraise() 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/_utils.py", line 425, in reraise 2023-03-23 19:14:23 raise self.exc_type(msg) 2023-03-23 19:14:23 TypeError: Caught TypeError in DataLoader worker process 1. 2023-03-23 19:14:23 Original Traceback (most recent call last): 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/_utils/worker.py", line 287, in _worker_loop 2023-03-23 19:14:23 data = fetcher.fetch(index) 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch 2023-03-23 19:14:23 data = [self.dataset[idx] for idx in possibly_batched_index] 2023-03-23 19:14:23 File "/opt/conda/lib/python3.9/site-packages/torch/utils/data/_utils/fetch.py", line 44, in 2023-03-23 19:14:23 data = [self.dataset[idx] for idx in possibly_batched_index] 2023-03-23 19:14:23 File "/mnt/MountVolume/username/code/GET3D/training/dataset.py", line 295, in getitem 2023-03-23 19:14:23 img = ori_img[:, :, :3][..., ::-1] 2023-03-23 19:14:23 TypeError: 'NoneType' object is not subscriptable

[Bathsheba]

looks like maybe not finding the data, check your dataset paths?

[SteveJunGao]

Yes,

this looks like the path you give to the loader is not correct (or the corresponding image is corrupted), can you double check it?

[aplatyps]

I had the exact same issue, the dataset path to the loader is correct. Did you found the way to solve it? I'm quite sure there must be a bug as it always get this error at a certain iteration but can't seem to find where.

[SteveJunGao]

@aplatyps, it sounds a bit weird to me. To double confirm, is the error happen during training (you can train the model with certain iterations, then it crashes)?

Can you show me the log and the error message?

[SteveJunGao]

Close the issue for now, feel free to re-open it if the same question arises