Tracking process issue

[EvdoTheo]

Hello, I'm trying to create a custom model from an image and although the preprocessing task goes smooth, when I use this command as mentioned in #3 python scripts/inference/rec.py --model_type nphm --exp_name pretrained_mononphm --ckpt 2500 --seq_name 00010 --no-intrinsics_provided --downsample_factor 0.33 I get this error.

OUTPUT

{ "decoder": { "decoder_nloc": 65, "ex": { "hidden_dim": 400, "lat_dim_ex": 100, "lat_dim_id": 16, "mode": "compress", "nhyper": 2, "nlayers": 6 }, "id": { "blend_std": 3.75, "gnn": { "hidden_dim_app": 200, "hidden_dim_geo": 200, "nfreq_bands_app": 0, "nfreq_bands_geo": 0, "nlayers_app": 4, "nlayers_geo": 4 }, "head": { "hidden_dim_app": 128, "hidden_dim_geo": 128, "lat_dim_app_aggr": 256, "lat_dim_geo_aggr": 256, "nlayers_app": 2, "nlayers_geo": 2 }, "lat_dim_glob": 64, "lat_dim_glob_app": 64, "lat_dim_loc_app": 32, "lat_dim_loc_geo": 32, "nloc": 65, "nneigh": 12, "nsymm_pairs": 30 } }, "training": { "batch_size": 16, "ckpt_interval": 500, "grad_clip": 1.0, "grad_clip_lat": 1.0, "lambdas": { "anchors": 7.5, "color": 1.0, "corresp": 1000.0, "eikonal": 0.15, "hyper": 0.01, "loss_neutral_zero": 0.001, "loss_reg_zero": 0.0025, "lpips": 0.1, "middle_dist": 0.0, "middle_dist_app": 0, "normals": 0.3, "reg_app": 0.005, "reg_app_var": 3e-07, "reg_expr": 0.05, "reg_expr_var": 3e-08, "reg_shape": 0.01, "reg_shape_var": 3e-08, "space_sdf": 0.01, "surf_sdf": 1.0, "symm_dist": 0.01, "symm_dist_app": 0.0001 }, "loss_type": "igr", "lr": 0.0005, "lr_decay_factor": 0.5, "lr_decay_factor_lat": 0.5, "lr_decay_interval": 500, "lr_decay_interval_lat": 500, "lr_lat": 0.002, "lr_lat_expr": 0.01, "mode": "shape_space", "npatches_per_batch": 1, "npoints_corresp": 250, "npoints_face": 1000, "npoints_non_face": 250, "npoints_off_surface": 250, "sigma_near": 0.01, "weight_decay": 0.0005 } } FOUND 1 GPUs ANCHORS HAVE SHAPE: torch.Size([1, 1, 65, 3]) creating DeepSDF with... lat dim 116 hidden_dim 400 Creating DeepSDF with input dim f119, hidden_dim f400 and output_dim 5 /home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3432: RuntimeWarning: Mean of empty slice. return _methods._mean(a, axis=axis, dtype=dtype, /home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/numpy/core/_methods.py:190: RuntimeWarning: invalid value encountered in double_scalars ret = ret.dtype.type(ret / rcount) USING EPOCH MULT OF: 1 Traceback (most recent call last): File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/scripts/inference/rec.py", line 194, in tyro.cli(main) File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/tyro/_cli.py", line 229, in cli return run_with_args_from_cli() File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/scripts/inference/rec.py", line 185, in main inverse_rendering(n3dmm, seq_name, _expressions, n_expr, File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/scripts/inference/rec.py", line 59, in inverse_rendering track(wrapped_net, cfg, seq_name, expressions, File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/tracking.py", line 1337, in track out_dict = tracker.idr(in_dict, condition, skip_render=False, neus_variance=variance, File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, kwargs) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/rendering.py", line 213, in forward points, net_values, steps = self.ray_tracer(sdf=lambda x: self.implicit_network(x, condition, include_color=False), File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/rendering.py", line 91, in forward min_mask_points, min_mask_dist, sampled_points, net_values, steps = self.minimal_sdf_points(num_pixels, sdf, cam_loc, ray_directions, mask_intersect, File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/rendering.py", line 155, in minimal_sdf_points mask_net_out_all.append(sdf(pnts)) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/rendering.py", line 213, in points, net_values, steps = self.ray_tracer(sdf=lambda x: self.implicit_network(x, condition, include_color=False), File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, kwargs) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/photometric_tracking/wrapper.py", line 15, in forward result = self.monoNPHM({'queries': positions}, cond=condition, skip_color=not include_color, return_grad=return_grad) File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(*args, *kwargs) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/models/neural3dmm.py", line 130, in forward pred = self.id_model(in_dict) File "/home/evdotheo/anaconda3/envs/mononphm/lib/python3.9/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl return forward_call(args, **kwargs) File "/home/evdotheo/workspace/INDUX_R/Talking_heads/test-bed/diffusion-avatars/MonoNPHM/src/mononphm/models/canonical_space.py", line 538, in forward out = self.propagate(edge_index, TypeError: propagate() got an unexpected keyword argument 'glob_feats_color'

[Zvyozdo4ka]

add flag --no-is_video as mentioned in #7.

I also encountered similar error:

File "/media/yulduz/hdd/platform-modules/3_monoNPHM/scripts/inference/rec.py", line 194, in <module>
    tyro.cli(main)
  File "/home/yulduz/anaconda3/envs/mononphm/lib/python3.9/site-packages/tyro/_cli.py", line 229, in cli
    return run_with_args_from_cli()
  File "/media/yulduz/hdd/platform-modules/3_monoNPHM/scripts/inference/rec.py", line 185, in main
    inverse_rendering(n3dmm, seq_name, _expressions, n_expr,
  File "/media/yulduz/hdd/platform-modules/3_monoNPHM/scripts/inference/rec.py", line 59, in inverse_rendering
    track(wrapped_net, cfg, seq_name, expressions,

Try to resolve it just by removing folder output/pretrained_mononphm/stage1/00010/00000. It worked for me.

[EvdoTheo]

Unfortunately, the problem was not solved by adding the flag --no-is_video.

[Zvyozdo4ka]

Unfortunately, the problem was not solved by adding the flag --no-is_video.

what about removing folder 00000? if it exists the script considers it as a video and looks for intrinsic parameters.

[EvdoTheo]

Sorry, i forgot to mention that! I deleted it but nothing changed. What image resolution did you use as input? Also, did you try the process with a video input?

[SimonGiebenhain]

Not sure what this error is, but it seem related to pytorch-geometric. Which version of pytorch-geometric are you using? I.e. what is the output of conda list | grep pyg

[Zvyozdo4ka]

Sorry, i forgot to mention that! I deleted it but nothing changed. What image resolution did you use as input? Also, did you try the process with a video input?

i did not try to process video, i followed instructions from #3 for image processing. I added in my code several steps to process images and put it to input directory to run mononphm.

1. I convert images to png
2. Took the mask of person on an image and then crop image to 512 x 512
3. Removed alpha channel from image.
4. In case of too small image, padding it to 512 x 512.

And then all my images worked, as you can see in #7, but outputs does not resemble identities :(

[EvdoTheo]

Not sure what this error is, but it seem related to pytorch-geometric. Which version of pytorch-geometric are you using? I.e. what is the output of conda list | grep pyg

Indeed, the problem was derived from some wrong versions of pyg. I reinstalled them and the problem was solved, but now i get cuda.OutOfMemory.

[Zvyozdo4ka]

Indeed, the problem was derived from some wrong versions of pyg. I reinstalled them and the problem was solved, but now i get cuda.OutOfMemory.

I had the same issue. I added at the end of run.sh:

python3 -c "import torch; torch.cuda.empty_cache()"
echo "CUDA cache cleared."

Or you can reduce batch size

[EvdoTheo]

Indeed, the problem was derived from some wrong versions of pyg. I reinstalled them and the problem was solved, but now i get cuda.OutOfMemory.

I had the same issue. I added at the end of run.sh:

python3 -c "import torch; torch.cuda.empty_cache()"
echo "CUDA cache cleared."

Or you can reduce batch size

The error i get is when i run python scripts/inference/rec.py --model_type nphm --exp_name pretrained_mononphm --ckpt 2500 --seq_name 00010 --no-intrinsics_provided --no-is_video

[Zvyozdo4ka]

The error i get is when i run python scripts/inference/rec.py --model_type nphm --exp_name pretrained_mononphm --ckpt 2500 --seq_name 00010 --no-intrinsics_provided --no-is_video

so add it to rec.py, or write run.sh:

#!/bin/bash
python3 scripts/inference/rec.py --model_type nphm --exp_name pretrained_mononphm --ckpt 2500 --seq_name 00010 --no-intrinsics_provided --no-is_video
python3 -c "import torch; torch.cuda.empty_cache()"
echo "CUDA cache cleared."

and then run the command: ./run.sh