I configured the setup for my custom videos. But before processing my own videos, I tried to test it on the given preprocessed_data to confirm that everything is configured correctly and working fine.
Data: I choose "preprocessed_data/dim_light/3" sequence for initial testing as I have groundtruth available. I removed all other folders except "unscreen", "unscreen_cropped" and "mask" for inference
I ran the following command after following all the given instructions in the README:
python ./metro/tools/end2end_inference_handmesh.py --resume_checkpoint ./models/metro_release/metro_hand_state_dict.bin --image_file_or_path data --do_crop
Terminal log looks like:
2023-12-22 13:21:05,927 METRO Inference INFO: Inference: Loading from checkpoint ./models/metro_release/metro_hand_state_dict.bin
2023-12-22 13:21:05,927 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter hidden_size: 768 -> 1024
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:05,928 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 4096
2023-12-22 13:21:07,769 METRO Inference INFO: Init model from scratch.
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter hidden_size: 768 -> 256
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:07,769 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 1024
2023-12-22 13:21:08,027 METRO Inference INFO: Init model from scratch.
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter num_hidden_layers: 12 -> 4
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter hidden_size: 768 -> 64
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter num_attention_heads: 12 -> 4
2023-12-22 13:21:08,028 METRO Inference INFO: Update config parameter intermediate_size: 3072 -> 256
2023-12-22 13:21:08,086 METRO Inference INFO: Init model from scratch.
=> loading pretrained model models/hrnet/hrnetv2_w64_imagenet_pretrained.pth
2023-12-22 13:21:10,246 METRO Inference INFO: => loading hrnet-v2-w64 model
2023-12-22 13:21:10,247 METRO Inference INFO: Transformers total parameters: 101182022
2023-12-22 13:21:10,252 METRO Inference INFO: Backbone total parameters: 128059944
2023-12-22 13:21:10,254 METRO Inference INFO: Loading state dict from checkpoint ./models/metro_release/metro_hand_state_dict.bin
2023-12-22 13:21:11,231 METRO Inference INFO: Run inference
- Skip data/dim_light/3/unscreen. Images already cropped
After coarse alignment: 284.209717
After fine alignment: 2.175960
save to data/dim_light/3/metro_image/0001_metro_pred.jpg
After coarse alignment: 277.326782
After fine alignment: 2.160460
save to data/dim_light/3/metro_image/0002_metro_pred.jpg
After coarse alignment: 279.830750
After fine alignment: 2.158962
save to data/dim_light/3/metro_image/0003_metro_pred.jpg
After coarse alignment: 271.266846
After fine alignment: 2.142965
Which looks fine to me as the error after alignment looks good. But "metro_image" and "mano_fit_image" folders have white images in it like this:
Further, I checked the "metro_mano" and printed the output of mano["joints"] from "metro_mano/0001_mano.pkl" and compared with the already given mano files in preprocessed_data and they were almost equal:
Thank you for open-sourcing your wonderful work.
I configured the setup for my custom videos. But before processing my own videos, I tried to test it on the given preprocessed_data to confirm that everything is configured correctly and working fine.
Data: I choose "preprocessed_data/dim_light/3" sequence for initial testing as I have groundtruth available. I removed all other folders except "unscreen", "unscreen_cropped" and "mask" for inference
I ran the following command after following all the given instructions in the README:
python ./metro/tools/end2end_inference_handmesh.py --resume_checkpoint ./models/metro_release/metro_hand_state_dict.bin --image_file_or_path data --do_crop
Terminal log looks like:
Which looks fine to me as the error after alignment looks good. But "metro_image" and "mano_fit_image" folders have white images in it like this:
Further, I checked the "metro_mano" and printed the output of mano["joints"] from "metro_mano/0001_mano.pkl" and compared with the already given mano files in preprocessed_data and they were almost equal:
Why output images are blank but the mano output looks correct?