search

shubham-goel / 4D-Humans

4DHumans: Reconstructing and Tracking Humans with Transformers
https://shubham-goel.github.io/4dhumans/
MIT License
1.18k stars 112 forks source link

pose_transformer_v2 (BERT style transformer) use while running model on video #116

Open thribhuvanrapolu opened 4 months ago

thribhuvanrapolu commented 4 months ago

In the run tracking demo on videos, the paper mentions that the BERT-style transformer model(pose_transformer_v2) enables future predictions and amodal completion of missing detections within the same framework.

However, in the PHALP.py script, after running the (pose_transformer_v2), its output is deleted at line 260(in PHALP.py), and I can't find the model output/values used anywhere.

Where exactly does the code utilize the pose transformer v2? Is it involved in the rendering process?

geopavlakos commented 4 months ago

The pose transformer is used to predict future poses for each tracklet (and compare them with the detected poses when doing identity tracking). These future poses are not visualized. Currently, we only visualize the single-frame estimates from HMR2.0.

thribhuvanrapolu commented 4 months ago

Thanks for clarifying!

thribhuvanrapolu commented 3 months ago

Has there been any work done to evaluate the performance of this pose_transformer_v2(BERT style transformer)? I have looked into the LART paper but the transformer model looks different from 4D-Humans.