search

yufu-wang / tram

TRAM: Global Trajectory and Motion of 3D Humans from in-the-wild Videos
MIT License
188 stars 13 forks source link

Problem running on other videos #6

Closed Marlod390 closed 1 month ago

Marlod390 commented 4 months ago

Dear author,

After running the code on the sample video, I also found some other videos on the web to test. Some videos with large lens movements can run normally, but some videos with fixed lenses will cause the following error: Running on /home/markusc/video_list/fixed/tango/hard_2_tango.mp4 ... No focal length provided ... Search for a good focal length for SLAM ... Traceback (most recent call last): File "/home/markusc/github/tram/scripts/slam_video.py", line 48, in <module> focal = search_focal_length(img_folder, masks) File "/home/markusc/github/tram/scripts/../lib/pipeline/masked_droid_slam.py", line 214, in search_focal_length best_err = test_slam(img_folder, img_msks, conf_msks, File "/home/markusc/github/tram/scripts/../lib/pipeline/masked_droid_slam.py", line 196, in test_slam droid.track(t, image, intrinsics=intrinsics, mask=conf_msk) File "/home/markusc/github/tram/thirdparty/DROID-SLAM/droid_slam/droid.py", line 73, in track self.backend() File "/home/markusc/anaconda3/envs/tram/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context return func(*args, **kwargs) File "/home/markusc/github/tram/thirdparty/DROID-SLAM/droid_slam/droid_backend.py", line 35, in __call__ graph.add_proximity_factors(rad=self.backend_radius, File "/home/markusc/github/tram/thirdparty/DROID-SLAM/droid_slam/factor_graph.py", line 396, in add_proximity_factors ii, jj = torch.as_tensor(es, device=self.device).unbind(dim=-1) ValueError: not enough values to unpack (expected 2, got 0)

This seems to be an error from Droid-SLAM, and I didn't find a solution in their issue. I was wondering if this project is specifically designed for videos with larger camera movements or if you know of any solution for this issue. This is one of the videos that will cause an error:

https://github.com/yufu-wang/tram/assets/67206431/2df2f116-03cd-4290-ac74-522e2cdf7103

rrenle commented 4 months ago

@Marlod390 Hello, have you solved this problem? Now, I have the same error.

Marlod390 commented 4 months ago

@rrenle No. It only works on videos with large camera movement. The video with fixed camera like your video still result this error.

yufu-wang commented 4 months ago

Thanks for trying out the code. This is likely a problem with the original DROID-SLAM itself and not from our modification. As the videos are with static camera, you can skip slam_video. But you have to slightly modify vimo_video (set a focal length) and tram_video (change the camera motion to identity). I can also support this function. I will try to add this in the coming weeks.

JiahongWu1995 commented 4 months ago

not enough values to unpack

Hi, How to handle the traj in tram_video.py？ @yufu-wang

ygx2011 commented 3 months ago

Thanks for trying out the code. This is likely a problem with the original DROID-SLAM itself and not from our modification. As the videos are with static camera, you can skip slam_video. But you have to slightly modify vimo_video (set a focal length) and tram_video (change the camera motion to identity). I can also support this function. I will try to add this in the coming weeks.

I also got the same problem, how to modify tram_video to solve the problem?

weizhuowang commented 3 months ago

Same issue, please advise!

ElviraLL commented 2 months ago

I tried to set a fixed focal and use zero tensors for cam_tran, identity for cam_rotation, couldn't get the final rendered video, Do I need to rotate the camera to a specific angle to make it render the target mesh? Thanks!

yufu-wang commented 1 month ago

Sorry for the delay. I refactored the code and it can now handle static cam. Please take a look at readme on the new scripts. It will try to catch static camera itself, or you can pass in the argument. python scripts/estimate_camera.py --video "./another_video.mov" --static_camera

yufu-wang commented 1 month ago 
python scripts/estimate_camera.py --video './hard_2_tango.mp4' 
python scripts/estimate_humans.py --video './hard_2_tango.mp4' --max_humans 1
python scripts/visualize_tram.py --video './hard_2_tango.mp4' --ground_estimate 0  --floor_scale 10

I can get result like this one

https://github.com/user-attachments/assets/16644dd9-0b29-450a-8618-32654bc9bcd4

Marlod390 commented 1 month ago

@yufu-wang Thank you for the update! I also tested it on other videos and confirmed that tram can handle videos with fixed camera positions correctly. Excellent work! It would be better if you could update the installation instructions to add a section on how to install deva.