search

16lemoing / dot

Dense Optical Tracking: Connecting the Dots
https://16lemoing.github.io/dot
MIT License
234 stars 15 forks source link

_pickle.UnpicklingError: invalid load key, '<'. #2

Open junminren2 opened 9 months ago

junminren2 commented 9 months ago

hi, I found sth maybe needs to be revised: the link is changed, below is correct. !wget -P checkpoints https://huggingface.co/16lemoing/dot/blob/main/cvo_raft_patch_8.pth !wget -P checkpoints https://huggingface.co/16lemoing/dot/blob/main/movi_f_raft_patch_4_alpha.pth !wget -P checkpoints https://huggingface.co/16lemoing/dot/blob/main/movi_f_cotracker_patch_4_wind_8.pth !wget -P datasets https://huggingface.co/16lemoing/dot/blob/main/demo.zip !unzip datasets/demo.zip -d datasets/ When I installed any package and ran the command below. !python demo.py --visualization_modes spaghetti_last_static --video_path orange.mp4 !python demo.py --visualization_modes spaghetti_last_static --video_path treadmill.mp4 I found an error here: Traceback (most recent call last): File "/content/dot/demo.py", line 310, in main(args) File "/content/dot/demo.py", line 269, in main model = create_model(args).cuda() File "/content/dot/dot/models/init.py", line 7, in create_model model = DenseOpticalTracker( File "/content/dot/dot/models/dense_optical_tracking.py", line 23, in init self.point_tracker = PointTracker(height, width, tracker_config, tracker_path, estimator_config, estimator_path) File "/content/dot/dot/models/point_tracking.py", line 19, in init self.model.load_state_dict(torch.load(tracker_path, map_location=device)) File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1028, in load return _legacy_load(opened_file, map_location, pickle_module, pickle_load_args) File "/usr/local/lib/python3.10/dist-packages/torch/serialization.py", line 1246, in _legacy_load magic_number = pickle_module.load(f, pickle_load_args) _pickle.UnpicklingError: invalid load key, '<'. I checked the track_path: movi_f_cotracker_patch_4_wind_8.pth and I guess there is a problem with the saving format of model. I really need your help, Thanks.

16lemoing commented 9 months ago

Hi, I have found what is causing your issue. You are not using the correct download links. Here are the correct links (replace blob by resolve):

wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/cvo_raft_patch_8.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_raft_patch_4_alpha.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_cotracker_patch_4_wind_8.pth
wget -P datasets https://huggingface.co/16lemoing/dot/resolve/main/demo.zip

I have already updated the README. Let me know if it works now.

junminren2 commented 9 months ago

Hi, I have found what is causing your issue. You are not using the correct download links. Here are the correct links (replace blob by resolve):

wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/cvo_raft_patch_8.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_raft_patch_4_alpha.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_cotracker_patch_4_wind_8.pth
wget -P datasets https://huggingface.co/16lemoing/dot/resolve/main/demo.zip

I have already updated the README. Let me know if it works now.

Thanks for your help, it could work now. But I found the speed is very slow like this. image There is any method to accelerate it?

junminren2 commented 9 months ago

Hi, I have found what is causing your issue. You are not using the correct download links. Here are the correct links (replace blob by resolve):

wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/cvo_raft_patch_8.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_raft_patch_4_alpha.pth
wget -P checkpoints https://huggingface.co/16lemoing/dot/resolve/main/movi_f_cotracker_patch_4_wind_8.pth
wget -P datasets https://huggingface.co/16lemoing/dot/resolve/main/demo.zip

I have already updated the README. Let me know if it works now.

Thanks for your help, it could work now. But I found the speed is very slow like this. image There is any method to accelerate it?

Refine target step: 0% 0/51 [00:00<?, ?it/s]/content/dot/dot/models/optical_flow.py:73: UserWarning: Torch3D is not available. For optimal speed and memory consumption, consider setting it up. coarse_flow, coarse_alpha = interpolate(data["src_points"], data["tgt_points"], self.coarse_grid,

16lemoing commented 9 months ago

Thanks for your help, it could work now. But I found the speed is very slow like this. There is any method to accelerate it?

Speed depends on the length of the input video. The time bottleneck is the computation of inital tracks with CoTracker. For instance, it takes ~40 sec for tracking a batch of 2048 points for a video with 120 frames.

You can trade accuracy for speed by using fewer point tracks during inference, e.g.:

python demo.py --visualization_modes spaghetti_last_static --video_path treadmill.mp4 --num_tracks 1024 --sim_tracks 1024

UserWarning: Torch3D is not available.

You can enable Torch3D by running:

cd dot/utils/torch3d/ && python setup.py install && cd ../../..

But this will not change speed much. It is mostly for reducing memory consumption for high resolution inputs.

junminren2 commented 9 months ago

Thanks for your help, it could work now. But I found the speed is very slow like this. There is any method to accelerate it?

Speed depends on the length of the input video. The time bottleneck is the computation of inital tracks with CoTracker. For instance, it takes ~40 sec for tracking a batch of 2048 points for a video with 120 frames.

You can trade accuracy for speed by using fewer point tracks during inference, e.g.:

python demo.py --visualization_modes spaghetti_last_static --video_path treadmill.mp4 --num_tracks 1024 --sim_tracks 1024

UserWarning: Torch3D is not available.

You can enable Torch3D by running:

cd dot/utils/torch3d/ && python setup.py install && cd ../../..

But this will not change speed much. It is mostly for reducing memory consumption for high resolution inputs.

Thanks for your help, Moing. I tried inputting my own video to get a demo, but I found the program quickly finished with just output '^C'. Do you know what problems happened?

16lemoing commented 9 months ago

Hi,don't you have any error message apart from '^C'? Can you share the video so that I can try and see if I can reproduce the issue?

junminren2 commented 9 months ago

Hi Moing: I am grateful for your help. This is my video. thanks best regards John

发件人: Guillaume Le Moing @.> 发送时间: 2023年12月29日 22:26 收件人: 16lemoing/dot @.> 抄送: REN Junming @.>; Author @.> 主题: [Ext] Re: [16lemoing/dot] _pickle.UnpicklingError: invalid load key, '<'. (Issue #2)

CAUTION: External email. Do not reply, click on links or open attachments unless you recognize the sender and know the content is safe.

Hi,don't you have any error message apart from '^C'? Can you share the video so that I can try and see if I can reproduce the issue?

― Reply to this email directly, view it on GitHubhttps://github.com/16lemoing/dot/issues/2#issuecomment-1872129758, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM3XXC5FNUMYTWUVUTZCBFDYL3HH3AVCNFSM6AAAAABBEFI7O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZSGEZDSNZVHA. You are receiving this because you authored the thread.Message ID: @.***>

junminren2 commented 9 months ago

Hi,don't you have any error message apart from '^C'? Can you share the video so that I can try and see if I can reproduce the issue?

Could you please send me your private email? Thanks

16lemoing commented 9 months ago

Hi, you can send me a message at guillaume.le-moing@inria.fr

junminren2 commented 9 months ago

Hi, you can send me a message at guillaume.le-moing@inria.fr

Thanks moing, have you already received my email? Thanks

16lemoing commented 9 months ago

Not yet. Can you try again? There was a typo in my original reply. You can also try 16lemoing@gmail.com

junminren2 commented 9 months ago

Not yet. Can you try again? There was a typo in my original reply. You can also try 16lemoing@gmail.com

Thanks Moing, I already sent to your email 16lemoing@gmail.com. Did you received well?

16lemoing commented 9 months ago

Hi @junminren2 ! I have received your video. It seems that your video is too high resolution ~900 frames at 1080p (reduced to 480p by default with the demo script) to be loaded all at once in memory. I suggest you try with a shorter video first. I can try and implement an online mode that loads frames one by one, to handle long videos with high-resolution frames like yours. Let me know if you are interested.

junminren2 commented 8 months ago

Hi bro: Sorry to reply late due to intensive experiments. Sure, the real-time demo is very good for me. But I don’t wish it takes your much time. Best regards John 获取 Outlook for iOShttps://aka.ms/o0ukef

发件人: Guillaume Le Moing @.> 发送时间: Friday, January 5, 2024 3:24:12 PM 收件人: 16lemoing/dot @.> 抄送: REN Junming @.>; Mention @.> 主题: [Ext] Re: [16lemoing/dot] _pickle.UnpicklingError: invalid load key, '<'. (Issue #2)

CAUTION: External email. Do not reply, click on links or open attachments unless you recognize the sender and know the content is safe.

Hi @junminren2https://github.com/junminren2 ! I have received your video. It seems that your video is too high resolution ~900 frames at 1080p (reduced to 480p by default with the demo script) to be loaded all at once in memory. I suggest you try with a shorter video first. I can try and implement an online mode that loads frames one by one, to handle long videos with high-resolution frames like yours. Let me know if you are interested.

― Reply to this email directly, view it on GitHubhttps://github.com/16lemoing/dot/issues/2#issuecomment-1878240785, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM3XXCZ2JEZCL6HP3NZZXULYM6TBZAVCNFSM6AAAAABBEFI7O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQNZYGI2DANZYGU. You are receiving this because you were mentioned.Message ID: @.***>

16lemoing commented 8 months ago

Hi! This feature has also been requested in another issue (#8) so I will try to implement it in the next few weeks.

junminren2 commented 8 months ago

Thanks for your help. I got my demo from my video. May I ask how to screen tracking points with no change? They totally covered those points with significant change. [cid:3205fbd4-de8d-4e63-a1b5-3401139e93e1]

发件人: Guillaume Le Moing @.> 发送时间: 2024年1月24日 17:21 收件人: 16lemoing/dot @.> 抄送: REN Junming @.>; Mention @.> 主题: [Ext] Re: [16lemoing/dot] _pickle.UnpicklingError: invalid load key, '<'. (Issue #2)

CAUTION: External email. Do not reply, click on links or open attachments unless you recognize the sender and know the content is safe.

Hi! This feature has also been requested in another issue (#8https://github.com/16lemoing/dot/issues/8) so I will try to implement it in the next few weeks.

― Reply to this email directly, view it on GitHubhttps://github.com/16lemoing/dot/issues/2#issuecomment-1907716318, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AM3XXCZP6NHL4HAS5U5NUA3YQDHDNAVCNFSM6AAAAABBEFI7O6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBXG4YTMMZRHA. You are receiving this because you were mentioned.Message ID: @.***>

16lemoing commented 8 months ago

You can specify a mask on the first frame to filter tracks (see varanus example in the README), more advanced filtering is not yet supported.