search

MCG-NJU / DDM

[CVPR 2022] Progressive Attention on Multi-Level Dense Difference Maps for Generic Event Boundary Detection
MIT License
49 stars 3 forks source link

Evaluation performance on GEBD validation set #5

Open SJTUwxz opened 1 year ago

SJTUwxz commented 1 year ago

Hi, this project is great and thanks for releasing the code! I've re-trained DMM and the evaluation result on GEBD val set is as follows, which is around 2% lower than the reported result.

+GEBD Performance on Kinetics-GEBD----+--------+--------+--------+--------+--------+--------+--------+--------+ | Rel.Dis. | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.45 | 0.50 | Avg | +----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+ | F1 | 0.7447 | 0.8252 | 0.8496 | 0.8615 | 0.8679 | 0.8722 | 0.8750 | 0.8774 | 0.8796 | 0.8817 | 0.8535 | +----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+

I've also tried loading the trained weights you've released and run the evaluation again, the result is still around 2% lower, which is,

+GEBD Performance on Kinetics-GEBD----+--------+--------+--------+--------+--------+--------+--------+--------+ | Rel.Dis. | 0.05 | 0.10 | 0.15 | 0.20 | 0.25 | 0.30 | 0.35 | 0.40 | 0.45 | 0.50 | Avg | +----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+ | F1 | 0.7462 | 0.8234 | 0.8462 | 0.8578 | 0.8642 | 0.8684 | 0.8715 | 0.8739 | 0.8758 | 0.8776 | 0.8505 | +----------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+--------+.

I would really appreciate it if you could provide any insights on possible reasons of this. Thanks a lot!

JackyTown commented 1 year ago

Maybe you can follow https://github.com/MCG-NJU/DDM/blob/main/GUIDE.md and find out the reason.

I guess the problem may be frame extraction or pickle preparation.

SJTUwxz commented 1 year ago

Thanks for quick response! Yeah I followed that guide and ignore videos that I couldn't download from Youtube anymore. I will take a closer look and check again. I have another two questions:

  1. Are the "multi-frames-GEBD-train-5.pkl" file and also val file released? I know that these pickle files are generated in data loading, so I just want to see if I can compare mine with yours and see if that's causing the trouble.
  2. It seems that in the data loading, the number of frames of the clip centering query frame is only 10 (T=10), 5 frames before the center frame and 5 frames after the center frame, where the center frame is not loaded in the sequence. This is causing inconsistency because the positional embeddings are initialized with num_locations = 11. I've changed this part of the original code to put center frame back to the sequence because in the paper it says 11 frames are loaded for each clip.

Thanks again!

Jo-won commented 1 year ago

Hi. I'm also experiencing the same problem with @SJTUwxz regarding reimplementation, so could you (@JackyTown) release the "multi-frames-GEBD-train-5.pkl" and "multi-frames-GEBD-val-5.pkl" you used for troubleshooting?

JackyTown commented 1 year ago

Have you tried this in generate_pickle.py?

-shift = np.arange(-frame_per_side, frame_per_side) -shift[shift >= 0] += 1 +shift = np.arange(-frame_per_side, frame_per_side + 1)

Ziwei-Zheng commented 1 year ago

Hi. I also have trouble with the reimplementation. Same to @SJTUwxz, I got a 2% performance drop on the validation set with both the model trained by myself and the provided pre-trained one. I wonder if you can release the "multi-frames-GEBD-val-5.pkl" so that I could check out the possible reasons during validation.

Thanks!

ForeverPs commented 6 months ago

Hello, I encounter some difficulties in preparing the Kinetics-GEBD data, could you give me some detailed help? And thanks for your excellent work!

Solved, thanks!

Loewen-Hob commented 5 months ago

Thanks for quick response! Yeah I followed that guide and ignore videos that I couldn't download from Youtube anymore. I will take a closer look and check again. I have another two questions:

  1. Are the "multi-frames-GEBD-train-5.pkl" file and also val file released? I know that these pickle files are generated in data loading, so I just want to see if I can compare mine with yours and see if that's causing the trouble.
  2. It seems that in the data loading, the number of frames of the clip centering query frame is only 10 (T=10), 5 frames before the center frame and 5 frames after the center frame, where the center frame is not loaded in the sequence. This is causing inconsistency because the positional embeddings are initialized with num_locations = 11. I've changed this part of the original code to put center frame back to the sequence because in the paper it says 11 frames are loaded for each clip.

Thanks again!

Hello! @SJTUwxz I also encountered the second issue you mentioned, but I haven't figured out how to put the center frame back to the sequence. Can you tell me which folder it is in

你在generate_pickle.py中尝试过这个吗？ -shift = np.arange（-frame_per_side，frame_per_side） -shift[shift >= 0] += 1 +shift = np.arange（-frame_per_side，frame_per_side + 1）

I solved this problem. I found that the above modifications were not made in the file DDM-Net/datasets/MultiFDataset.py. my question!