"feature_extratcor.py" seem to not be making CD3 files for all videos given to it

[PHDPeter]

Hi, Me again. Coming back to this code after a while I am having a problem where "feature_extratcor.py" is not creating a CD3 feature files for every video I give it (5 out of 100 in my test). I know that every video file is being loaded in by "data_loader" (i have done this using both the batch loading system used here as well as the single loading method I made here (https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/issues/7)).

I think the problem is with getitem_from_raw_video() and could be related to the cumulative_sizes variable not being right However, I have looked at the cumulative_sizes variable using both the batch and single load methods and they are the same and look fine to me (example below). self.cumulative_sizes =[224, 275, 305, 334, 457, 541, 677, 733, 780, 865, 902, 982, 998, 1046, 1083, 1142, 1179, 1209, 1237, 1405, 1417, 1546, 1767, 1810, 1910, 2010, 2059, 2115, 2738, 2789, 2813, 2927, 3042, 3066, 3131, 3179, 3190, 3315, 3411, 3432, 3569, 3588, 3616, 3731, 3944, 4038, 4197, 4325, 4351, 4420, 4433, 4451, 4507, 4702, 4762, 4778, 4999, 5034, 5083, 5394, 5420, 5479, 5529, 5549, 5689, 5720, 5776, 5822, 6006, 7986, 8014, 8126, 8160, 8189, 8230, 8465, 8525, 9004, 9047, 9104, 9132, 9148, 9222, 9261, 9320, 9357, 9365, 9448, 9508, 9526, 9699, 9729, 9758, 9788, 9950, 10001, 10085, 10099, 10168, 10222]

I guess it could thus be to issues (https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/issues/14) as I made the suggested changes there and that is in the getitem_from_raw_video() method? My getitem_from_raw_video looks like this:

` def getitem_from_raw_video(self, idx):

get current video info

    video, _, _, _ = self.video_clips.get_clip(idx)
    video_idx, clip_idx = self.video_clips.get_clip_location(idx)
    video_path = self.video_clips.video_paths[video_idx]
    in_clip_frames = list(range(0, self.total_clip_length_in_frames, self.frames_stride))
    if self.video_transform is not None:
        video = self.video_transform(video)
    if "Normal" not in video_path:
        label = 1
    else:
        label = 0

    dir, file = video_path.split(os.sep)[-2:]
    file = file.split('.')[0]
    #video=video.numpy()
    #test=video.shape
    #t=video[:][0]
    #video[in_clip_frames]
    return video[:, in_clip_frames, :, :], label, clip_idx, dir, file`

Any help or points of where to look for the issue would be greatly appreciated :-) thanks

[ekosman]

@PHDPeter That's interesting. Have you tried to look at self.video_transforms? Let me know if it holds all the relevant files.

https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/blob/15060776bd3030e4236ed0016a58b471937063d1/data_loader.py#L28

[PHDPeter]

Not sure I am following @ekosman? The self.video_transforms equals the build_transforms from the utils.py in utils folder, witch is the same as the one here (see below)

def build_transforms(): mean = [124 / 255, 117 / 255, 104 / 255] std = [1 / (.0167 255)] 3 res = transforms.Compose([ transforms_video.ToTensorVideo(), transforms_video.RandomResizedCropVideo(256, 224), transforms_video.NormalizeVideo(mean=mean, std=std) ])

return res

video_transform is only used after defining video_idx in the getitem_from_raw_video() method of data_loader, which is where I think the problem is as that defines the video_path used and I have cheeked that video_paths is correct. Thank for the replay btw :-)

[ekosman]

@PHDPeter I relate to self.video_list, not the transforms. It should hold the list of file names.

[PHDPeter]

Oww I get you @ekosman :-) I did look at self.video_list and it looks right to me I.e. a list of 100 different video paths (100 being the number of videos I am using in my test). That is why I think the issues are the video_idx as it just keeps coming out at the same number when it should be changing more often

[ekosman]

@PHDPeter Oh well, video_idx is is the index of the video in the video_clip.video_paths variable. I think it has something to do with torchvision's videoClips module. I wrote this code when torchvision ver4 was beta (and buggy), so I just copied the module to a new file and fixed what I needed. I think it's a great time to try using the current module of torchvision, if it works properly now (just use import): https://github.com/pytorch/vision/blob/master/torchvision/datasets/video_utils.py

[PHDPeter]

Hi @ekosman I think I get you and have a workaround. I am not 100% as I don't have any file outputs that I know is right. For example, I am correct in saying that there should be one .txt file for each video given in the annotation file right? Maybe if I add a clip and we could see if are feature extraction is the same?

If that is right then my current fix is just to change line 39 in the data_loader.py (https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/blob/15060776bd3030e4236ed0016a58b471937063d1/data_loader.py#L39 ) to video_path = self.video_clips.video_paths[idx]#video_idx That looks correct, please let me know if anyone sees any problems with that?

[ekosman]

@PHDPeter I'm not sure why passing idx would fix that, don't you get an IndexError?

[PHDPeter]

Yeah, I tested it along with @wzy1845 from (https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/issues/1) and it does not seem to have fixed the problem (so at least we know it is replicable).

The reason I thought passing 'idx' would work was that I noticed that the 'idx' was never higher than 100 (for the 100 videos I had listed) but the 'cumulative_sizes' list starts with 224, i.e. the number of frames. Thus I thought that the scale of the too might have been off and that was why it was not working. Guess I can't have been right but I am sure that the problem is not with the 'cumulative_sizes' or 'self.video_clips.get_clip_location(idx)' with uses the 'cumulative_sizes'.

@ekosman Could you give me a bit more info on how the 'idx' is generated from the batch unpacking of 'train_iter' in 'for i_batch, (data, target, sampled_idx, dirs, vid_names) in tqdm(enumerate(train_iter)):' ?

[ekosman]

@PHDPeter Your last comment gave me insights and I think I know exactly what is the problem now: https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/blob/7034ee7db850956eb574fe1130221ba7f9f12423/data_loader.py#L43

Right now, it returns: https://github.com/ekosman/AnomalyDetectionCVPR2018-Pytorch/blob/7034ee7db850956eb574fe1130221ba7f9f12423/data_loader.py#L44

But should return:

return len(self.video_clips)

[ekosman]

Did it fix the problem for you? I consider closing the issue

[PHDPeter]

Hi @ekosman sorry for the delay. Needed my work PC to run the code and was busy but yes it seems to be working :-D