Problem about extracted pnr frames

[Manuel-Z]

I got some problems during PNR extracting when I use i3drenset to extract PNR by the fho_oscc-pnr_train.json. Thus I have got some positive clips and negative clips.

1. When the loader num is 5, I got "RuntimeError: Dataloader worker is killed by signal", this can be solved by zeroing, with a slower speed,
2. The following problem still occurs after zeroing: "TypeError: default_collate: batch must contain tensors, NumPy arrays, numbers, dicts or lists: found <class 'NoneType'>" and "TypeError: Could not infer dtype of NoneType". During the process, the error may be caused by None-pnr_frame, I try to fix it but failed :( By the side, python==3.7.15 pytorch==1.11.0 PyTorch-lightning==1.2.10, and the full_scale dataset is validated for 28348 clips, annotation== fho_oscc-pnr_train.json. Looking forward to anyone's help to solve the problem sincerely.

[Sid2697]

Hello @Manuel-Z, can you please specify what you mean by "loader num"? Also, it looks like because of the changes you made, no images are being loaded and hence the TypeError.

[Manuel-Z]

Hello @Manuel-Z, can you please specify what you mean by "loader num"? Also, it looks like because of the changes you made, no images are being loaded and hence the TypeError.

HI! @Sid2697 , but it also reports error before i make that change, even i decrease it to 5 i change it in the path: "i3dresnet/configs/2021-09-18_keyframe_loc_release1-v2_main-experiment.yaml"

[Sid2697]

@Manuel-Z thanks for sharing. I have a few questions:

• Are you reducing the number of workers (NUM_WORKERS) to 5?
• How many CPU cores do you have in your machine?
• Can you please confirm if the video downloading was successful and you're able to read them using the code provided? You can try playing the video using VLC media player or you can use OpenCV to load the video once and extract all the frames. Usually, NoneType error occurs when there is an issue reading the frames.

[Manuel-Z]

@Manuel-Z thanks for sharing. I have a few questions:

• Are you reducing the number of workers (NUM_WORKERS) to 5?
• How many CPU cores do you have in your machine?
• Can you please confirm if the video downloading was successful and you're able to read them using the code provided? You can try playing the video using VLC media player or you can use OpenCV to load the video once and extract all the frames. Usually, NoneType error occurs when there is an issue reading the frames.

Really thanks for your kind help! @Sid2697 👍 I got 20 cores, and the program suggests trying 20, but it always breakdown the system:( besides, I have checked the videos by the codes below:

clips = json.load("ANN_DIR")["clips"]

for clip in tqdm(clips):

video = video_path + "/" + str(clip["video_uid"]) + ".mp4"

container = av.open(video)

and the result is 28348/28348 thus have got some positive clips and negative clips, i‘m pretty sure the program works, and looks forward to doing some research under these clips.

[Sid2697]

@Manuel-Z Glad to be of help! I see, this looks good. Did you have a look at the extracted video frames saved in the folders? They will be at two locations specified by the following variables:

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/configs/2021-09-18_keyframe_loc_release1-v2_main-experiment.yaml#L5

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/configs/2021-09-18_keyframe_loc_release1-v2_main-experiment.yaml#L6

[Manuel-Z]

@Manuel-Z Glad to be of help! I see, this looks good. Did you have a look at the extracted video frames saved in the folders? They will be at two locations specified by the following variables:

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/configs/2021-09-18_keyframe_loc_release1-v2_main-experiment.yaml#L5

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/configs/2021-09-18_keyframe_loc_release1-v2_main-experiment.yaml#L6

yep, and each successfully executed folder contains 241 images while the unsuccessful folder contains less, even 0. these can be solved by the following codes i think:

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/datasets/StateChangeDetectionAndKeyframeLocalisation.py#L116

but the RuntimeError or TypeError still occurs.

[Sid2697]

Ah! I see. Can you please re-run the extraction code? As you can see here, if there are folders with less number of frames, we will delete them and re-extract the frames.

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/datasets/StateChangeDetectionAndKeyframeLocalisation.py#L119

Re-running the extraction code will solve all such cases.

[Manuel-Z]

Ah! I see. Can you please re-run the extraction code? As you can see here, if there are folders with less number of frames, we will delete them and re-extract the frames.

https://github.com/EGO4D/hands-and-objects/blob/0aea0a4ed6ed1de4ffcb6b0cfa555da5c3c08bd7/state-change-localization-classification/i3d-resnet50/datasets/StateChangeDetectionAndKeyframeLocalisation.py#L119

Re-running the extraction code will solve all such cases.

Thanks a lot for the idea and your time! makes me feel not alone in this case:) I tried this but after some successful extracts, the error occurred. And in torch.collate.py shows, some element's type is <class 'NoneType'>.

[Sid2697]

That is strange. Is it happening for some specific videos in Ego4D? In this case, you might want to check if they were downloaded properly.

[Manuel-Z]

That is strange. Is it happening for some specific videos in Ego4D? In this case, you might want to check if they were downloaded properly.

Really Thanks! and i will try this and look for further discussion about this task 💯

[Sid2697]

Closing the issue assuming it is solved. Feel free to re-open if the issue persists and a discussion is required.

Regards, Siddhant Bansal

[cmh1027]

Hello, I got the same error when executing the code. The question is, if the video has no state change, then isn't it normal that pnr_frame field becomes 'None'? I don't think this problem is related to the download issue.