search

tfaehse / DashcamCleaner

Censor identifiable information in videos, in particular dashcam recordings in Germany.
GNU Affero General Public License v3.0
130 stars 27 forks source link

Output Video has stand image #61

Closed idkwhatido closed 1 year ago

idkwhatido commented 1 year ago

The original video play completely fine, but the output video has a stand image after the first half. The sound is still playing fine, it's just the video that doesn't play anymore.

In the console there is no error, there is only the following warning after start: IMAGEIO FFMPEG_WRITER WARNING: input image is not divisible by macro_block_size=16, resizing from (1920, 1080) to (1920, 1088) to ensure video compatibility with most codecs and players. To prevent resizing, make your input image divisible by the macro_block_size or set the macro_block_size to 1 (risking incompatibility).

Setup of my system & environment:

Have you any idea what i'm doing wrong?

joshinils commented 1 year ago

i have seen issues with vlc playing back videos with quality=10, i guess because there is too much data per frame thus the decoding is too slow for playback at actual speed.

i have no idea why that would happen at quality=5 with your machine.

you could try re-muxing the output with ffmpeg and see if the artifacts/playback issues remain. i.e. ffmpeg -i inputname.mp4 outputname.mp4 (do not use the same filename!) (without any other settings regarding, like copy streams or quality, then ffmpeg will use the default quality settings and decode and re-encode the frames again, resulting in a different file)

idkwhatido commented 1 year ago

thank you for the response. I tried it out, unfortunaly the re-muxing doesn't have any effect and the issue is still the same. Maybe i can try it on another host the next days, i will give feedback after it again.

joshinils commented 1 year ago

alternatively try to re-mux before blurring (quality caveats aside), maybe the encoding is somehow messed up from the camera you used but vlc is for example is more forgiving during playback.

idkwhatido commented 1 year ago

tried with vlc and also re-mux before blurring, but still doesn't work. However while playing the final blurred version in vlc i noticed that after 12s (from 22s video) the playback completely stop and after 10s in that state it behave like the video ended fine. I also tried it with another 1min video now, but it seems that always the last 10-12s are missing and i don't understand why :/

tfaehse commented 1 year ago

Would it be possible for you to share a video from the same source (could be just you recording a wall, it doesn't matter)?

I'm very, very far from knowledgeable regarding video codecs - but as far as I can tell, imageio-ffmpeg should support whatever FFMPEG supports, so virtually everything. I suspect that there's something wrong with the duration metadata in these files, I use it as input for the VideoWriter. However, imageio-ffmpeg's reader may also use fields like the duration or total frames, maybe there's a standard?

idkwhatido commented 1 year ago

Sure, no problem. I uploaded 2 videos to mega, the .mov is the original 4K 1min file from the camera and the .mp4 is a 1080p version cutted with clipchamp.

I will remove the link as soon as you downloaded it, please let me know when you downloaded it so i can remove it. Link: https://mega.nz/folder/5MhQHTKC#Tf7UmzycUj-WiBFew4UJNQ

Thank you for your help again :)

joshinils commented 1 year ago

when i use the short mp4 file with default arguments for the cli.py i get an error because of this line:

https://github.com/tfaehse/DashcamCleaner/blob/main/dashcamcleaner/src/blurrer.py#L142

https://github.com/tfaehse/DashcamCleaner/commit/b3dc71dc77617a8121f38a6f5b83f7b6cfbd62ac#diff-dc15d6e41ef6c6b5e2ae45f31eb3dd289b44ea277afa9438d0d8e4cb1ee90090R140

➜  DashcamCleaner git:(feature/parallel-blur) dashcamcleaner/cli.py -i idkwhatido/eng\ überholen\ +\ gegenverkehr\ ausbremsen.mp4 -o eng.mp4
zsh: correct 'eng2.mp4' to 'eng.mp4' [nyae]? n
Start blurring video: idkwhatido/eng überholen + gegenverkehr ausbremsen.mp4
Blurring parameter: {'input_path': 'idkwhatido/eng überholen + gegenverkehr ausbremsen.mp4', 'output_path': 'eng2.mp4', 'weights': '720p_medium_mosaic', 'batch_size': 1, 'blur_size': 9, 'inference_size': 720, 'threshold': 0.4, 'roi_multi': 1.0, 'quality': 10, 'feather_edges': 5, 'no_faces': False, 'export_mask': False, 'export_colored_mask': False}
YOLOv5 🚀 2022-7-7 Python-3.8.10 torch-1.10.2+cu102 CPU

Fusing layers... 
Model summary: 308 layers, 21041679 parameters, 0 gradients
Adding AutoShape... 
Using CPU.
Worker created
Processing video:   0%|                                                                                                                                                                                                                       | 0/534 [00:00<?, ?frames/s]
Traceback (most recent call last):
  File "dashcamcleaner/cli.py", line 240, in <module>
    cli.start_blurring()
  File "dashcamcleaner/cli.py", line 48, in start_blurring
    self.start_blurring_file()
  File "dashcamcleaner/cli.py", line 63, in start_blurring_file
    blurrer.blur_video()
  File "/home/niels/Documents/DashcamCleaner/dashcamcleaner/src/blurrer.py", line 94, in blur_video
    new_detections: List[List[Detection]] = self.detect_identifiable_information(frame_buffer)
  File "/home/niels/Documents/DashcamCleaner/dashcamcleaner/src/blurrer.py", line 39, in detect_identifiable_information
    results_list = self.detector(images, size=(scale,)).xyxy
  File "/home/niels/.local/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/home/niels/.local/lib/python3.8/site-packages/torch/autograd/grad_mode.py", line 28, in decorate_context
    return func(*args, **kwargs)
  File "/home/niels/.cache/torch/hub/ultralytics_yolov5_master/models/common.py", line 595, in forward
    g = (size / max(s))  # gain
TypeError: unsupported operand type(s) for /: 'tuple' and 'int'

when i change that line 

diff --git a/dashcamcleaner/src/blurrer.py b/dashcamcleaner/src/blurrer.py
index fb0b269..fc878b4 100644
--- a/dashcamcleaner/src/blurrer.py
+++ b/dashcamcleaner/src/blurrer.py
@@ -36,7 +36,7 @@ class VideoBlurrer:
         :return: detected faces and plates
         """
         scale = self.parameters["inference_size"]
-        results_list = self.detector(images, size=(scale,)).xyxy
+        results_list = self.detector(images, size=scale).xyxy
         return [
             [
                 Detection(

it works, and the output video is 17s long and all plates are blurred. :man_shrugging:

idkwhatido commented 1 year ago

Ok, i didn't got this error. I redownloaded the repo and tried it out via cli, saved as mp4 with all settings default and it worked absolutly fine. Then i tried it with the gui again and had the same issue. After that i rechecked with the cli and again it worked fine, finally i tried again with cli but this time save as mkv because the gui only allow to save as mkv and got the same issue.

It seem that saving as mkv produce this issue, at least for me, so it should be good to allow other filetypes than mkv in the output dialog in the gui.

But finally i can use it without issues and cli is not a big problem for me, thank you so much for you help

joshinils commented 1 year ago

hrm, problem is i only look at the code from the cli perspective, i don't like to use the gui.

edit: i also processed the long mov with the default cli settings and it generated a 9GB file, too large for playback with vlc, putting it through ffmpeg makes it 376MB large and playable, blurred and complete.

tfaehse commented 1 year ago

That's... interesting. Unfortunately I'm too late to download the video, so I can't really debug this.

Support for other file types will be added to the GUI, but this doesn't really solve the issue - I'd like to get to the bottom of this, I have a hard time understanding how a different container would result in a shorter output video.

idkwhatido commented 1 year ago

Hi @tfaehse, i reuploaded it for you so can better debug the issue. Here is the Link: https://mega.nz/folder/8YxjAKxa#VQhRUxQ_wDFhdQ5cdhGSEw

Please let me know when you downloaded it, so i can remove it again

tfaehse commented 1 year ago

Thanks! I got the video, I'll have a look in the coming days.

tfaehse commented 1 year ago

Hi @idkwhatido, sorry for the ridiculously late reply. I just checked again, the videos process just fine for me. Could you add a line after meta = reader.get_meta_data() in qt_wrapper.py (line 56)? Just add print(meta) to show what image thinks about the video file, on your system.

thgoldha commented 1 year ago

I have the same issue. I noticed, that I get a stand image at about 50% of the output video. When I break the link to FFMPEG, then after blurring an error appears, but I have the whole cleaned video without audio. Might be the problem appears in the "post processing"?

idkwhatido commented 1 year ago

Hi @idkwhatido, sorry for the ridiculously late reply. I just checked again, the videos process just fine for me. Could you add a line after meta = reader.get_meta_data() in qt_wrapper.py (line 56)? Just add print(meta) to show what image thinks about the video file, on your system.

Sure, I can test it in the next days. For now I fixed it myself by just changing the output format from mkv to mp4.

tfaehse commented 1 year ago

When I break the link to FFMPEG, then after blurring an error appears, but I have the whole cleaned video without audio. Might be the problem appears in the "post processing"?

I think that explains it! In theory, I'm trying to just tell ffmpeg to combine the video from the blurred, mute video and the audio from the original input video. Looks like this doesn't work as expected, I'll look into it! Same question as above: would it be at all possible to share an affected video? It can be of anything, a wall or whatever, as long as you get the same issues with it.

thgoldha commented 1 year ago

I switched to new YOLO v8. Works fine now. I cannot reproduce this error anymore.