Issue re-encoding videos

[IsabelleSajonia]

HI,

I had some issues running inference where some frames weren't found. Additionally, predictions looked correct but shifted compared to the frame. I checked documentation and it seemed like I needed to re-encode my videos. I ran this in my conda terminal: ffmpeg -y -i "C:\Users\irs3th\Desktop\Vehicle vids\Veh 3_3_2024 arena 1.mp4" -c:v libx264 -pix_fmt yuv420p -preset superfast -crf 23 "output.mp4"

However, I'm getting the following error (this is just a portion):

Last message repeated 1 times [h264 @ 00000175311d9140] Invalid NAL unit size (0 > 1599). [h264 @ 00000175311d9140] Error splitting the input into NAL units. [vist#0:0/h264 @ 000001753052fcc0] Error submitting packet to decoder: Invalid data found when processing input Last message repeated 14 times [out#0/mp4 @ 000001753109b000] video:1104077kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.383153% frame=359990 fps=2515 q=-1.0 Lsize= 1108307kB time=03:59:59.48 bitrate= 630.5kbits/s dup=114 drop=0 speed= 101x [libx264 @ 00000175300d7d80] frame I:1440 Avg QP:13.76 size: 39433 [libx264 @ 00000175300d7d80] frame P:90718 Avg QP:15.23 size: 8153 [libx264 @ 00000175300d7d80] frame B:267832 Avg QP:15.96 size: 1248 [libx264 @ 00000175300d7d80] consecutive B-frames: 0.8% 0.0% 0.0% 99.2% [libx264 @ 00000175300d7d80] mb I I16..4: 8.7% 34.7% 56.5% [libx264 @ 00000175300d7d80] mb P I16..4: 3.3% 10.7% 2.1% P16..4: 49.3% 0.0% 0.0% 0.0% 0.0% skip:34.5% [libx264 @ 00000175300d7d80] mb B I16..4: 0.3% 0.7% 0.2% B16..8: 16.5% 0.0% 0.0% direct: 2.6% skip:79.7% L0:42.5% L1:53.0% BI: 4.4% [libx264 @ 00000175300d7d80] 8x8 transform intra:62.7% inter:26.1% [libx264 @ 00000175300d7d80] coded y,uvDC,uvAC intra: 70.5% 0.0% 0.0% inter: 7.7% 0.0% 0.0% [libx264 @ 00000175300d7d80] i16 v,h,dc,p: 29% 15% 30% 26% [libx264 @ 00000175300d7d80] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 20% 15% 26% 7% 6% 8% 6% 7% 6% [libx264 @ 00000175300d7d80] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 25% 21% 20% 7% 6% 5% 5% 5% 5% [libx264 @ 00000175300d7d80] i8c dc,h,v,p: 100% 0% 0% 0% [libx264 @ 00000175300d7d80] Weighted P-Frames: Y:0.0% UV:0.0% [libx264 @ 00000175300d7d80] kb/s:628.11

[IsabelleSajonia]

For more information: I did find an artifact in the 4 hour video so this may be an issue with our recording software... However, when I crop just the artifact out at the end of the video I still get this error, and when I run inference on just the artifact I'm able to get predictions (although they of course don't look great). I'm attaching an image of the weird frame for you. This may be unrelated but I also get an optimization loop failure message when running videos, though I get results that look ok. Sorry, this is a lot of information but I'm not sure which issue is causing these errors all of a sudden.

[talmo]

Hi @IsabelleSajonia,

Woof, that looks gnarly!

Yeah, I think the issue is with the video recording software. The best we can offer is the reencoding solution, so if that's not working there's not much we can do since we use pretty standard implementations of the video readers through OpenCV/ffmpeg.

When you reencode and trim out the artifact frames, are you able to run inference in SLEAP with the shortened video?

Talmo

[IsabelleSajonia]

Yes, I'm able to run inference after reencoding and trimming! When I reencode, the video does appear to be almost zoomed in (one of the walls of the cage is cropped out). Is there a way to fix this? This is possibly unrelated to the video recording software issues, but we also get this error that say optimization loop failed. The results still look ok though. Thanks!

[IsabelleSajonia]

Update: I figured out the issue and the reencoding did wonders for our results!