Errors

[Vladislavik]

Hi, i trying this module on many mp4 files h264, aac, and have some errors on some files, this videos work fine on local mp4 player (5kplayer on mac), there are: [error] 8887#8887: *373 ngx_http_vod_validate_streams: no matching streams were found while reading media header, client: 127.0.0.1, server: site.com, request: "GET /1574629202z199u.mp4/index.m3u8 HTTP/1.1", host: "site.com" this file (warning adult file): https://mega.nz/#!IdUz3Y6b!tEBch9O0gAUidyx4DKF4MVCSc5ch6p-7PtFsSU2p5E8

another error: [error] 31456#31456: *25 mp4_parser_parse_stsc_atom: chunk index 1 is smaller than the previous index 1 (1) while reading media header, client: 127.0.0.1, server: site.com, request: "GET /15000149776elvg.mp4/Frag-3-v1-a1.ts HTTP/1.1", host: "site.com" this file (warning adult file): https://mega.nz/#!BYch2CgJ!OUVcDVzrS-gHb4nWd0HlsxO-SZSINh11xjDUP-q-iEE

another error: [error] 31465#31465: *49 mp4_to_annexb_write: zero size packet while processing frames, client: 127.0.0.1, server: site.com, request: "GET /1564603166cmplj.mp4/Frag-4-v1-a1.ts HTTP/1.1", host: "site.com" this file (warning adult file):https://mega.nz/#!0ZNjkIwJ!uYDnEHyzsXtW7jiB60DK_g8n8OZqoMmx8ZnAkUivacE

            # vod settings
            vod_mode local;
            vod_fallback_upstream_location /fallback;
            vod_last_modified 'Sun, 19 Nov 2000 08:52:00 GMT';
            vod_last_modified_types *;
            vod_segment_duration 20000;
            vod_hls_absolute_master_urls off;
            vod_hls_absolute_index_urls off;
            vod_hls_container_format mpegts;
            vod_hls_absolute_iframe_urls off;
            vod_force_playlist_type_vod on;
            vod_hls_segment_file_name_prefix Frag;
            vod_open_file_thread_pool default_pool;
            # vod caches
            vod_metadata_cache metadata_cache 4096m;
            vod_response_cache response_cache 128m;
            vod_cache_buffer_size 512k;

[erankor]

Please provide a link to the MP4 file

[Vladislavik]

it is updated, look up

[erankor]

I checked the files -

1. This file is fragmented MP4 - this is not supported. You need to repackage it to regular MP4 (can be done with ffmpeg -a:c copy -v:c copy)
2. There is a problem with this file, ffmpeg complains about it too:

   # ffprobe.sh -i 15000149776elvg.mp4 -show_packets 2>&1 | grep STSC | head
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119468 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119467 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119466 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119465 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119464 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119463 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119462 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119461 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119460 is invalid (first=1 count=1 id=1)
   [mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119459 is invalid (first=1 count=1 id=1)

   Ffmpeg treats it as a warning and works around it, this module is more strict and fails the request. In this case as well, repackaging the MP4 will probably solve it.

3. This file is corrupt:

   # ffprobe.sh -i 1564603166cmplj.mp4 -show_frames 2>&1 | grep '@' | head
   [h264 @ 0x4572b80] Invalid NAL unit size (0 > 787).
   [h264 @ 0x4572b80] Error splitting the input into NAL units.
   [aac @ 0x46567c0] channel element 0.0 is not allocated
   [h264 @ 0x4572b80] Invalid NAL unit size (0 > 5019).
   [h264 @ 0x4572b80] Error splitting the input into NAL units.
   [aac @ 0x46567c0] channel element 0.0 is not allocated
   [h264 @ 0x4572b80] Invalid NAL unit size (0 > 2844).
   [h264 @ 0x4572b80] Error splitting the input into NAL units.
   [aac @ 0x46567c0] channel element 0.0 is not allocated
   [h264 @ 0x4572b80] Invalid NAL unit size (0 > 3213).

[joaoserra]

Hi @erankor , Just curious: Why isn't fragmented MP4s supported? Just because that feature is not top priority or is there an architectural problem with supporting this?

[erankor]

I looked into it at some point, the main problem is that the module depends on the ability to seek efficiently within the MP4. Some fragmented MP4 have a kind of index at the end (I don't remember the name of the atom...) that points to the different moof atoms within the file. It is possible to implement support for fMP4s that have this index, but it's not a small task, the module would need to -

1. Read the index atom, and derive from it the offsets of all moof atoms relevant to the request
2. Parse the relevant moof atoms to get the offset/size/duration of the frames
3. Since the module can't assume that the moof atoms are aligned with the segments, the code will need to handle all sorts of cases like reading frames from the middle of the moof, reading multiple moofs etc.

Other than the coding effort -

1. Many fMP4s don't have this index (the one shared here is such example) - those files will still be unsupported. When we initially considered fMP4 it was for live - unlike regular MP4, fMP4 can be easily appended. And, in this case, we won't have the index, or we'll have to implement some non-standard way of saving the index separately from the fMP4.
2. Some of the features of this module rely on the ability to read the metadata of ALL frames, such features will have to be disabled - for example, generating the iframe list and vod_manifest_segment_durations_mode accurate (these would need to read all moofs in the file, which means an unbounded number of reads/seeks...)

[Vladislavik]

1. This file is fragmented MP4 - this is not supported. You need to repackage it to regular MP4 (can be done with `ffmpeg -a:c copy -v:c copy`)

How to check with ffmpeg or ffprobe, that file is fMP4?

[erankor]

I don't know... I saw it in hex viewer :) If it contains moof atoms, it's fragmented.

[Vladislavik]

I don't know... I saw it in hex viewer :) If it contains moof atoms, it's fragmented.

can you check, what wrong with this file, on PC it is work perfect https://mega.nz/#!5VFiTSpb!CLy3l-Y4gPeqQPUIChHqYidRegAeYDxbXKOukZ4YpRI but videojs+hls.js can play it, theere is error (on Frag-4-v1-a1.ts): VIDEOJS: ERROR: (CODE:4 MEDIA_ERR_SRC_NOT_SUPPORTED) The media could not be loaded, either because the server or network failed or because the format is not supported. Object { code: 4, message: "The media could not be loaded, either because the server or network failed or because the format is not supported." } video.min.js:18:25414 TypeError: t is null videojs-hlsjs-plugin.1.js:8:6562 VIDEOJS: ERROR: (CODE:2 MEDIA_ERR_NETWORK) HLS.js error: networkError - fatal: true - manifestLoadError

[Vladislavik]

Adobe Media Server can play this file, on local PC player can play too, kaltura cant, please check, reason: https://mega.nz/#!QYEAGQyT!vbHQQJXep6IftasRMD2tIsRJb_8GF7C6azouueWcrsg 2020/01/05 09:41:46 [error] 5014#5014: *65544019 mp4_parser_parse_stsc_atom: chunk index 1 is smaller than the previous index 1 (2) while reading media header, client: 127.0.0.1, server: *, request: "GET /1553709802ceo8r.mp4/Frag-1-v1-a1.ts

[erankor]

I referred to this error above, repasting it here - "There is a problem with this file, ffmpeg complains about it too:

# ffprobe.sh -i 15000149776elvg.mp4 -show_packets 2>&1 | grep STSC | head
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119468 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119467 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119466 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119465 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119464 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119463 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119462 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119461 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119460 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x40b4700] STSC entry 119459 is invalid (first=1 count=1 id=1)

Ffmpeg treats it as a warning and works around it, this module is more strict and fails the request. In this case as well, repackaging the MP4 will probably solve it."

[Vladislavik]

what about 1553709802ceo8r.mp4 file?

[erankor]

Same...

ffprobe.sh -i 1553709802ceo8r.mp4 -show_packets 2>&1 | grep STSC | head
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119458 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119457 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119456 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119455 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119454 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119453 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119452 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119451 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119450 is invalid (first=1 count=1 id=1)
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x3d63700] STSC entry 119449 is invalid (first=1 count=1 id=1)

[Vladislavik]

hi, there is error: on Frag-47-v1-a1.ts next Frag work... 2020/05/13 23:57:52 [error] 3328#3328: 84751797 mp4_to_annexb_write: zero size packet while processing frames, client: 127.0.0.1, server: .., request: "GET /1559582346ddedh.mp4/Frag-47-v1-a1.ts HTTP/1.1", host: ""

on this config:

vod_mode local;
vod_fallback_upstream_location /fallback;
vod_last_modified 'Sun, 19 Nov 2000 08:52:00 GMT';
vod_last_modified_types *;
vod_segment_duration 20000;
vod_hls_absolute_master_urls off;
vod_hls_absolute_index_urls off;
vod_hls_container_format mpegts;
vod_hls_absolute_iframe_urls off;
vod_force_playlist_type_vod on;
vod_hls_segment_file_name_prefix Frag;
vod_open_file_thread_pool default_pool;
# vod caches
vod_metadata_cache metadata_cache 6144m;
vod_response_cache response_cache 128m;
vod_performance_counters perf_counters;
vod_output_buffer_pool 64k 32;
vod_hls_mpegts_align_frames on; 
vod_hls_mpegts_interleave_frames on;

file: https://mega.nz/file/VEES3IpZ#aRHZiAEZGF0bge4pZoTj_iAbKJ3nqwPBHZWJNXEsZ80

i have try ffprobe, but cant see any errors: ffprobe -i /1559582346ddedh.mp4 ffprobe version 3.1.4-static http://johnvansickle.com/ffmpeg/ Copyright (c) 2007-2016 the FFmpeg developers built with gcc 5.4.1 (Debian 5.4.1-2) 20160904 configuration: --enable-gpl --enable-version3 --enable-static --disable-debug --enable-libmp3lame --enable-libx264 --enable-libx265 --enable-libwebp --enable-libspeex --enable-libvorbis --enable-libvpx --enable-libfreetype --enable-fontconfig --enable-libxvid --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libtheora --enable-libvo-amrwbenc --enable-gray --enable-libopenjpeg --enable-libopus --enable-libass --enable-gnutls --enable-libvidstab --enable-libsoxr --enable-frei0r --enable-libfribidi --disable-indev=sndio --disable-outdev=sndio --enable-librtmp --enable-libzimg --cc=gcc-5 --disable-ffplay libavutil 55. 28.100 / 55. 28.100 libavcodec 57. 48.101 / 57. 48.101 libavformat 57. 41.100 / 57. 41.100 libavdevice 57. 0.101 / 57. 0.101 libavfilter 6. 47.100 / 6. 47.100 libswscale 4. 1.100 / 4. 1.100 libswresample 2. 1.100 / 2. 1.100 libpostproc 54. 0.100 / 54. 0.100 Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/1559582346ddedh.mp4': Metadata: major_brand : isom minor_version : 512 compatible_brands: isomiso2avc1mp41 encoder : Lavf57.20.100 Duration: 00:43:43.38, start: 0.000000, bitrate: 1150 kb/s Stream #0:0(eng): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 640x420 [SAR 1:1 DAR 32:21], 1017 kb/s, 25 fps, 25 tbr, 10k tbn, 50 tbc (default) Metadata: handler_name : VideoHandler Stream #0:1(eng): Audio: aac (LC) (mp4a / 0x6134706D), 48000 Hz, stereo, fltp, 127 kb/s (default) Metadata: handler_name : SoundHandler

[erankor]

You didn't test it correctly with ffprobe, you should run it with -show_frames (pipe the output to some file or /dev/null), surely you will get errors...

[Vladislavik]

Browser load and play this video, you can try on mega hosting watch it online, but some frames are skiped, hls player freeze on skiped frames, can kaltura be fixed to skip this frames too, to play video full and no freeze on this moments? It is many videos on internet with same bugs and kaltura cant work with it....

[erankor]

I'm sorry, but I have no interest in trying to work around corrupt files.

[Vladislavik]

Can it be done with payment?

[Vladislavik]

Have you thought about paying to add frame skipping on damaged files? I have a lot of these files, and it's terrible when the video just freezes, when I use pseudo mp4 streaming - such frames are simply skipped

[erankor]

Sorry, I can't do it

[Vladislavik]

HI, can you check why this video work fine on browser, but when it played via kaltura HLS, it is only audio, no video.

https://mega.nz/file/AAlE1YwY#eR6bB9CGKXblj_wjarOa6d34ho3X0ebY47svZuOewgc mega player played this video fine too

[Vladislavik]

Hi, can you please check this video: https://mega.nz/file/gIkwXbqb#lJN8khNPM1poChHM8PEAC2bYCWa7sJ6bIPOWNW9EZ6w

it work fine in firefox with hls.js 1.1.2, but when i try to play this video in chrome with same hls.js it is many errors 'Buffer stalled error' on times: 1:39 2:20 2:59 4:20

it can be fixed only with

vod_align_segments_to_key_frames: on 
vod_manifest_segment_durations_mode: accurate

but with this settings long videos cant be played, like if video 9 hours, module start return 404 when request index.m3u8 file

There is chrome timeline, and it is holes on 10 segment

There is firefox timeline and it is no holes on 10 segment with fully same kaltura config