Invalid offset in track--large files

[Algebro7]

Hey,

I'm trying to recover a very large mp4 file (~13GB) and untrunc can't seem to handle it. I've tried running with -i and -a against a known-good mp4 and I'm getting the same errors. I have plenty of RAM (~32gb) and it only uses up to 20GB or so when it's running, so it doesn't seem to be a memory issue. The error output looks like a 32 bit integer is wrapping around and coming back negative:

Composition time offset atom found. Out of order samples possible.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'Hellblade_Video_1.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.84.100
  Duration: 01:50:19.03, start: 0.000000, bitrate: 20506 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 20368 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 122 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
found avcC after: 102
remaining len:50
parsing avcC ...
len_sps: 26
decoding SPS ...
log2_max_frame_num: 5
Info: avcC got decoded
i = 25078
offset = -2147278276
mdat->start = 40
mdat->length = 16953873145
offset - mdat->start = -2147278316
Invalid offset in track!

Looking at the code, I'm seeing lots of 32 bit addresses, particularly in bitmasking operations in codec.cpp, so I'm wondering if the current codebase just can't handle files over 0xFFFFFFFF in size? If that's the case, how much work do you think it would be to patch in support for bigger files?

[Algebro7]

Just an update--I tried again with a smaller "working" file and untrunc runs and finishes, but the _fixed .mp4 starts erroring out about 1/3 of the way through with messages like this:

[h264 @ 0x7fe06aab4100] Invalid NAL unit size (-2117781260 > 52042).
[h264 @ 0x7fe06aab4100] Error splitting the input into NAL units.

Interestingly, 1/3rd of the file is about 4GB, which would be the maximum size with 32 bit offsets. Do you think that could be the issue?

[anthwlock]

Have you tried to change codec.cpp:70 int offset into int64_t offset? I dont

Not related to your problem, but do you use ponchio/untrunc? Because this branch shouldn't print messages from the libav h264 decoder and shouldn't consume more than 10%*input size.

[anthwlock]

Sadly I don't have a >4GB mp4 file but the notes at FILE MEDIA DATA in this text seem to be releveant to this problem.

[Algebro7]

Not related to your problem, but do you use ponchio/untrunc? Because this branch shouldn't print messages from the libav h264 decoder and shouldn't consume more than 10%*input size.

Sorry, that output was actually from @davitf's fork. I'm getting a different error trying to use this fork which I've pasted below. a _fixed.mp4 file gets created but it's only 200 KB.

$ untrunc/untrunc -vv sample.mp4 2018-08-03\ 14-19-25.mp4 
Info: reading sample.mp4
Info: parsing healthy moov atom ... 

start_ = 0
requests: 4 at offset : 0
length_ = 32
requests: 4 at offset : 4
name_ = ftyp
requests: 24 at offset : 8

start_ = 32
requests: 4 at offset : 32
length_ = 8
requests: 4 at offset : 36
name_ = free
requests: 0 at offset : 40

start_ = 40
requests: 4 at offset : 40
length_ = 173990742
requests: 4 at offset : 44
name_ = mdat
requests: 173990734 at offset : 48
reallocating the file buffer

start_ = 173990782
requests: 4 at offset : 0
length_ = 277285
requests: 4 at offset : 4
name_ = moov

start_ = 173990790
requests: 4 at offset : 8
length_ = 108
requests: 4 at offset : 12
name_ = mvhd
requests: 100 at offset : 16

start_ = 173990898
requests: 4 at offset : 116
length_ = 218029
requests: 4 at offset : 120
name_ = trak

start_ = 173990906
requests: 4 at offset : 124
length_ = 92
requests: 4 at offset : 128
name_ = tkhd
requests: 84 at offset : 132

start_ = 173990998
requests: 4 at offset : 216
length_ = 36
requests: 4 at offset : 220
name_ = edts

start_ = 173991006
requests: 4 at offset : 224
length_ = 28
requests: 4 at offset : 228
name_ = elst
requests: 20 at offset : 232

start_ = 173991034
requests: 4 at offset : 252
length_ = 217893
requests: 4 at offset : 256
name_ = mdia

start_ = 173991042
requests: 4 at offset : 260
length_ = 32
requests: 4 at offset : 264
name_ = mdhd
requests: 24 at offset : 268

start_ = 173991074
requests: 4 at offset : 292
length_ = 45
requests: 4 at offset : 296
name_ = hdlr
requests: 37 at offset : 300

start_ = 173991119
requests: 4 at offset : 337
length_ = 217808
requests: 4 at offset : 341
name_ = minf

start_ = 173991127
requests: 4 at offset : 345
length_ = 20
requests: 4 at offset : 349
name_ = vmhd
requests: 12 at offset : 353

start_ = 173991147
requests: 4 at offset : 365
length_ = 36
requests: 4 at offset : 369
name_ = dinf

start_ = 173991155
requests: 4 at offset : 373
length_ = 28
requests: 4 at offset : 377
name_ = dref
requests: 20 at offset : 381

start_ = 173991183
requests: 4 at offset : 401
length_ = 217744
requests: 4 at offset : 405
name_ = stbl

start_ = 173991191
requests: 4 at offset : 409
length_ = 152
requests: 4 at offset : 413
name_ = stsd
requests: 144 at offset : 417

start_ = 173991343
requests: 4 at offset : 561
length_ = 24
requests: 4 at offset : 565
name_ = stts
requests: 16 at offset : 569

start_ = 173991367
requests: 4 at offset : 585
length_ = 540
requests: 4 at offset : 589
name_ = stss
requests: 532 at offset : 593

start_ = 173991907
requests: 4 at offset : 1125
length_ = 78040
requests: 4 at offset : 1129
name_ = ctts
requests: 78032 at offset : 1133

start_ = 174069947
requests: 4 at offset : 79165
length_ = 68980
requests: 4 at offset : 79169
name_ = stsc
requests: 68972 at offset : 79173

start_ = 174138927
requests: 4 at offset : 148145
length_ = 40752
requests: 4 at offset : 148149
name_ = stsz
requests: 40744 at offset : 148153

start_ = 174179679
requests: 4 at offset : 188897
length_ = 29248
requests: 4 at offset : 188901
name_ = stco
requests: 29240 at offset : 188905

start_ = 174208927
requests: 4 at offset : 218145
length_ = 59042
requests: 4 at offset : 218149
name_ = trak

start_ = 174208935
requests: 4 at offset : 218153
length_ = 92
requests: 4 at offset : 218157
name_ = tkhd
requests: 84 at offset : 218161

start_ = 174209027
requests: 4 at offset : 218245
length_ = 36
requests: 4 at offset : 218249
name_ = edts

start_ = 174209035
requests: 4 at offset : 218253
length_ = 28
requests: 4 at offset : 218257
name_ = elst
requests: 20 at offset : 218261

start_ = 174209063
requests: 4 at offset : 218281
length_ = 58870
requests: 4 at offset : 218285
name_ = mdia

start_ = 174209071
requests: 4 at offset : 218289
length_ = 32
requests: 4 at offset : 218293
name_ = mdhd
requests: 24 at offset : 218297

start_ = 174209103
requests: 4 at offset : 218321
length_ = 45
requests: 4 at offset : 218325
name_ = hdlr
requests: 37 at offset : 218329

start_ = 174209148
requests: 4 at offset : 218366
length_ = 58785
requests: 4 at offset : 218370
name_ = minf

start_ = 174209156
requests: 4 at offset : 218374
length_ = 16
requests: 4 at offset : 218378
name_ = smhd
requests: 8 at offset : 218382

start_ = 174209172
requests: 4 at offset : 218390
length_ = 36
requests: 4 at offset : 218394
name_ = dinf

start_ = 174209180
requests: 4 at offset : 218398
length_ = 28
requests: 4 at offset : 218402
name_ = dref
requests: 20 at offset : 218406

start_ = 174209208
requests: 4 at offset : 218426
length_ = 58725
requests: 4 at offset : 218430
name_ = stbl

start_ = 174209216
requests: 4 at offset : 218434
length_ = 103
requests: 4 at offset : 218438
name_ = stsd
requests: 95 at offset : 218442

start_ = 174209319
requests: 4 at offset : 218537
length_ = 32
requests: 4 at offset : 218541
name_ = stts
requests: 24 at offset : 218545

start_ = 174209351
requests: 4 at offset : 218569
length_ = 28
requests: 4 at offset : 218573
name_ = stsc
requests: 20 at offset : 218577

start_ = 174209379
requests: 4 at offset : 218597
length_ = 29252
requests: 4 at offset : 218601
name_ = stsz
requests: 29244 at offset : 218605

start_ = 174238631
requests: 4 at offset : 247849
length_ = 29248
requests: 4 at offset : 247853
name_ = stco
requests: 29240 at offset : 247857

start_ = 174267879
requests: 4 at offset : 277097
length_ = 26
requests: 4 at offset : 277101
name_ = sgpd
requests: 18 at offset : 277105

start_ = 174267905
requests: 4 at offset : 277123
length_ = 28
requests: 4 at offset : 277127
name_ = sbgp
requests: 20 at offset : 277131

start_ = 174267933
requests: 4 at offset : 277151
length_ = 36
requests: 4 at offset : 277155
name_ = udta
requests: 28 at offset : 277159

start_ = 174267969
requests: 4 at offset : 277187
length_ = 98
requests: 4 at offset : 277191
name_ = udta
requests: 90 at offset : 277195
Composition time offset atom found. Out of order samples possible.
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from 'sample.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 512
    compatible_brands: isomiso2avc1mp41
    encoder         : Lavf57.84.100
  Duration: 00:02:49.72, start: 0.000000, bitrate: 8214 kb/s
    Stream #0:0(und): Video: h264 (High) (avc1 / 0x31637661), yuv420p, 1920x1080, 8073 kb/s, 60 fps, 60 tbr, 15360 tbn, 120 tbc (default)
    Metadata:
      handler_name    : VideoHandler
    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 128 kb/s (default)
    Metadata:
      handler_name    : SoundHandler
found avcC after: 102
remaining len:50
parsing avcC ...
len_sps: 26
decoding SPS ...
log2_max_frame_num: 5
Info: avcC got decoded
found avcC after: 52
remaining len:51
parsing esds ...
esds:
 object_type: 
avg_bitrate: 128012
max_bitrate: 128012
frame_length_flag: 0Info: audio-config (esds) got decoded
Info: parsing mdat from truncated file ... 
start_ = 0
requests: 4 at offset : 0
length_ = 32
requests: 4 at offset : 4
name_ = ftyp
start_ = 32
requests: 4 at offset : 32
length_ = 8
requests: 4 at offset : 36
name_ = free
start_ = 40
requests: 4 at offset : 40
length_ = 0
requests: 4 at offset : 44
name_ = mdat
mdat->contentSize = 13387956224

(reading element from mdat)
Offset: 0 ae020000 ffff0506
Track codec: mp4a
Failure because of NULL header
Track codec: avc1
avc1: Match with 0 header
sps_info (before): 1 4 5 0
sps_info (after):  1 5 5 0
---
Length: 690
Ref idc: 0
Nal type: 6
Partial avc1-length: 690
---
Length: 196831
Ref idc: 3
Nal type: 5
First mb: 0
pic paramter set id: 0
Frame num: 1
Poc lsb: 23
Partial avc1-length: 197521
---
Length: 3601
Ref idc: 2
Nal type: 1
First mb: 0
pic paramter set id: 0
Frame num: 2
Poc lsb: 8
Different frame number
frame-length: 197521
- found as avc1

(reading element from mdat)
Offset: 197521 d0e0000 18249a41
Track codec: mp4a
Failure because of NULL header
Track codec: avc1
avc1: Match with 0 header
---
Length: 3601
Ref idc: 2
Nal type: 1
First mb: 0
pic paramter set id: 0
Frame num: 2
Poc lsb: 8
Partial avc1-length: 3601
---
Length: 330
Ref idc: 2
Nal type: 1
First mb: 0
pic paramter set id: 0
Frame num: 4
Poc lsb: 4
Different frame number
frame-length: 3601
- found as avc1

(reading element from mdat)
Offset: 201122 46010000 42429e41
Track codec: mp4a
Failure because of NULL header
Track codec: avc1
avc1: Match with 0 header
---
Length: 330
Ref idc: 2
Nal type: 1
First mb: 0
pic paramter set id: 0
Frame num: 4
Poc lsb: 4
Partial avc1-length: 330
---
First byte expected 0
failed parsing nal-header
frame-length: 330
- found as avc1

(reading element from mdat)
Offset: 201452 60041021 1c8c
Track codec: mp4a
mp4a: Success because of large s value
[aac @ 0x55e9986bb0c0] Multiple frames in a packet.
nb_samples: 1024
frame-length: 6

(reading element from mdat)
Offset: 201458 72000000 44619e01
Track codec: mp4a
Failure because of NULL header
Track codec: avc1
avc1: Match with 0 header
---
Length: 118
Ref idc: 0
Nal type: 1
First mb: 0
pic paramter set id: 0
Frame num: 6
Poc lsb: 2
Partial avc1-length: 118
---
First byte expected 0
failed parsing nal-header
frame-length: 118
- found as avc1

(reading element from mdat)
Offset: 201576 60041021 1c8c
Track codec: mp4a
mp4a: Success because of large s value
[aac @ 0x55e9986bb0c0] Got unexpected packet size after a partial decode
nb_samples: 0
frame-length: -22
Error: Invalid length: -22. Wrong match in track: 0
Track codec: avc1
avc1: failed for not particular reason
Error: unable to find correct codec -> premature end
mdat->file_end: 201624
Info: Found 5 packets ( mp4a: 1 avc1: 4 avc1-keyframes: 1 )
Info: Duration of mp4a: 23ms  (23 ms)
Info: Duration of avc1: 68ms  (68 ms)
Info: saving 2018-08-03 14-19-25.mp4_fixed.mp4
requests: 201576 at offset : 48

[anthwlock]

Could you downgrade libavcodec (to 3.2.12) and try again? This issue seems related.

[Algebro7]

Ok, I've built 3.2.12 from source and I also had to symlink the following shared libraries to get untrunc to run:

lrwxrwxrwx 1 algebro algebro     22 Aug 14 12:01 libavcodec.so.58 -> /usr/lib/libavcodec.so
lrwxrwxrwx 1 algebro algebro     23 Aug 14 12:01 libavformat.so.58 -> /usr/lib/libavformat.so
lrwxrwxrwx 1 algebro algebro     21 Aug 14 12:02 libavutil.so.56 -> /usr/lib/libavutil.so

untrunc finished creating the file, but the same issue occurred where at about 27 minutes in, the video stops and the vlc command line starts throwing the following error repeatedly:

[h264 @ 0x7f7d1eb28a40] Invalid NAL unit size (1454516137 > 111046).
[h264 @ 0x7f7d1eb28a40] Error splitting the input into NAL units.
[h264 @ 0x7f7d1eb450c0] Invalid NAL unit size (1148675787 > 149088).
[h264 @ 0x7f7d1eb450c0] Error splitting the input into NAL units.
[h264 @ 0x7f7d1eb61800] Invalid NAL unit size (-692919758 > 128488).
[h264 @ 0x7f7d1eb61800] Error splitting the input into NAL units.
[...]

Once again, 27 minutes is about where the offset should hit 0xFFFFFFFF. Do you think that explanation is plausible and that I should try changing offset to a 64 bit integer? And would that break any of the 32 bit masking operations that are going on elsewhere?

As for changing codec.cpp:70 to int64_t, it looks like offsets is a vector<int>, and several other places in the code that touch this vector use 32 bit integers (such as track.cpp:50-95, track.h, etc). Are you able to tell which lines need to be updated to use 64 bit ints without breaking anything else?

[anthwlock]

Yes, just give it a try. I looked at the bitwise operations on mask0_ and mask1_ and they seem useless to me. It basically checks whether (x & s) == s & (x & s). Feel free to remove those and open a PR when you are done.

[Algebro7]

I changed these two lines at codec:cpp:69-70 to int64_t:

    for(int64_t i = 0; i < offsets.size(); i++) {
        int64_t offset = offsets[i];

The process seemed to run WAY faster for some reason (like 3 minutes instead of 20) but the result was the same. The video length correctly shows 1:37:00 but at about the 27 minute mark VLC stops and shows the NAL unit size errors. I edited my comment above just before you responded so you might not've seen it, but I'm wondering if there are other places in the code like track.cpp and track.h where we need to convert to 64 bit ints. Thoughts?

[anthwlock]

In Track::saveChunkOffsets() it looks like untrunc replaces the co64 with stco. This is likely a problem. See here if you dont know what this is.

[anthwlock]

I think you will have to add Atom::writeInt overload for int64_t.

[vman411gamer]

Sorry this doesn't really have to do with this project, but @Algebro7 I was having similar issues as you (having a large file myself), and I was messing around with the C code for a while, but eventually I continued looking for a fix and found this mp4fixer: https://github.com/bookkojot/mp4fixer

Run it like it says: perl fixer.pl <good_file.mp4> <bad_file.mp4> <output_prefix>

At this point you can test the output being able to play at all with ffplay [output_prefix]-out-video.h264

Now, if you know the audio is encoded with mp3, which it should be, you can just rename the [output_prefix]-out-audio.raw file to an .mp3, then...

Put the audio and video back together with ffmpeg -i [output_prefix]-out-video.h264 -i [output_prefix]-out-audio.mp3 -c:v copy -c:a copy -f mp4 output_file.mp4

I was able to recover up to where I ran out of disk space, when I wasn't able to play it at all initially.

If it isn't mp3 audio, you'll have to follow the mp4fixer instructions.

Hope this helps.

[Algebro7]

Thanks a lot for the suggestion @vman411gamer . I tried mp4fixer and unfortunately the results don't look good--ffplay can't play the video at all and aacfixer was only able to recover about 30 seconds of audio before crashing with a segfault. I've opened a ticket with the mp4fixer maintainer but I may keep trying to fix untrunc to work with bigger files if the mp4fixer maintainer isn't able to help.

[Algebro7]

@anthwlock,

I'm trying to compile untrunc and link against libavcodec 3.2.12 so I can debug this as I try to add 64 bit support, but I'm having some issues and was wondering if you'd be able to tell me where I'm going wrong. I'm trying to keep ffmpeg completely separated from my system directories so that it doesn't interfere with other applications that I need the latest version for.

First, I downloaded the ffmpeg n3.2.12 source and configured with the following options:

./configure --enable-gpl --enable-libx264 --enable-nonfree --prefix=/home/algebro/Documents/projects/ffmpeg/install --disable-static --enable-avresample --enable-fontconfig --enable-gmp --enable-gnutls --enable-gpl --enable-ladspa --enable-libass --enable-libbluray --enable-libfreetype --enable-libfribidi --enable-libgsm --enable-libiec61883 --enable-libmodplug --enable-libmp3lame --enable-libopencore_amrnb --enable-libopencore_amrwb --enable-libopenjpeg --enable-libopus --enable-libpulse --enable-libsoxr --enable-libspeex --enable-libssh --enable-libtheora --enable-libv4l2 --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxcb --enable-libxvid --enable-nvenc --enable-omx --enable-shared --enable-version3

After running make and make install, I then compiled untrunc and tried linking to the custom-built ffmpeg libraries:

LD_LIBRARY_PATH=/home/algebro/Documents/projects/ffmpeg/install/lib LIBRARY_PATH=/home/algebro/Documents/projects/ffmpeg/install/lib g++ -ggdb -o untrunc *.cpp -O3 -I/home/algebro/Documents/projects/ffmpeg/install/include -L/home/algebro/Documents/projects/ffmpeg/install/lib -lswresample -lavformat -lavcodec -lavutil  -L/usr/lib -lbz2 -llzma -lX11 -lvdpau -ldl  -lm -lz

The binary compiles successfully, but when I run it I get the following output:

$ LD_LIBRARY_PATH=/home/algebro/Documents/projects/ffmpeg/install/lib ./untrunc /data/hellblade/sample.mp4 /data/hellblade/2018-08-03\ 14-19-25.mp4
Info: reading /data/hellblade/sample.mp4
Info: parsing healthy moov atom ... 
Composition time offset atom found. Out of order samples possible.
[h264 @ 0x55b077d2cc60] Warning: not compiled with thread support, using thread emulation
[aac @ 0x55b077d2f520] Warning: not compiled with thread support, using thread emulation
[h264 @ 0x55b077d2c0e0] Warning: not compiled with thread support, using thread emulation
Info: avcC got decoded
[aac @ 0x55b077d2f0a0] Warning: not compiled with thread support, using thread emulation
Info: audio-config (esds) got decoded
Info: parsing mdat from truncated file ... 
[aac @ 0x55b077d2f0a0] Multiple frames in a packet.
Warning: Different poc lsb
Warning: Different poc lsb
Warning: Different poc lsb
Warning: Different nal type (5, 1)
[repeats Warning: Different poc lsb] 

Can you tell if I did anything wrong during those steps? I'm guessing I messed something up with the linker but I don't have a ton of experience compiling with non-standard library paths.

Thanks!

edit: My bad, looks like it was actually working all along. The output just looked different because I wasn't passing -vv like I was originally.

[anthwlock]

There is a section "Manual Libav installation" on ponchio/untrunc. Have you alredy made progress with adding 64-bit addressing support?

[Algebro7]

I have begun at https://github.com/Algebro7/untrunc/tree/64bit-offsets but I'm still trying to trace out what all needs to be changed. It looks like a -lot- of decisions were made based on 32 bit addresses throughout the codebase but I'm not sure all of it needs changed.

I may have broken things with this commit but the program still runs and produces the same result as your master branch (a 1:37:00 video that stops and throws NAL unit size errors at 27 minutes). If you have any ideas or tips that would be great (I don't have a lot of experience with C++, especially pre-C++11). I've also been reading the resources you linked for the mp4 format which have been very helpful.

[anthwlock]

I think you also have to resize the content_ of the co64 object in Track::saveChunkOffsets().

I dont understand why you cast sizes_.size() in Track::saveSampleTimes() to int. Does this change anything?

Other than this, your changes seem reasonable. If it still doesn't work, maybe printing out the contents of the relevant atoms will give you a clue.

[Algebro7]

I think you also have to resize the content_ of the co64 object in Track::saveChunkOffsets().

Yeah, I was trying to figure that part out but during debugging, I set a breakpoint on co64->writeInt((int64_t)offsets_.size(), 8); and never hit it--I only drop into the else which I guess suggests that the atom is a stco and not a co64? I haven't figured out yet if that's because that's what's in the file or if some other function already converted the co64 atoms to stcos.

It's also possible that I'm misunderstanding Atoms work in general :stuck_out_tongue:

I dont understand why you cast sizes_.size() in Track::saveSampleTimes() to int. Does this change anything?

I had to do this because of the Atom::writeInt() overload--the compiler was complaining that the function call is ambiguous if I didn't explicitly cast it to int.

[Algebro7]

@anthwlock, I was just doing some more debugging and I noticed something interesting. I've been using a very small sample mp4 as the "good" input, because a large "good" file (over 4GB) terminates with the following error:

found avcC after: 102
remaining len:50
parsing avcC ...
len_sps: 26
decoding SPS ...
log2_max_frame_num: 5
Info: avcC got decoded
i = 25079
offset = -2147201660
mdat->start = 40
mdat->length = 16953873145
offset - mdat->start = -2147201700
Invalid offset in track!

I ran untrunc with -a -vv against both a large "good" file and a small "good" file and noticed that there are no co64 atoms in the small file, but stcos instead. There are co64 atoms in the big file. I'm guessing OBS used stcos to save space when the video was less than 4GB.

So, basically, I think we have to fix the parsing of big input files. The snippet above looks like an integer overflow in Codec::parse, although I'm not sure why because in my branch I've already made offset an int64_t:

    //this was a stupid attempt at trying to detect packet type based on bitmasks
    mask1_ = 0xffffffff;
    mask0_ = 0xffffffff;
    //build the mask:
    for(int i = 0; i < offsets.size(); i++) {
        int64_t offset = offsets[i];
        if(offset < mdat->start_ || offset - mdat->start_ > mdat->length_) {
            cout << "i = " << i;
            cout << "\noffset = " << offset
                 << "\nmdat->start = " << mdat->start_
                 << "\nmdat->length = " << mdat->length_
                 << "\noffset - mdat->start = " << offset - mdat->start_;
            cout << "\nInvalid offset in track!\n";
            exit(0);
        }

I'll continue debugging and let you know if I figure anything else out.

[anthwlock]

Maybe you should check if track.offsets_.back() >= (1<<32) instead of weither the healthy file used co64. But in case your healthy file is also bigger than 4 GB it shouldn't matter.

[anthwlock]

There is no Atom::readInt64() to use for Track::getChunkOffsets(). This is why you still get the overflow.

[Algebro7]

There is no Atom::readInt64() to use for Track::getChunkOffsets(). This is why you still get the overflow.

Thanks! I've written Atom::readInt64() here:

int64_t Atom::readInt64(int64_t offset) {
    return swap64(*(int64_t *)&(content_[offset]));
}

Do I need to change the argument getting passed to it in Track::getChunkOffsets()? I'm not sure what the 12 is for:

        for(int64_t i = 0; i < nchunks; i++)
            chunk_offsets.push_back(co64->readInt64(12 + i*8));

[anthwlock]

I think this has to do with the representation of int64_t and their conversion to int32_t. When it adds +4 bytes to the read-offset it get's the lower half of the int64_t. So now where we read int64_t the +4 are not needed anymore.

[Algebro7]

Does Atom::parseHeader:22 need updated to read an int64_t? I've modified the Track::getChunkOffsets() function to call readInt64(8 + i*8)) but now we're running into a condition where offset - mdat->start_ == 0x134000000f1, which is quite a bit bigger than mdat->length_, so it's failing with invalid offset.

update: I tried adding changing Atom::parseHeader:22 to call file.readInt64() instead of file.readInt() and now I'm getting a SIGABRT: std::bad_alloc for some reason. It looks from the output like it's parsing the length as 139157862768, which would be 139 gigabytes:

start_ = 0
requests: 8 at offset : 0
length_ = 139157862768
requests: 4 at offset : 8
name_ = isom
terminate called after throwing an instance of 'std::bad_alloc'
  what():  std::bad_alloc
Signal: SIGABRT (Aborted)

[anthwlock]

If length_ == 0x1 then yes, see here.

I think offset - mdat->start_ == 0x134000000f1 is alredy too big.

[anthwlock]

Maybe the +12 has another reason. Since in MP4 integers are encoded in big endian.

[anthwlock]

Did you get past the SIGABRT: std::bad_alloc ?

[Algebro7]

No, I did some more tracing of the code and I don't believe Atom::parseHeader() needs to read an int64_t for length_, as large Atoms use 1 as the length header field. Is this correct?

I've just pushed my current WIP but I'm starting to get a little lost with all the Atom structures and type conversions :disappointed: The branch currently builds and runs but errors out due to invalid offsets in Codec::parse() here:

    for(uint64_t i = 0; i < offsets.size(); i++) {
        int64_t offset = offsets[i];
        if(offset < mdat->start_ || offset - mdat->start_ > mdat->length_) {
            cout << "i = " << i;
            cout << "\noffset = " << offset
                 << "\nmdat->start = " << mdat->start_
                 << "\nmdat->length = " << mdat->length_
                 << "\noffset - mdat->start = " << offset - mdat->start_;
            cout << "\nInvalid offset in track!\n";
            exit(0);
        }

Here is a view of the local variables...offset is clearly messed up but I'm not sure why:

[anthwlock]

length_ == 1 means that you have to read length_ at a different position. It is explained here. Untrunc does this alredy.

[anthwlock]

I see your Track::offsets_ has still type of vector<int>. Maybe this is a problem?

[Algebro7]

Are you looking at the 64bit-offsets branch? It looks like a vector<int64_t>/vector<long> to me

[Algebro7]

I've been doing some more debugging and it looks like something is wrong with the Atom::readInt64() function. Here are some of the values it's adding while looping through the co64 condition in Track::getChunkOffsets(). Sorry for all the screenshots--Im not sure if there is a better way to share output from Clion's debugger.

Do you have any ideas what could be going wrong? 0x2e8a400000000 doesn't look like it could be a correct offset but maybe I'm wrong. When writing the readInt64 function I essentially copied what you had for readInt but called swap64 instead and cast it to an int64_t:

int64_t Atom::readInt64(int64_t offset) {
    return swap64(*(int64_t *)&(content_[offset]));
}

[anthwlock]

I think you are using a wrong offset in track::getChunkOffsets. Check out the new commits on master, I hope it works now.

[Algebro7]

@anthwlock the video was recovered, thank you so much!!! VLC has the duration of the video wrong it seems--it's reporting 1 hour 50 minutes, but the video is actually only 1 hour 25 minutes or so. Just figured I'd point that out in case it was something you felt like investigating.

The commit to not store "mdat" in memory was also a really good call!

[anthwlock]

I'm glad to hear that. I think some audio packets get matched as video packets, which is why the avc1 track is too long. Feel free to open another issue, so I don't forget this problem exists.