RuntimeException - UInt64

Hangs when trying to select any individual sample in Track1 tab - throws RuntimeException:

Exception in thread "JavaFX Application Thread" java.lang.RuntimeException: I don't know how to deal with UInt64! long is not sufficient and I don't want to use BigInt at com.coremedia.iso.IsoTypeReader.readUInt64(IsoTypeReader.java:105) at com.coremedia.iso.AbstractBoxParser.parseBox(AbstractBoxParser.java:84) at com.googlecode.mp4parser.BasicContainer.next(BasicContainer.java:185) at com.googlecode.mp4parser.BasicContainer.hasNext(BasicContainer.java:161) at com.googlecode.mp4parser.util.LazyList$1.hasNext(LazyList.java:55) ...

4K MP4 file with no audio.

Hi, this exception is thrown when a 64bit value between 9223372036854775808 and 18446744073709551616 is read. This is out of range for a 64bit signed long and I haven't implemented that because I have never seen that for real - typically the exception is thrown the underlying parser reads 'garbage' from a file cause some offsets were wrong. Especially in your case it looks like a problem with a file because in line 84 of the AbstractBoxParser the size of a box is read. Or is your file larger than 8000 petabyte? If you can it would be great if you'd make the file available so that I can check the file but typically 4k cannot be shared with anyone I guess. It would also help if you get me the hex dump of a few bytes: somewhere in the file there is the string 'mdat'. To check if the file is corrupt or if my parser has a probelm I'd need 4 bytes before mdat (in expected 0001) and 32 bytes after the mdat string. [.... mdat .... .... .... .... .... .... .... .... ]

Sadly, the files aren't petabyte sized. They're a mere 1.2GB and 4.0GB - I'll upload to google drive and share the link once they're up in a few hours (I've only got poor man DSL)... Content isn't anything exotic, a pleasant drone flight showing the autumn colours. The files were created by a hardware encoder that breaks input every 4GB, so they're in sequence. Both of these files (and any other that the drone creates) behave the same way in isoviewer, but at least the 4GB doesn't have any issues that I know about. There's something wrong with the 1.2GB file, and I'm trying to figure out what so I can fix it as I'm starting to get quite a few from the drone that have the same problem. Was hoping isoviewer could help me pinpoint where/why things have gone wrong.

(Sadly, I'm not an expert at MP4, but I wrote a crappy little program that traces through all the frame pointers and know that the broken files typically have one that's shorter than what is specified in the first few bytes of the frame, but removing the short frame and adjusting the mdat pointers doesn't fix things... I also know that the picture data itself is still intact in the entire file - someone wrote an open-source program that pulls the frames out of the MP4 and puts them into a raw h264 file, which will is complete... But who wants that!)

Anyway, from the 4.0GB file, the 4 bytes before the mdat (at byte 36) are FA 53 F4 E2 (4199806178), which points to the 36 bytes before the moov at the end of the file. The whole 36 bytes around the mdat are:

FA 53 F4 E2 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

From the 1.2GB file, the 4 bytes before the mdat (also at byte 36) are 46 DB B5 10 (1188803856), also points to the 36 bytes before the moov at the end of the file...

The whole 36 bytes are:

46 DB B5 10 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

On Thu, Oct 29, 2015 at 2:40 AM, Sebastian Annies notifications@github.com wrote:

Hi, this exception is thrown when a 64bit value between 9223372036854775808 and 18446744073709551616 is read. This is out of range for a 64bit signed long and I haven't implemented that because I have never seen that for real

typically the exception is thrown the underlying parser reads 'garbage' from a file cause some offsets were wrong. Especially in your case it looks like a problem with a file because in line 84 of the AbstractBoxParser the size of a box is read. Or is your file larger than 8000 petabyte? If you can it would be great if you'd make the file available so that I can check the file but typically 4k cannot be shared with anyone I guess. It would also help if you get me the hex dump of a few bytes: somewhere in the file there is the string 'mdat'. To check if the file is corrupt or if my parser has a probelm I'd need 4 bytes before mdat (in expected 0001) and 32 bytes after the mdat string. [.... mdat .... .... .... .... .... .... .... .... ]

— Reply to this email directly or view it on GitHub https://github.com/sannies/isoviewer/issues/11#issuecomment-152096268.

The 1.2GB file is here: https://drive.google.com/file/d/0BxMshxVyQ5M_SXlTYzdrdEhwQmM/view?usp=sharing

4GB one still uploading

On Thu, Oct 29, 2015 at 9:23 PM, Andrew Ion aaf.ion@gmail.com wrote:

Sadly, the files aren't petabyte sized. They're a mere 1.2GB and 4.0GB - I'll upload to google drive and share the link once they're up in a few hours (I've only got poor man DSL)... Content isn't anything exotic, a pleasant drone flight showing the autumn colours. The files were created by a hardware encoder that breaks input every 4GB, so they're in sequence. Both of these files (and any other that the drone creates) behave the same way in isoviewer, but at least the 4GB doesn't have any issues that I know about. There's something wrong with the 1.2GB file, and I'm trying to figure out what so I can fix it as I'm starting to get quite a few from the drone that have the same problem. Was hoping isoviewer could help me pinpoint where/why things have gone wrong.

(Sadly, I'm not an expert at MP4, but I wrote a crappy little program that traces through all the frame pointers and know that the broken files typically have one that's shorter than what is specified in the first few bytes of the frame, but removing the short frame and adjusting the mdat pointers doesn't fix things... I also know that the picture data itself is still intact in the entire file - someone wrote an open-source program that pulls the frames out of the MP4 and puts them into a raw h264 file, which will is complete... But who wants that!)

Anyway, from the 4.0GB file, the 4 bytes before the mdat (at byte 36) are FA 53 F4 E2 (4199806178), which points to the 36 bytes before the moov at the end of the file. The whole 36 bytes around the mdat are:

FA 53 F4 E2 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

From the 1.2GB file, the 4 bytes before the mdat (also at byte 36) are 46 DB B5 10 (1188803856), also points to the 36 bytes before the moov at the end of the file...

The whole 36 bytes are:

46 DB B5 10 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

On Thu, Oct 29, 2015 at 2:40 AM, Sebastian Annies < notifications@github.com> wrote:

Hi, this exception is thrown when a 64bit value between 9223372036854775808 and 18446744073709551616 is read. This is out of range for a 64bit signed long and I haven't implemented that because I have never seen that for real

typically the exception is thrown the underlying parser reads 'garbage' from a file cause some offsets were wrong. Especially in your case it looks like a problem with a file because in line 84 of the AbstractBoxParser the size of a box is read. Or is your file larger than 8000 petabyte? If you can it would be great if you'd make the file available so that I can check the file but typically 4k cannot be shared with anyone I guess. It would also help if you get me the hex dump of a few bytes: somewhere in the file there is the string 'mdat'. To check if the file is corrupt or if my parser has a probelm I'd need 4 bytes before mdat (in expected 0001) and 32 bytes after the mdat string. [.... mdat .... .... .... .... .... .... .... .... ]

— Reply to this email directly or view it on GitHub https://github.com/sannies/isoviewer/issues/11#issuecomment-152096268.

And here's the 'good' 4GB one

https://drive.google.com/file/d/0BxMshxVyQ5M_SDRLejdaT0xjSEE/view?usp=sharing

On Thu, Oct 29, 2015 at 9:27 PM, Andrew Ion aaf.ion@gmail.com wrote:

The 1.2GB file is here: https://drive.google.com/file/d/0BxMshxVyQ5M_SXlTYzdrdEhwQmM/view?usp=sharing

4GB one still uploading

On Thu, Oct 29, 2015 at 9:23 PM, Andrew Ion aaf.ion@gmail.com wrote:

Sadly, the files aren't petabyte sized. They're a mere 1.2GB and 4.0GB - I'll upload to google drive and share the link once they're up in a few hours (I've only got poor man DSL)... Content isn't anything exotic, a pleasant drone flight showing the autumn colours. The files were created by a hardware encoder that breaks input every 4GB, so they're in sequence. Both of these files (and any other that the drone creates) behave the same way in isoviewer, but at least the 4GB doesn't have any issues that I know about. There's something wrong with the 1.2GB file, and I'm trying to figure out what so I can fix it as I'm starting to get quite a few from the drone that have the same problem. Was hoping isoviewer could help me pinpoint where/why things have gone wrong.

(Sadly, I'm not an expert at MP4, but I wrote a crappy little program that traces through all the frame pointers and know that the broken files typically have one that's shorter than what is specified in the first few bytes of the frame, but removing the short frame and adjusting the mdat pointers doesn't fix things... I also know that the picture data itself is still intact in the entire file - someone wrote an open-source program that pulls the frames out of the MP4 and puts them into a raw h264 file, which will is complete... But who wants that!)

Anyway, from the 4.0GB file, the 4 bytes before the mdat (at byte 36) are FA 53 F4 E2 (4199806178), which points to the 36 bytes before the moov at the end of the file. The whole 36 bytes around the mdat are:

FA 53 F4 E2 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

From the 1.2GB file, the 4 bytes before the mdat (also at byte 36) are 46 DB B5 10 (1188803856), also points to the 36 bytes before the moov at the end of the file...

The whole 36 bytes are:

46 DB B5 10 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

On Thu, Oct 29, 2015 at 2:40 AM, Sebastian Annies < notifications@github.com> wrote:

Hi, this exception is thrown when a 64bit value between 9223372036854775808 and 18446744073709551616 is read. This is out of range for a 64bit signed long and I haven't implemented that because I have never seen that for real

typically the exception is thrown the underlying parser reads 'garbage' from a file cause some offsets were wrong. Especially in your case it looks like a problem with a file because in line 84 of the AbstractBoxParser the size of a box is read. Or is your file larger than 8000 petabyte? If you can it would be great if you'd make the file available so that I can check the file but typically 4k cannot be shared with anyone I guess. It would also help if you get me the hex dump of a few bytes: somewhere in the file there is the string 'mdat'. To check if the file is corrupt or if my parser has a probelm I'd need 4 bytes before mdat (in expected 0001) and 32 bytes after the mdat string. [.... mdat .... .... .... .... .... .... .... .... ]

— Reply to this email directly or view it on GitHub https://github.com/sannies/isoviewer/issues/11#issuecomment-152096268.

my initial guess was wrong. Has nothing to do with the bytes I asked you to send me ;)

Ohh my ... this hardware encoder sucks. Let's start with things I've seen:

Why would anyone record 4k in NTSC framerate of 29.97fps. Use 30fps otherwise shit hits fan when processing the content later.
The bitstream inside the MP4 still contains emulation prevention - this should be removed before putting it into an MP4 (it's a kind of binary escaping: you write 0x00 00 03 00 01 instead of 0x00 00 00 01 as in the so-called Annex B format this 0001 separate two NAL units)
The chunks of the MP4 contains 1 sample only. Chunks have been invented to group samples. Technically 1 is also group but ... you get the point. This causes lots of overhead when reading the samples
The H264 stream is High@5.1 but doesn't make use of B frames (efficiency!!!!)
for continuous recording I would not use traditional MP4 files, I'd rather use fragmented MP4. But in this case - as there is no audio - I wouldn't bother using a container format - I would dumped the raw stream (Annex B format) on SD card. This has the nice side effect that it is still readable when the recording is suddenly interrupted. If recording is forcefully interrupted (and I've seen drones falling from the sky ;) ) the moov box - which contains the table of contents - might not be written.
The AU delimiter should not be muxed into the MP4 as they only make sense when writing raw h264 data

The problem in the smaller file is that the table of contents is no longer in sync with the data. Starting from frame 82 the chunk offsets (/moov[0]/trak[0]/mdia[0]/minf[0]/stbl[0]/stco[0]) no longer point to the beginning of a sample: Open the file in a hex editor and go to 20037226 which 81st offset in the chunk offset box. You can see the distinctive pattern 0x00 00 00 02 09 (which is the obsolete AU delimiter and your jackpot!), if you go to 21473661 which is the 82nd offset in the chunk offset box you don't see that distinctive pattern - meaning: you are lost.

Long story short: In your special case you can reconstruct the data with the following easy algorithm (pseudo code not very efficient most likely):

List offsets = new List() long pos; // search for all offsets of the special pattern while (not at the end of the file) { if (last6Bytes.equal(0x00 00 00 02 09)) { offfsets.add(pos); } pos++; }

OutputStream o = new FileOutputStream("out.h264"); for (offset in offsets) { goto offset in file while (current position in file < next offset in offsets list) { int nalSizes = readUInt32 from file (== 4bytes) o.write(new byte[]{0,0,0,1}) byte[] nal = read nalSizes bytes from file; o.write(nal); } } and the resulting file would be a so called Annex B file. You can now mux the file with MP4Box or ffmpeg (and video codec copy so that you don't re-encode the video)

Another option is to write exactly these value from the offsets List in the chunk offset box (you will notice that the first 81 are the same) and repair the file by fixing this chunk offset box. Perhaps you'll also need to fix the sample size box - in this case the value need to be the differences between two subsequent chunk offsets.

Hope that helps. Sebastian

2015-10-30 12:03 GMT+01:00 nigel-q notifications@github.com:

And here's the 'good' 4GB one

https://drive.google.com/file/d/0BxMshxVyQ5M_SDRLejdaT0xjSEE/view?usp=sharing

On Thu, Oct 29, 2015 at 9:27 PM, Andrew Ion aaf.ion@gmail.com wrote:

The 1.2GB file is here:

https://drive.google.com/file/d/0BxMshxVyQ5M_SXlTYzdrdEhwQmM/view?usp=sharing

4GB one still uploading

On Thu, Oct 29, 2015 at 9:23 PM, Andrew Ion aaf.ion@gmail.com wrote:

Sadly, the files aren't petabyte sized. They're a mere 1.2GB and 4.0GB - I'll upload to google drive and share the link once they're up in a few hours (I've only got poor man DSL)... Content isn't anything exotic, a pleasant drone flight showing the autumn colours. The files were created by a hardware encoder that breaks input every 4GB, so they're in sequence. Both of these files (and any other that the drone creates) behave the same way in isoviewer, but at least the 4GB doesn't have any issues that I know about. There's something wrong with the 1.2GB file, and I'm trying to figure out what so I can fix it as I'm starting to get quite a few from the drone that have the same problem. Was hoping isoviewer could help me pinpoint where/why things have gone wrong.

(Sadly, I'm not an expert at MP4, but I wrote a crappy little program that traces through all the frame pointers and know that the broken files typically have one that's shorter than what is specified in the first few bytes of the frame, but removing the short frame and adjusting the mdat pointers doesn't fix things... I also know that the picture data itself is still intact in the entire file - someone wrote an open-source program that pulls the frames out of the MP4 and puts them into a raw h264 file, which will is complete... But who wants that!)

Anyway, from the 4.0GB file, the 4 bytes before the mdat (at byte 36) are FA 53 F4 E2 (4199806178), which points to the 36 bytes before the moov at the end of the file. The whole 36 bytes around the mdat are:

FA 53 F4 E2 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

From the 1.2GB file, the 4 bytes before the mdat (also at byte 36) are 46 DB B5 10 (1188803856), also points to the 36 bytes before the moov at the end of the file...

The whole 36 bytes are:

46 DB B5 10 6D 64 61 74 00 00 00 02 09 10 00 00 00 20 06 00 0D 80 99 CF 00 15 F9 80 99 CF 00 15 F9 C0 01 09

On Thu, Oct 29, 2015 at 2:40 AM, Sebastian Annies < notifications@github.com> wrote:

Hi, this exception is thrown when a 64bit value between 9223372036854775808 and 18446744073709551616 is read. This is out of range for a 64bit signed long and I haven't implemented that because I have never seen that for real

typically the exception is thrown the underlying parser reads 'garbage' from a file cause some offsets were wrong. Especially in your case it looks like a problem with a file because in line 84 of the AbstractBoxParser the size of a box is read. Or is your file larger than 8000 petabyte? If you can it would be great if you'd make the file available so that I can check the file but typically 4k cannot be shared with anyone I guess. It would also help if you get me the hex dump of a few bytes: somewhere in the file there is the string 'mdat'. To check if the file is corrupt or if my parser has a probelm I'd need 4 bytes before mdat (in expected 0001) and 32 bytes after the mdat string. [.... mdat .... .... .... .... .... .... .... .... ]

— Reply to this email directly or view it on GitHub <https://github.com/sannies/isoviewer/issues/11#issuecomment-152096268 .

— Reply to this email directly or view it on GitHub https://github.com/sannies/isoviewer/issues/11#issuecomment-152496579.

sannies / isoviewer

RuntimeException - UInt64 #11