search

MediaArea / RAWcooked

Encodes RAW audio-visual data into the Matroska container (MKV), using the video codec FFV1 for the image and audio codec FLAC for the sound.
https://mediaarea.net/RAWcooked
BSD 2-Clause "Simplified" License
43 stars 11 forks source link

Attachment shows up as FFV1 track #244

Open kieranjol opened 5 years ago

kieranjol commented 5 years ago

Hi - I am having an issue with one DPX sequence only. In all my testing, rawcooked makes a fully reversible sequence - it just appears that there are issues with the Matroska file itself when read in ffmpeg. There appears to be two issues:

  1. The attachment shows up as an FFV1 track
  2. Read errors appear for the file via ffmpeg

I tested this with Rawcooked 18.10 on Windows 7 and OSX, and I just finished testing on OSX El Capitan with rawcooked git master. The same issue occurs. I tried to replicate the issue with just 25000 or 50000 files and the issue did not appear. I was hoping to figure out at what point the issue occurs.

This is what appears with 4.1.1 which was the version of ffmpeg that rawcooked used

$ ffmpeg -i /Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv 
ffmpeg version 4.1.1 Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.1.1 --enable-shared --enable-pthreads --enable-version3 --enable-hardcoded-tables --enable-avresample --cc=clang --host-cflags=-I/System/Library/Frameworks/JavaVM.framework/Versions/Current/Headers/ --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libmp3lame --enable-libopus --enable-librubberband --enable-libsnappy --enable-libtesseract --enable-libtheora --enable-libvorbis --enable-libvpx --enable-libx264 --enable-libx265 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-librtmp --enable-libspeex --enable-videotoolbox --disable-libjack --disable-indev=jack --enable-libaom --enable-libsoxr
  libavutil      56. 22.100 / 56. 22.100
  libavcodec     58. 35.100 / 58. 35.100
  libavformat    58. 20.100 / 58. 20.100
  libavdevice    58.  5.100 / 58.  5.100
  libavfilter     7. 40.101 /  7. 40.101
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  3.100 /  5.  3.100
  libswresample   3.  3.100 /  3.  3.100
  libpostproc    55.  3.100 / 55.  3.100
[matroska,webm @ 0x7f9e5c000000] Invalid length 0x36b16ede > 0x10000000 for syntax element 5
[matroska,webm @ 0x7f9e5c000000] Duplicate element
[matroska,webm @ 0x7f9e5c000000] Read error at pos. 187 (0xbb)
[matroska,webm @ 0x7f9e5c000000] Duplicate element
    Last message repeated 4 times
[matroska,webm @ 0x7f9e5c000000] Invalid length 0x36b16ede > 0x10000000 for syntax element 5
[matroska,webm @ 0x7f9e5c000000] Duplicate element
[matroska,webm @ 0x7f9e5c000000] incomplete attachment
    Last message repeated 1 times
[matroska,webm @ 0x7f9e5c000000] Could not find codec parameters for stream 1 (Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '/Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv':
  Metadata:
    ENCODER         : Lavf58.20.100
  Duration: 01:28:52.71, start: 0.000000, bitrate: 350886 kb/s
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc58.35.100 ffv1
      DURATION        : 01:28:52.709000000
    Stream #0:1: Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc58.35.100 ffv1
      DURATION        : 01:28:52.709000000
At least one output file must be specified

and here is what appears with ffmpeg git master - it seems like the same errors combined with the general Matroska errors that I reported here: http://ffmpeg.org/pipermail/ffmpeg-user/2019-July/044725.html 

$ ./ffmpeg -i /Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv 
ffmpeg version N-94421-gb3b7523feb Copyright (c) 2000-2019 the FFmpeg developers
  built with Apple LLVM version 8.0.0 (clang-800.0.42.1)
  configuration: 
  libavutil      56. 32.100 / 56. 32.100
  libavcodec     58. 55.100 / 58. 55.100
  libavformat    58. 30.100 / 58. 30.100
  libavdevice    58.  9.100 / 58.  9.100
  libavfilter     7. 58.100 /  7. 58.100
  libswscale      5.  6.100 /  5.  6.100
  libswresample   3.  6.100 /  3.  6.100
[matroska,webm @ 0x7ffd13808600] Invalid length 0x36b16ede > 0x10000000 for element with ID 0x465C at 0x31c
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] 0x00 at pos 186 (0xba) invalid as first byte of an EBML number
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] Element at 0x61 ending at 0x3c4dbb8c53abee exceeds containing master element ending at 0x140d
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] Element at 0x70 ending at 0x8b4dbb8e5423 exceeds containing master element ending at 0x141c
[matroska,webm @ 0x7ffd13808600] Element at 0x7f ending at 0x16b1f4 exceeds containing master element ending at 0x142b
[matroska,webm @ 0x7ffd13808600] Element at 0x90 ending at 0x1675c6 exceeds containing master element ending at 0x143c
[matroska,webm @ 0x7ffd13808600] Invalid length 0x36b16ede > 0x10000000 for element with ID 0x465C at 0x31c
[matroska,webm @ 0x7ffd13808600] Duplicate element
[matroska,webm @ 0x7ffd13808600] incomplete attachment
    Last message repeated 1 times
[matroska,webm @ 0x7ffd13808600] Could not find codec parameters for stream 1 (Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '/Volumes/kieran_tests/oe13172/f69fb94e-412b-45d8-bb21-1988399e2ccc/objects/f69fb94e-412b-45d8-bb21-1988399e2ccc.mkv.RAWcooked/24e6bb3c-fd29-47bc-bf3e-b4d64ed0c23f.mkv':
  Metadata:
    ENCODER         : Lavf58.20.100
  Duration: 01:28:52.71, start: 0.000000, bitrate: 350886 kb/s
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc58.35.100 ffv1
      DURATION        : 01:28:52.709000000
    Stream #0:1: Video: ffv1 (FFV1 / 0x31564646), none, 2048x1536, SAR 1:1 DAR 4:3, 24 fps, 24 tbr, 1k tbn, 1k tbc (default)
    Metadata:
      ENCODER         : Lavc58.35.100 ffv1
      DURATION        : 01:28:52.709000000
At least one output file must be specified

I am not sure if these are relevant, but I'll attach the ffmpeg log of an earlier run, as well as the rawcooked_reversibility_data which I copy/pasted during the rawcooking. I do not know how to extract the attachment in the MKV file seeing as ffmpeg sees it as a video track.

Logs: https://we.tl/t-MfiegwDn4r metadata - mediainfo/mediatrace of MKV file as well as the source sequence (in supplemental) https://we.tl/t-rNwSDKx9EP

kieranjol commented 5 years ago

Also the rawcooked command originally would have been with 18.10 rawcooked input --check full -o ffv1_path

when i rechecked with git-master of rawcooked, i just did rawcooked path/to/dpx

and now i'm trying with git master: rawcooked --all path/to/dpx

kieranjol commented 5 years ago

I got the same ffmpeg error with the file created via rawcooked git-master. The final check still seems to be running though.

kieranjol commented 5 years ago

The final post transcode check just said that everything was fine btw.

JeromeMartinez commented 5 years ago

Looks like a FFmpeg has some difficulties to decode the file created by itself. I heard that there is some heuristical checks for detecting corrupted streams, maybe the issue is there. RAWcooked has its own Matroska parser, so if reversibility is fine with RAWcooked I guess that this is on FFmpeg side which can not demux the file.

I tested your FFmpeg command line having errors with their git-master, no issue, maybe it would be fine with this version of FFmpeg?

kieranjol commented 5 years ago

Which version of ffmpeg do you mean,current git-master? I compiled ffmpeg after that fix was out in. I've only ever seen this happen with one single MKV file created by rawcooked. Should I go to ffmpeg for further help,or is it worth trying to figure out what it is about this one rawcooked job that is causing issues?

JeromeMartinez commented 5 years ago

Currently RAWcooked has no encoder & muxer, it uses FFmpeg, you can see that by running rawcooked -d YourDpxDirectoryName, then launching the FFmpeg command displayed. If you try to dig with FFmpeg people, it is better to show them the FFmpeg command line (+maybe provide the attachment file).

Anyway, the main issue for the moment is to be able to reproduce the issue with a small input so I or Ffmpeg people can reproduce the issue. I didn't download the files (I was too late), do you mean by "as well as the source sequence" that the package you provided we permitting me to reproduce the error? If not, is it possible to have the 1st MB (should be enough for getting the 1st FFmpeg error) ?

kieranjol commented 5 years ago

Ahhh,use dd or something to just slice of the first megabyte of the MKV file? I can do that!

On Thu, 15 Aug 2019, 17:35 Jérôme Martinez, notifications@github.com wrote:

Currently RAWcooked has no encoder & muxer, it uses FFmpeg, you can see that by running rawcooked -d YourDpxDirectoryName, then launching the FFmpeg command displayed. If you try to dig with FFmpeg people, it is better to show them the FFmpeg command line (+maybe provide the attachment file).

Anyway, the main issue for the moment is to be able to reproduce the issue with a small input so I or Ffmpeg people can reproduce the issue. I didn't download the files (I was too late), do you mean by "as well as the source sequence" that the package you provided we permitting me to reproduce the error? If not, is it possible to have the 1st MB (should be enough for getting the 1st FFmpeg error) ?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/MediaArea/RAWcooked/issues/244?email_source=notifications&email_token=AAITFPWAPX63WYQIBDHVBWDQEWAWLA5CNFSM4IHZYGF2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4MKHAI#issuecomment-521708417, or mute the thread https://github.com/notifications/unsubscribe-auth/AAITFPV34HZR3OA7DMJDNF3QEWAWLANCNFSM4IHZYGFQ .

JeromeMartinez commented 5 years ago

Ahhh,use dd or something to just slice of the first megabyte of the MKV file?

Right, just need that. I am not a Linux/Mac user, so not sure, but looks like that head head -c 1M input.file > output.file could also do the job.

kieranjol commented 5 years ago

Hi Jerome, i emailed the first meg - hope it helps with the test..

JeromeMartinez commented 5 years ago

IIUC the issue, the attachment is huge (1 GB) and FFmpeg does not appreciate that (looks like it considers the stream as buggy in the case the attachment size is > 256 MB, I understand a bit that as a 1 GB attachment is not classic).

I'll try to fix FFmpeg for that.

Note that the attachment is big because I need to store 8 KB of "random" data for each DPX frame, for reversibility, looks like the DPX files have 8 KB of extra data after the real content, I'll also check that this is coherent.

JeromeMartinez commented 5 years ago

Additional remark: IMO this MKV file is conform to spec, I don't remind that there is a limitation for theFileData Matroska element. @robUx4 would you mind to confirm that there is no limitation in element size even for attachments, or am I wrong there?

JeromeMartinez commented 5 years ago

@kieranjol beside the issue, I see that your DPX has a weird size. File size is 0xC04000 (and DPX dedicated field says same) Header size is 0x2000 Content size is 0xC0000 (0x800 x 0x600 x 4 bytes per pixel) So remaining 0x2000 bytes at the end are present, and they look like random (so can not be compressed, so a big RAWcooked side car data file!). Is it wanted?

JeromeMartinez commented 5 years ago

The issue can be reproduced without RAWcooked, just by using FFmpeg, so it is not a RAWcooked issue (except blaming it for creating a big attachment), I have open a dedicated ticket https://trac.ffmpeg.org/ticket/8083 and I'll try to provide a patch to FFmpeg.

kieranjol commented 5 years ago

@kieranjol beside the issue, I see that your DPX has a weird size. File size is 0xC04000 (and DPX dedicated field says same) Header size is 0x2000 Content size is 0xC0000 (0x800 x 0x600 x 4 bytes per pixel) So remaining 0x2000 bytes at the end are present, and they look like random (so can not be compressed, so a big RAWcooked side car data file!). Is it wanted?

Woah - it's not wanted, this sequence was delivered by a vendor so I don't have much further info.

The issue can be reproduced without RAWcooked, just by using FFmpeg, so it is not a RAWcooked issue (except blaming it for creating a big attachment), I have open a dedicated ticket https://trac.ffmpeg.org/ticket/8083 and I'll try to provide a patch to FFmpeg.

Thanks for opening the ticket - the conversation is very interesting. I concur that it's not a RAWcooked issue. I think from my side, I'll either ZIP this sequence and bypass rawcooked, or else I'll hold off for the ffmpeg issue to be fixed an then process it with rawcooked.

Thanks so much for the detailed analysis.

JeromeMartinez commented 5 years ago

I'll either ZIP this sequence and bypass rawcooked,

IMO RAWcooked is still interesting here for other advantages (VLC playback, compression performance), just that you can not handle the file with FFmpeg, consider the resulting file as a compressed file better than a ZIP.

In other words, in both cases (RAWcooked and ZIP), you can not use the file directly with current FFmpeg, but with RAWcooked you keep the other advantages as well as expecting to have FFmpeg support in a near future.

robUx4 commented 5 years ago

Additional remark: IMO this MKV file is conform to spec, I don't remind that there is a limitation for theFileData Matroska element. @robUx4 would you mind to confirm that there is no limitation in element size even for attachments, or am I wrong there?

No limitation on the size apart from the fact it needs to fit in 56 bits unsigned. 1GB is fine.