search

MediaArea / RAWcooked

Encodes RAW audio-visual data into the Matroska container (MKV), using the video codec FFV1 for the image and audio codec FLAC for the sound.
https://mediaarea.net/RAWcooked
BSD 2-Clause "Simplified" License
42 stars 11 forks source link

Error: undecodable files from output are not same as files from source. #355

Open vampyresqvid opened 3 years ago

vampyresqvid commented 3 years ago

Hello,

I have a project with 16 sets of DPX files, and four of the sets have given me this error. The others processed as expected. I'm running rawcooked with the --all flag, default license. One example of the error output below:

"Decoding was checked, issues detected, see below.

Error: undecodable files from output are not same as files from source. GRADED DPX/Painting in Object4396.dpx GRADED DPX/Painting in Object4399.dpx GRADED DPX/Painting in Object4400.dpx GRADED DPX/Painting in Object4401.dpx GRADED DPX/Painting in Object4402.dpx GRADED DPX/Painting in Object4403.dpx GRADED DPX/Painting in Object4404.dpx GRADED DPX/Painting in Object4405.dpx GRADED DPX/Painting in Object4406.dpx GRADED DPX/Painting in Object4407.dpx ... "

I'm attaching mediainfo output for one of the files the error message flagged, and for one of the files it did not flag. There are a number of differences between them, but I don't know what might be causing the problem. I'm curious if you can explain why this error is occurring and if there's anything I can do about it.

Let me know if any other information would be helpful. Thank you.

PaintingInObject0280.txt PaintingInObject4400.txt

JeromeMartinez commented 3 years ago

Thanks for the traces. Unfortunately I don't see an obvious difference which would have an impact on the reversibility. Is it possible to have one of the files? here or info@mediaarea.net. I don't need the data part, having e.g. the 16 first KB should be enough if there is an issue with the content itself.

JeromeMartinez commented 3 years ago

I can not reproduce the issue with the file you send to me. Tried with RAWcooked 21.09 and FFmpeg 4.4. Can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...)

jerome@MacMini1100 355 % rawcooked --all GRADED\ DPX                              
Using default license.
Consider to support RAWcooked: https://MediaArea.net/RAWcooked

Track 1:                                                    
  Painting in Object%04d.dpx
 (4400 --> 4400)
  DPX/Raw/RGB/10bit/FilledA/BE

ffmpeg version 4.4 Copyright (c) 2000-2021 the FFmpeg developers
  built with Apple clang version 12.0.5 (clang-1205.0.22.9)
  configuration: --prefix=/usr/local/Cellar/ffmpeg/4.4_2 --enable-shared --enable-pthreads --enable-version3 --cc=clang --host-cflags= --host-ldflags= --enable-ffplay --enable-gnutls --enable-gpl --enable-libaom --enable-libbluray --enable-libdav1d --enable-libmp3lame --enable-libopus --enable-librav1e --enable-librubberband --enable-libsnappy --enable-libsrt --enable-libtesseract --enable-libtheora --enable-libvidstab --enable-libvorbis --enable-libvpx --enable-libwebp --enable-libx264 --enable-libx265 --enable-libxml2 --enable-libxvid --enable-lzma --enable-libfontconfig --enable-libfreetype --enable-frei0r --enable-libass --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libspeex --enable-libsoxr --enable-libzmq --enable-libzimg --disable-libjack --disable-indev=jack --enable-avresample --enable-videotoolbox
  libavutil      56. 70.100 / 56. 70.100
  libavcodec     58.134.100 / 58.134.100
  libavformat    58. 76.100 / 58. 76.100
  libavdevice    58. 13.100 / 58. 13.100
  libavfilter     7.110.100 /  7.110.100
  libavresample   4.  0.  0 /  4.  0.  0
  libswscale      5.  9.100 /  5.  9.100
  libswresample   3.  9.100 /  3.  9.100
  libpostproc    55.  9.100 / 55.  9.100
Input #0, image2, from 'GRADED DPX/Painting in Object%04d.dpx':
  Duration: 00:00:00.04, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: dpx, gbrp10le(gbr/unknown/unknown), 4096x3112, 24 fps, 24 tbr, 24 tbn, 24 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (dpx (native) -> ffv1 (native))
  File GRADED DPX.rawcooked_reversibility_data -> Stream #0:1
Press [q] to stop, [?] for help
Output #0, matroska, to 'GRADED DPX.mkv':
  Metadata:
    encoder         : Lavf58.76.100
  Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le(gbr/unknown/unknown, progressive), 4096x3112, q=2-31, 200 kb/s, 24 fps, 1k tbn
    Metadata:
      encoder         : Lavc58.134.100 ffv1
  Stream #0:1: Attachment: none
    Metadata:
      mimetype        : application/octet-stream
      filename        : RAWcooked reversibility data
frame=    1 fps=0.4 q=-0.0 Lsize=    1113kB time=00:00:00.00 bitrate=9118104.0kbits/s speed=0.000443x    
video:1112kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.119202%
2.7 MiB/s                              

Info: Reversibility data created by RAWcooked 21.09.
Info: Uncompressed file hashes (used by reversibility check) present.

Reversability was checked, no issue detected.
JeromeMartinez commented 3 years ago

@vampyresqvid can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...)

vampyresqvid commented 3 years ago

Hello,

Sorry for the delay, I was out for a long weekend.

I was able to get the same error with this limited number of 14 DPX files from the same directory: https://www.dropbox.com/sh/o9xqdi7oy623bqy/AACTl0jXr-WbLn15BlQk7Wy8a?dl=0

-Dan

On Mon, Sep 27, 2021 at 4:17 PM Jérôme Martinez @.***> wrote:

@vampyresqvid https://github.com/vampyresqvid can you reproduce the issue with a single file on your machine? If not, can you provide he full directory (the smallest one with an issue...)

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/MediaArea/RAWcooked/issues/355#issuecomment-928237642, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEVL52X36I27FWK2FDPZKM3UEDGPPANCNFSM5EMUSDYA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

JeromeMartinez commented 3 years ago

Thank you. Actually you need the first file listed as having a problem and the previous one, e.g. 4395 and 4396, it is enough for creating the issue.

Good news: issue is in the decoder. So created MKV files will be correctly decoded after a patch.

The issue comes from the difference between the encoder (currently a third party tool, FFmpeg) and the decoder (home made), and the algorithm between both tools is not same in this use case. Use case is different "flavors" of DPX between files in the same sequence, e.g. some files are in little endian (written by Premiere Pro) and some others are in big endian (written by Photoshop). I guess that some files where edited, and the decoder uses the first file as the reference then write correctly the headers but use the "flavor" of the first file as the reference (which is wrong).

The encoder (FFmpeg) behaves correctly, and we need to adapt the decoder (RAWcooked) for such case.