search

anthwlock / untrunc

Restore a truncated mp4/mov. Improved version of ponchio/untrunc
GNU General Public License v2.0
2.08k stars 199 forks source link

Fail to build on Mac OSX – fatal error: too many errors emitted #107

Closed o3jvind closed 2 years ago

o3jvind commented 2 years ago

Sorry if this is trivial.

I get a lot of errors like this:

In file included from src/atom.cpp:1:
In file included from src/atom.h:10:
In file included from src/common.h:5:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/iostream:39:
In file included from /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/istream:162:
./ffmpeg-3.3.9/version:1:1: error: expected unqualified-id
3.3.9

And the build stops running with:

fatal error: too many errors emitted, stopping now [-ferror-limit=]
20 errors generated.
make: *** [.build_3.3.9/src/atom.o] Error 1

Mac OS X 12.0.1, Xcode 13.1, Intel. Any any ideas of what I'm doing wrong?

anthwlock commented 2 years ago

I wonder what created the file "./ffmpeg-3.3.9/version", since I don't seem to have it. What happens if you delete/rename it?

If that does not help, maybe the full log (and compiler version) could be helpful. Also what commands did you use? Just make FF_VER=3.3.9

o3jvind commented 2 years ago

I download the source and run make FF_VER=3.3.9.

"./ffmpeg-3.3.9/version" seems to be part of the download "https://www.ffmpeg.org/releases/ffmpeg-3.3.9.tar.xz" which is downloaded when I run make FF_VER=3.3.9.

I'm new to this and I don't know how to delete "./ffmpeg-3.3.9/version" and then compile again?

For your information. I'm able to run makeand get a working copy of untrunc and I'm able to compile FFmpeg when I follow this "https://trac.ffmpeg.org/wiki/CompilationGuide". So I guess I have what I need to compile. I just don't know how to do it.

anthwlock commented 2 years ago

I'm new to this and I don't know how to delete "./ffmpeg-3.3.9/version" and then compile again?

Try the following, it should create a build.log file, please upload it in case of errors:

mv ffmpeg-3.3.9/version ffmpeg-3.3.9/version.bak
make FF_VER=3.3.9 >build.log 2>&1

What does readlink -f $(which c++) output? If the compiler comes from Xcode, maybe it helps to install clang in another way.

Can you upload the file /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/c++/v1/istream? I wonder why it seems to include "./ffmpeg-3.3.9/version" ..

o3jvind commented 2 years ago

OK

readlink -f $(which c++)returns:

readlink: illegal option -- f usage: readlink [-n] [file ...]

build.log istream.txt

Please note that I added the ".txt" extension to "istream". Otherwise I couldn't upload it

anthwlock commented 2 years ago

Just out of curiosity, what does the following output? command -v c++ python -c 'import os,sys;print(os.path.realpath(sys.argv[1]))' $(command -v c++) command -v greadlink

Now to your problem. It looks like your filesystem is case insensitive, which is why ffmpeg-3.3.9/VERSION is seen as ffmpeg-3.3.9/version. This combined with the Makefile using -I./ffmpeg-3.3.9 and a #include <version> resulted in the first issue. The second issue (visible in your build.log) is apparently caused by missing -framework arguments.

Please checkout the branch 107_macos, and (again) send me the build.log in case of errors.

o3jvind commented 2 years ago

Error from 107_macos :

ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [untrunc] Error 1

build.log

And for your curiosity :-)

command -v c++
/usr/bin/c++
python -c 'import os,sys;print(os.path.realpath(sys.argv[1]))' $(command -v c++)
/usr/bin/c++
command -v greadlink
/usr/local/bin/greadlink
anthwlock commented 2 years ago

python -c 'import os,sys;print(os.path.realpath(sys.argv[1]))' $(command -v c++) /usr/bin/c++

Interesting! What about the following? ls -l /usr/bin/c++ /usr/bin/c++ --version

build.log

It looks like some -framework arguments were missing. I added some with a force push on 107_macos. If you still get the Undefined symbols for architecture x86_64 error, please try to add some of the frameworks listed here to the Makefile here, and let me know what frameworks were needed.

o3jvind commented 2 years ago 
ls -l /usr/bin/c++
-rwxr-xr-x  1 root  wheel  137696 18 Okt 05:30 /usr/bin/c++
/usr/bin/c++ --version
Apple clang version 13.0.0 (clang-1300.0.29.3)
Target: x86_64-apple-darwin21.1.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

Still no joy

build.log

The only two frameworks that are doing any difference are:

AudioToolbox and CoreVideo

It doesn't matter if any of the others are added.

anthwlock commented 2 years ago

Maybe try to

  1. add -liconv to the LDFLAGS here
  2. add --enable-vda here

Because 2. changes the way ffmpeg gets build, you should remove the directory ffmpeg-3.3.9, to trigger a rebuild. If 2. does not remove the _ff_vda_create_decoder errors, maybe also try to remove all options after ./configure (to test if the defaults work better)

Btw, what version of macOS are you using?

o3jvind commented 2 years ago

With 1 and 2 I get:

Undefined symbols for architecture x86_64:
  "_ff_vda_create_decoder", referenced from:
      _vdadec_init in libavcodec.a(vda_h264_dec.o)
  "_ff_vda_destroy_decoder", referenced from:
      _vdadec_init in libavcodec.a(vda_h264_dec.o)
      _vdadec_close in libavcodec.a(vda_h264_dec.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [untrunc] Error 1

And if I remove all the options after ./configure I get tons of errors.

Mac OS X 12.0.1, Xcode 13.1, Intel.

anthwlock commented 2 years ago

Please try 107_macos again. I added --disable-vda --disable-audiotoolbox --disable-videotoolbox, it should disable the code that uses these symbols.

And if I remove all the options after ./configure I get tons of errors.

More Undefined symbols errors or also other types of errors?

o3jvind commented 2 years ago

SUCCESS – no errors!!

Will untrunc be limited by disabling these and will we be gaining any benefits by using a more recent version of FFmpeg?

Without any options after ./configure I got:

Undefined symbols for architecture x86_64:
  "_BZ2_bzDecompress", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
  "_BZ2_bzDecompressEnd", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
  "_BZ2_bzDecompressInit", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
  "_CMBlockBufferCopyDataBytes", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMBlockBufferCreateWithMemoryBlock", referenced from:
      _videotoolbox_common_end_frame in libavcodec.a(videotoolbox.o)
  "_CMSampleBufferCreate", referenced from:
      _videotoolbox_common_end_frame in libavcodec.a(videotoolbox.o)
  "_CMSampleBufferGetDataBuffer", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMSampleBufferGetDecodeTimeStamp", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMSampleBufferGetFormatDescription", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
      _vtenc_output_callback in libavcodec.a(videotoolboxenc.o)
  "_CMSampleBufferGetPresentationTimeStamp", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMSampleBufferGetSampleAttachmentsArray", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMSampleBufferGetTotalSampleSize", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_CMTimeMake", referenced from:
      _vtenc_send_frame in libavcodec.a(videotoolboxenc.o)
  "_CMVideoFormatDescriptionCreate", referenced from:
      _av_videotoolbox_default_init2 in libavcodec.a(videotoolbox.o)
  "_CMVideoFormatDescriptionGetH264ParameterSetAtIndex", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
      _get_params_size in libavcodec.a(videotoolboxenc.o)
      _copy_param_sets in libavcodec.a(videotoolboxenc.o)
  "_SSLClose", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
      _tls_close in libavformat.a(tls_securetransport.o)
  "_SSLCopyPeerTrust", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLCreateContext", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLHandshake", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLRead", referenced from:
      _tls_read in libavformat.a(tls_securetransport.o)
  "_SSLSetCertificate", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLSetConnection", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLSetIOFuncs", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLSetPeerDomainName", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLSetSessionOption", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SSLWrite", referenced from:
      _tls_write in libavformat.a(tls_securetransport.o)
  "_SecIdentityCreate", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SecItemImport", referenced from:
      _import_pem in libavformat.a(tls_securetransport.o)
  "_SecTrustEvaluate", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_SecTrustSetAnchorCertificates", referenced from:
      _tls_open in libavformat.a(tls_securetransport.o)
  "_VTCompressionSessionCompleteFrames", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
      _vtenc_close in libavcodec.a(videotoolboxenc.o)
  "_VTCompressionSessionCreate", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_VTCompressionSessionEncodeFrame", referenced from:
      _vtenc_send_frame in libavcodec.a(videotoolboxenc.o)
  "_VTCompressionSessionPrepareToEncodeFrames", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_VTDecompressionSessionCreate", referenced from:
      _av_videotoolbox_default_init2 in libavcodec.a(videotoolbox.o)
  "_VTDecompressionSessionDecodeFrame", referenced from:
      _videotoolbox_common_end_frame in libavcodec.a(videotoolbox.o)
  "_VTDecompressionSessionInvalidate", referenced from:
      _av_videotoolbox_default_free in libavcodec.a(videotoolbox.o)
  "_VTDecompressionSessionWaitForAsynchronousFrames", referenced from:
      _videotoolbox_common_end_frame in libavcodec.a(videotoolbox.o)
  "_VTSessionCopyProperty", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_VTSessionSetProperty", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_compress", referenced from:
      _encode_strip in libavcodec.a(tiffenc.o)
  "_compress2", referenced from:
      _flashsv2_encode_frame in libavcodec.a(flashsv2enc.o)
      _flashsv_encode_frame in libavcodec.a(flashsvenc.o)
  "_deflate", referenced from:
      _encode_frame in libavcodec.a(pngenc.o)
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _flashsv2_encode_frame in libavcodec.a(flashsv2enc.o)
      _encode_frame in libavcodec.a(lclenc.o)
      _encode_frame in libavcodec.a(zmbvenc.o)
  "_deflateBound", referenced from:
      _encode_png in libavcodec.a(pngenc.o)
      _encode_apng in libavcodec.a(pngenc.o)
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _encode_frame in libavcodec.a(lclenc.o)
  "_deflateEnd", referenced from:
      _png_enc_close in libavcodec.a(pngenc.o)
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _flashsv2_encode_frame in libavcodec.a(flashsv2enc.o)
      _flashsv_encode_end in libavcodec.a(flashsvenc.o)
      _encode_end in libavcodec.a(lclenc.o)
      _encode_end in libavcodec.a(zmbvenc.o)
  "_deflateInit2_", referenced from:
      _png_enc_init in libavcodec.a(pngenc.o)
  "_deflateInit_", referenced from:
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _flashsv2_encode_frame in libavcodec.a(flashsv2enc.o)
      _encode_init in libavcodec.a(lclenc.o)
      _encode_init in libavcodec.a(zmbvenc.o)
  "_deflateReset", referenced from:
      _encode_frame in libavcodec.a(pngenc.o)
      _encode_frame in libavcodec.a(lclenc.o)
      _encode_frame in libavcodec.a(zmbvenc.o)
  "_inflate", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
      _zlib_refill in libavformat.a(swfdec.o)
      _decode_frame_common in libavcodec.a(pngdec.o)
      _decode_text_chunk in libavcodec.a(pngdec.o)
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _http_read_stream in libavformat.a(http.o)
      _decode_frame in libavcodec.a(lcldec.o)
      ...
  "_inflateEnd", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
      _swf_read_close in libavformat.a(swfdec.o)
      _decode_frame_apng in libavcodec.a(pngdec.o)
      _decode_frame_png in libavcodec.a(pngdec.o)
      _decode_text_chunk in libavcodec.a(pngdec.o)
      _flashsv_decode_init in libavcodec.a(flashsv.o)
      _flashsv_decode_end in libavcodec.a(flashsv.o)
      ...
  "_inflateInit2_", referenced from:
      _http_read_header in libavformat.a(http.o)
  "_inflateInit_", referenced from:
      _matroska_decode_buffer in libavformat.a(matroskadec.o)
      _swf_read_header in libavformat.a(swfdec.o)
      _decode_frame_apng in libavcodec.a(pngdec.o)
      _decode_frame_png in libavcodec.a(pngdec.o)
      _decode_text_chunk in libavcodec.a(pngdec.o)
      _flashsv_decode_init in libavcodec.a(flashsv.o)
      _flashsv2_decode_init in libavcodec.a(flashsv.o)
      ...
  "_inflateReset", referenced from:
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
      _decode_frame in libavcodec.a(lcldec.o)
      _decode_frame in libavcodec.a(tscc.o)
      _zerocodec_decode_frame in libavcodec.a(zerocodec.o)
      _decode_frame in libavcodec.a(zmbv.o)
  "_inflateSync", referenced from:
      _flashsv_decode_frame in libavcodec.a(flashsv.o)
  "_kCMFormatDescriptionExtension_SampleDescriptionExtensionAtoms", referenced from:
      _av_videotoolbox_default_init2 in libavcodec.a(videotoolbox.o)
  "_kCMSampleAttachmentKey_NotSync", referenced from:
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
  "_kCMTimeIndefinite", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
      _vtenc_frame in libavcodec.a(videotoolboxenc.o)
      _vtenc_close in libavcodec.a(videotoolboxenc.o)
  "_kCMTimeInvalid", referenced from:
      _vtenc_send_frame in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_AllowFrameReordering", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_AverageBitRate", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_ColorPrimaries", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_DataRateLimits", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_MaxKeyFrameInterval", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_MoreFramesAfterEnd", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_MoreFramesBeforeStart", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_PixelAspectRatio", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_ProfileLevel", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_TransferFunction", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTCompressionPropertyKey_YCbCrMatrix", referenced from:
      _vtenc_create_encoder in libavcodec.a(videotoolboxenc.o)
  "_kVTEncodeFrameOptionKey_ForceKeyFrame", referenced from:
      _vtenc_send_frame in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Baseline_1_3", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Baseline_3_0", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Baseline_3_1", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Baseline_3_2", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Baseline_4_1", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_High_5_0", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_3_0", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_3_1", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_3_2", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_4_0", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_4_1", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_kVTProfileLevel_H264_Main_5_0", referenced from:
      _vtenc_init in libavcodec.a(videotoolboxenc.o)
  "_lzma_code", referenced from:
      _decode_frame in libavcodec.a(tiff.o)
  "_lzma_end", referenced from:
      _decode_frame in libavcodec.a(tiff.o)
  "_lzma_stream_decoder", referenced from:
      _decode_frame in libavcodec.a(tiff.o)
  "_swr_alloc", referenced from:
      _opus_decode_init in libavcodec.a(opusdec.o)
  "_swr_close", referenced from:
      _opus_decode_packet in libavcodec.a(opusdec.o)
      _opus_decode_flush in libavcodec.a(opusdec.o)
  "_swr_convert", referenced from:
      _opus_decode_packet in libavcodec.a(opusdec.o)
  "_swr_free", referenced from:
      _opus_decode_close in libavcodec.a(opusdec.o)
  "_swr_init", referenced from:
      _opus_decode_packet in libavcodec.a(opusdec.o)
  "_swr_is_initialized", referenced from:
      _opus_decode_packet in libavcodec.a(opusdec.o)
  "_uncompress", referenced from:
      _id3v2_read_internal in libavformat.a(id3v2.o)
      _mov_read_cmov in libavformat.a(mov.o)
      _swf_read_packet in libavformat.a(swfdec.o)
      _decode_frame in libavcodec.a(cscd.o)
      _decode_frame in libavcodec.a(dxa.o)
      _decode_block in libavcodec.a(exr.o)
      _g2m_decode_frame in libavcodec.a(g2meet.o)
      ...
     (maybe you meant: _ff_snappy_peek_uncompressed_length, _ff_snappy_uncompress , _ff_lzf_uncompress )
  "_zlibCompileFlags", referenced from:
      _http_read_header in libavformat.a(http.o)
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make: *** [untrunc] Error 1
anthwlock commented 2 years ago

Will untrunc be limited by disabling these

No, I don't think so. Currently untrunc does not make use of any hardware acceleration features (like the Apple specific vda, audiotoolbox and videotoolbox). What matters to untrunc is that ffmpeg's avcodec_decode_audio4 and avcodec_decode_video2 routines consume exactly one frame, even if the passed in AVPacket references a buffer which contains >1 frames.

In my understanding this behavior (1 decoded frame per invocation) is/should not be influenced by the chosen decoder. In fact, I just looked and it says Some decoders may support multiple frames in a single AVPacket, such decoders would then just decode the first frame here.

and will we be gaining any benefits by using a more recent version of FFmpeg

In theory yes, however untrunc uses only a small subset of ffmpeg, and unless there was a bug in that subset, it does not really matter. There are however downsides to using versions >=3.4, because starting with that version:

  1. avcodec_decode_video2 expects its AVPacket to contain exactly one frame, see here. This is however (currently) only relevant to the mp4v codec.
  2. less problematic, avcodec_decode_audio4 now uses an internal buffer, which gets reallocated with every flush. This makes it very slow to just "try it out" (e.g. in -s mode), but currently untrunc doesn't do that anyway, since the false positive rate is often too high.
o3jvind commented 2 years ago

Thank you very much for your patience and for your thorough explanation(s). I have tried the compiled version of untrunc on a few machines now and it works flawlessly.

anthwlock commented 2 years ago

Great! For the record, the relevant commit is 878c94802b72d8a2d7c3acede91ab8663dfc6d24