search

holzschu / a-Shell-commands

shell commands, pre-compiled to webAssembly, ready to use in a-Shell
554 stars 21 forks source link

Addition of VideoToolbox hardware acceleration codecs in ffmpeg #8

Open LPFchan opened 3 years ago

LPFchan commented 3 years ago

Hi,

I've noticed when I was encoding a video and audio file together in a-Shell, it takes a bit of patience for it to finish and heats the device up considerably, which lead me into researching a bit more. Turns out ffmpeg does support hardware acceleration on iOS and it's called VideoToolbox. Here's more info from ffmpeg:

https://trac.ffmpeg.org/wiki/HWAccelIntro#VideoToolbox

As far as I can tell, a build of ffmpeg embedded within a-Shell has no ability of hardware acceleration. But, you can add --enable-videotoolbox flag in FFmpeg configure script to enable it.

$ ffmpeg -h encoder=h264_videotoolbox configuration: --cc=/Users/holzschu/src/Xcode_iPad/wasi-sdk/opt/bin/clang --ar=/Users/holzschu/src/Xcode _iPad/wasi-sdk/opt/bin/ar --ranlib=/Users/holzschu/src/Xcode_iPad/wasi-sdk/opt/bin/ranlib --disable-asm -- enable-cross-compile --disable-avfoundation --disable-appkit --disable-network --disable-pthreads --disabl e-debug --enable-libx264 --enable-gpl Codec 'h264_videotoolbox' is not recognized by FFmpeg.

Apparently you can check it in ./configure after running the build script: https://github.com/kewlbear/FFmpeg-iOS-build-script/issues/58

Here's a build configuration example shared by devs of some app (just in case it helps): https://medium.com/liveop-x-team/accelerating-h264-decoding-on-ios-with-ffmpeg-and-videotoolbox-1f000cb6c549

holzschu commented 3 years ago

Thanks for the detailed report and the links. That would obviously require compiling ffmpeg for iOS instead of WebAssembly, but that seems like a good idea anyway.

holzschu commented 2 years ago

Update: ffmpeg compiles for iOS with --enable-videotoolbox --disable-audiotoolbox.

Enabling Arm64 assembly, even using the tips described at https://github.com/DeviLeo/DLGPlayer/issues/19#issuecomment-384847629 did not work.

There are still many steps to make it compatible with the rest of a-Shell, but that's a beginning.

holzschu commented 2 years ago

Update: everything works except videotoolbox acceleration. I get some weird "DTS is invalid" errors, and the encoding stops.

holzschu commented 2 years ago

Update: ffmpeg compiled natively to arm64, with videotoolbox acceleration, is available for TestFlight testing. To make it work, you need to delete or rename the WebAssembly version.

lxgr commented 2 years ago

This is has been shipped, right?

holzschu commented 2 years ago

Yes. If you already installed the WebAssembly version of ffmpeg, you need to uninstall it.

m-ruhl commented 1 year ago

@holzschu The quality mode encoding is currently not working

Error: -q:v qscale not available for encoder. Use -b:v bitrate instead.

It should be supported for Apple Silicon (https://trac.ffmpeg.org/wiki/HWAccelIntro#VideoToolbox)

I think you just need to enable neon during compilation as Homebrew: https://github.com/homebrew-ffmpeg/homebrew-ffmpeg/blob/88ca387a1dd19923a957965cedd6724a30a29dcc/Formula/ffmpeg.rb#L141

holzschu commented 1 year ago

If I remember correctly, --enable-neon uses assembly code, and crashes immediately on all the Arm iPads and iPhones I've tried. It seems there are subtle differences between assembly for Arm iPads and assembly for Arm Macs.

m-ruhl commented 1 year ago

If I remember correctly, --enable-neon uses assembly code, and crashes immediately on all the Arm iPads and iPhones I've tried. It seems there are subtle differences between assembly for Arm iPads and assembly for Arm Macs.

Seems to me, that ffmpeg-kit has neon enabled:

https://github.com/arthenica/ffmpeg-kit/blob/47f85fa9ea3f8c34f3c817b87d8667b61b87d0bc/scripts/apple/ffmpeg.sh#L58

Maybe this helps