V4L2 Hardware Codecs support

joek85 commented 3 years ago

Hello, I am testing the latest release on Opi Plus2e sdcard version. i played a 1080p video using the default NuPlayer, it was very laggy and skips lots of frames, sometimes it freezes. on Skytube and NewPipe videos will play smoothly when reducing the resolution to 360p and below, hovering the mouse over the video will show some random flickering. it seems it is still not hardware accelerated? could you please point me to the right direction where to solve the problem? i would like to help.

rsglobal commented 3 years ago

Hi,

To do this you have to implement codec2 components that will support v4l2 request API. Or somehow use this: https://android.googlesource.com/platform/external/v4l2_codec2/

Both are not trivial and require a lot of knowledge, time and paitiance for harware-decoder bring-up.

joek85 commented 3 years ago

Hello,

I've enabled

CONFIG_MEDIA_SUPPORT CONFIG_MEDIA_CONTROLLER CONFIG_MEDIA_CONTROLLER_REQUEST_API CONFIG_V4L_MEM2MEM_DRIVERS CONFIG_VIDEO_SUNXI_CEDRUS

and successfully got /dev/video0 /dev/media0

now, looking at android v4l2_codec2 make file, it seems that it can only be built for device/google/cheets2 which is not available in the AOSP source tree. all other dependencies that are required by v4l2_codec2 are available except /device/google/cheets2/codec2/vdastore/include

however i will try to install VLC and see what happens, just to test if the kernel driver is working as expected.

rsglobal commented 3 years ago

device/google/cheets2

This was required on Android-10, Looks like Android-11 and master doesn't need it.

joek85 commented 3 years ago

This was required on Android-10,

you're right, my bad, didn't catch that. i am currently working on Android10, i don't have enough bandwidth to download Android11. i will try to port it to my source and hope it will compile with no problems.

acrolink commented 3 years ago

Any updates on this? Does this build support video/avc (H264) hardware encoding and decoding?

rsglobal commented 3 years ago

Codecs are coming. At least stateful (for rpi4).
https://android-review.googlesource.com/c/platform/external/v4l2_codec2/+/1850649 https://android-review.googlesource.com/c/device/amlogic/yukawa/+/1850767

CC: @konstat

KonstaT commented 3 years ago

@rsglobal Actually gave this a go a few days ago and couldn't get much out of it. https://forum.xda-developers.com/t/dev-rom-unofficial-lineageos-18-1-android-11-for-raspberry-pi-4-b.4212945/page-14#post-85784801 Don't think I missed anything obvious so I'm not sure why it's not finding any device for H.264.

Also seeing John Stultz's tweet (https://twitter.com/johnstultz_work/status/1449268379622068226) I wasn't expecting this to be fully functional quite yet anyway.

rsglobal commented 3 years ago

Ah, I see.

Just want to double check, have you applied Dmitry's patches and configurations from his readme?

https://android-review.googlesource.com/q/project:platform%252Fexternal%252Fv4l2_codec2+status:open

KonstaT commented 3 years ago

Yes, had all the 14 gerrit patches on top of master and followed the new readme.

What I'm not sure about is the ro.vendor.v4l2_codec2.drm_device_name and ro.vendor.v4l2_codec2.drm_device_path for VC4/V3D.

volkertb commented 2 years ago

Quoting this reply in the duplicate topic:

Duplicate of #152 .

Currently only software codecs are supported. There is a way to integrate stateful codecs support which we will do for rpi4 near future, but allwinner has only stateless codec APIs which require a lot of work to enable on mainline Android.

Have you considered vicodec? Apparently, with Dafna Hirschfeld's improvements, it can provide stateful codec emulation for stateless hardware codecs, such as the one in the Pinephone's SoC.

More info on this:

rsglobal commented 2 years ago

@volkertb ,

Unfortunately all I can find in regard to vicodec in the kernel code and Dafna's blog is an stateless API for software encoding/decoding based on FWHT algorithm.

volkertb commented 2 years ago

In her blog she says that the vicodec driver should expose three interfaces under /dev:

One for a stateful encoder (not applicable for the Allwinner SoC)
One for a stateless decoder (which basically passes through to the new stateless "Request API")
One for a stateful decoder (which should provide stateful decoder emulation and talks to stateless codec drivers such as Cedrus on the backend through the Request API)

So as far as I understand it, the chain goes like this:

V4L2 -> vicodec -> Cedrus (the driver for the Allwinner video engine)

This seems to be the relevant commit:

https://github.com/torvalds/linux/commit/997deb811bf55731a7c0bd972d8d0a266ef8d7e5

There is also this PDF document from Hans Verkuil at Cisco Systems: https://elinux.org/images/7/76/Elce2019v4l2.pdf

In that document, he specifically refers to vicodec as a "virtual driver" that "emulates a stateful encoder, a stateful decoder and a stateless decoder".

Would this work for GloDroid?

Perhaps we could email Dafna and/or Hans for questions if there are still any missing puzzle pieces. :slightly_smiling_face:

rsglobal commented 1 year ago

ffmpeg-based codec2 service is a good candidate to add stateless (and stateful) v4l2-based codec support:

ffmpeg + request API patches: https://github.com/jernejsk/FFmpeg
codec2 to ffmpeg layer (dirty at the moment): https://github.com/goffioul/external-stagefright-plugins

KonstaT commented 1 year ago

I managed to get FFmpeg H.264/H.265/VP8/VP9 software decoders working on Pi 4 with slightly modified https://github.com/waydroid/android_external_ffmpeg https://github.com/waydroid/android_external_stagefright-plugins and relevant patches to libstagefright on frameworks/av.

Tried to use FFmpeg's h264_v4l2m2m for H.264 hardware decoding but not getting any buffers out of it.

01-15 17:19:18.641   335  1920 I GenericSource: start
01-15 17:19:18.662   335  1931 D CCodec  : allocate(c2.android.aac.decoder)
01-15 17:19:18.663   335  1932 D CCodec  : allocate(c2.ffmpeg.h264.decoder)
01-15 17:19:18.674   335  1932 I CCodec  : setting up 'default' as default (vendor) store
01-15 17:19:18.674   282   695 D android.hardware.media.c2@1.2-service: createComponent: c2.ffmpeg.h264.decoder
01-15 17:19:18.676   282   695 D C2FFMPEGVideoDecodeComponent: C2FFMPEGVideoDecodeComponent: mediaType = video/avc
01-15 17:19:18.677   335  1931 I CCodec  : setting up 'default' as default (vendor) store
01-15 17:19:18.678   343   701 V C2Store : in init
01-15 17:19:18.679   343   701 V C2Store : loading dll
01-15 17:19:18.691   335  1931 I CCodec  : Created component [c2.android.aac.decoder]
01-15 17:19:18.697   335  1931 D CCodecConfig: read media type: audio/mp4a-latm
01-15 17:19:18.668   335   335 I CodecLooper: type=1400 audit(0.0:153): avc: denied { transfer } for scontext=u:r:mediaserver:s0 tcontext=u:r:init:s0 tclass=binder permissive=1
01-15 17:19:18.706   335  1932 I CCodec  : Created component [c2.ffmpeg.h264.decoder]
01-15 17:19:18.707   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
01-15 17:19:18.707   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: output.subscribed-indices.values
01-15 17:19:18.707   335  1932 D CCodecConfig: read media type: video/avc
01-15 17:19:18.708   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: input.buffers.allocator-ids.values
01-15 17:19:18.708   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.allocator-ids.values
01-15 17:19:18.709   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.allocator-ids.values
01-15 17:19:18.709   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
01-15 17:19:18.709   335  1931 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.pool-ids.values
01-15 17:19:18.711   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.max-count.values
01-15 17:19:18.712   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: output.subscribed-indices.values
01-15 17:19:18.712   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: input.buffers.allocator-ids.values
01-15 17:19:18.713   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.allocator-ids.values
01-15 17:19:18.713   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.allocator-ids.values
01-15 17:19:18.713   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: output.buffers.pool-ids.values
01-15 17:19:18.713   335  1932 D ReflectedParamUpdater: extent() != 1 for single value type: algo.buffers.pool-ids.values
01-15 17:19:18.716   335  1932 D ReflectedParamUpdater: ignored struct field coded.color-format.locations
01-15 17:19:18.716   335  1932 D CCodecConfig: ignoring local param raw.size (0xd2001800) as it is already supported
01-15 17:19:18.717   335  1932 D ReflectedParamUpdater: ignored struct field raw.hdr-static-info.mastering
01-15 17:19:18.719   335  1932 I CCodecConfig: query failed after returning 8 values (BAD_INDEX)
01-15 17:19:18.719   335  1932 D CCodecConfig: c2 config diff is Dict {
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 coded.pl.level = 20491
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 coded.pl.profile = 20482
01-15 17:19:18.719   335  1932 D CCodecConfig:   Buffer coded.raw-codec-data.value = {
01-15 17:19:18.719   335  1932 D CCodecConfig:   }
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 default.color.matrix = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 default.color.primaries = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 default.color.range = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 default.color.transfer = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 input.delay.value = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   string input.media-type.value = "video/avc"
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 output.delay.value = 8
01-15 17:19:18.719   335  1932 D CCodecConfig:   string output.media-type.value = "video/raw"
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.pixel-format.value = 256
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::i32 raw.rotation.flip = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::i32 raw.rotation.value = 0
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.sar.height = 1
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.sar.width = 1
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.size.height = 240
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.size.width = 320
01-15 17:19:18.719   335  1932 D CCodecConfig:   c2::u32 raw.surface-scaling.value = 1
01-15 17:19:18.719   335  1932 D CCodecConfig: }
01-15 17:19:18.672   282   282 I HwBinder:282_3: type=1400 audit(0.0:154): avc: denied { transfer } for scontext=u:r:init:s0 tcontext=u:r:mediaserver:s0 tclass=binder permissive=1
01-15 17:19:18.720   335  1932 W ColorUtils: expected specified color aspects (0:0:0:0)
01-15 17:19:18.724   335  1928 D SurfaceUtils: disconnecting from surface 0xee7c3278, reason onConfigure
01-15 17:19:18.724   335  1929 D SurfaceUtils: connecting to surface 0xee7c3278, reason connectToSurface
01-15 17:19:18.725   335  1929 I MediaCodec: [c2.ffmpeg.h264.decoder] setting surface generation to 343041
01-15 17:19:18.725   335  1929 D SurfaceUtils: disconnecting from surface 0xee7c3278, reason connectToSurface(reconnect)
01-15 17:19:18.726   335  1929 D SurfaceUtils: connecting to surface 0xee7c3278, reason connectToSurface(reconnect)
01-15 17:19:18.726   335  1931 I CCodecConfig: query failed after returning 20 values (BAD_INDEX)
01-15 17:19:18.726   335  1931 D CCodecConfig: c2 config diff is Dict {
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 coded.aac-packaging.value = 0
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 coded.bitrate.value = 64000
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 coded.pl.level = 0
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 coded.pl.profile = 8192
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::i32 coding.drc.album-mode.value = 0
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::float coding.drc.attenuation-factor.value = 1
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::float coding.drc.boost-factor.value = 1
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::i32 coding.drc.compression-mode.value = 3
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::i32 coding.drc.effect-type.value = 3
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::float coding.drc.encoded-level.value = 0.25
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::float coding.drc.reference-level.value = -16
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 input.buffers.max-size.value = 8192
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 input.delay.value = 0
01-15 17:19:18.726   335  1931 D CCodecConfig:   string input.media-type.value = "audio/mp4a-latm"
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 output.delay.value = 2
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::float output.drc.output-loudness.value = 0.25
01-15 17:19:18.726   335  1931 D CCodecConfig:   string output.media-type.value = "audio/raw"
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 raw.channel-count.value = 1
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 raw.channel-mask.value = 0
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 raw.max-channel-count.value = 8
01-15 17:19:18.726   335  1931 D CCodecConfig:   c2::u32 raw.sample-rate.value = 44100
01-15 17:19:18.726   335  1931 D CCodecConfig: }
01-15 17:19:18.728   335  1931 D CCodec  : [c2.android.aac.decoder] buffers are bound to CCodec for this session
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for durationUs
01-15 17:19:18.728   335  1932 D CCodec  : [c2.ffmpeg.h264.decoder] buffers are bound to CCodec for this session
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for track-id
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for language
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for encoder-delay
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for encoder-padding
01-15 17:19:18.728   335  1931 D CCodecConfig: no c2 equivalents for flags
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for durationUs
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for frame-count
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for track-id
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for language
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for display-width
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for display-height
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for csd-1
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for native-window
01-15 17:19:18.728   335  1932 D CCodecConfig: no c2 equivalents for flags
01-15 17:19:18.729   335  1931 D CCodecConfig: config failed => CORRUPTED
01-15 17:19:18.729   335  1932 D CCodecConfig: config failed => CORRUPTED
01-15 17:19:18.729   335  1931 D CCodecConfig: c2 config diff is   c2::u32 coded.bitrate.value = 240291
01-15 17:19:18.729   335  1931 D CCodecConfig:   c2::u32 raw.channel-count.value = 2
01-15 17:19:18.729   335  1931 D CCodecConfig:   c2::u32 raw.sample-rate.value = 48000
01-15 17:19:18.729   335  1932 D CCodecConfig: c2 config diff is   c2::u32 coded.pl.profile = 20484
01-15 17:19:18.729   335  1932 D CCodecConfig:   c2::u32 raw.size.height = 1080
01-15 17:19:18.729   335  1932 D CCodecConfig:   c2::u32 raw.size.width = 1920
01-15 17:19:18.730   335  1931 W Codec2Client: query -- param skipped: index = 1107298332.
01-15 17:19:18.730   335  1932 D CCodec  : client requested max input size 321818, which is smaller than what component recommended (1048576); overriding with component recommendation.
01-15 17:19:18.730   335  1931 D CCodec  : client requested max input size 1312, which is smaller than what component recommended (8192); overriding with component recommendation.
01-15 17:19:18.730   335  1932 W CCodec  : This behavior is subject to change. It is recommended that app developers double check whether the requested max input size is in reasonable range.
01-15 17:19:18.730   335  1931 W CCodec  : This behavior is subject to change. It is recommended that app developers double check whether the requested max input size is in reasonable range.
01-15 17:19:18.730   335  1932 D CCodec  : encoding statistics level = 0
01-15 17:19:18.730   335  1931 D CCodec  : encoding statistics level = 0
01-15 17:19:18.730   335  1932 D CCodec  : setup formats input: AMessage(what = 0x00000000) = {
01-15 17:19:18.730   335  1932 D CCodec  :   int32_t height = 1080
01-15 17:19:18.730   335  1932 D CCodec  :   int32_t level = 2048
01-15 17:19:18.730   335  1932 D CCodec  :   string mime = "video/avc"
01-15 17:19:18.730   335  1932 D CCodec  :   int32_t profile = 8
01-15 17:19:18.730   335  1932 D CCodec  :   Buffer raw-codec-data = {
01-15 17:19:18.730   335  1932 D CCodec  :   }
01-15 17:19:18.730   335  1932 D CCodec  :   int32_t width = 1920
01-15 17:19:18.730   335  1932 D CCodec  :   Rect crop(0, 0, 1919, 1079)
01-15 17:19:18.730   335  1932 D CCodec  :   int64_t android._C2MemoryUsage = 0
01-15 17:19:18.730   335  1932 D CCodec  :   int32_t max-input-size = 1048576
01-15 17:19:18.730   335  1932 D CCodec  : }
01-15 17:19:18.730   335  1931 D CCodec  : setup formats input: AMessage(what = 0x00000000) = {
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-drc-album-mode = 0
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-drc-boost-level = 127
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-drc-cut-level = 127
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-drc-effect-type = 3
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-encoded-target-level = -1
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-max-output-channel_count = 8
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t aac-target-ref-level = 64
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t bitrate = 240291
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t channel-count = 2
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t channel-mask = 0
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t level = 0
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t max-input-size = 8192
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t max-output-channel-count = 8
01-15 17:19:18.730   335  1931 D CCodec  :   string mime = "audio/mp4a-latm"
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t profile = 2
01-15 17:19:18.730   335  1931 D CCodec  :   int32_t sample-rate = 48000
01-15 17:19:18.730   335  1931 D CCodec  : }
01-15 17:19:18.731   335  1932 D CCodec  : setup formats output: AMessage(what = 0x00000000) = {
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t android._color-format = 256
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t android._video-scaling = 1
01-15 17:19:18.731   335  1932 D CCodec  :   Rect crop(0, 0, 1919, 1079)
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t color-standard = 1
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t color-range = 2
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t color-transfer = 3
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t height = 1080
01-15 17:19:18.731   335  1932 D CCodec  :   string mime = "video/raw"
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t rotation-degrees = 0
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t sar-height = 1
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t sar-width = 1
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t width = 1920
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t android._dataspace = 260
01-15 17:19:18.731   335  1932 D CCodec  :   int32_t color-format = 2130708361
01-15 17:19:18.731   335  1932 D CCodec  : }
01-15 17:19:18.731   335  1931 D CCodec  : setup formats output: AMessage(what = 0x00000000) = {
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-drc-album-mode = 0
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-drc-boost-level = 127
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-drc-cut-level = 127
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-drc-effect-type = 3
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-drc-output-loudness = -1
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-encoded-target-level = -1
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-max-output-channel_count = 8
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t aac-target-ref-level = 64
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t channel-count = 2
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t channel-mask = 0
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t max-output-channel-count = 8
01-15 17:19:18.731   335  1931 D CCodec  :   string mime = "audio/raw"
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t sample-rate = 48000
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t encoder-delay = 0
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t encoder-padding = 0
01-15 17:19:18.731   335  1931 D CCodec  :   int32_t android._config-pcm-encoding = 2
01-15 17:19:18.731   335  1931 D CCodec  : }
01-15 17:19:18.731   335  1932 I CCodecConfig: query failed after returning 8 values (BAD_INDEX)
01-15 17:19:18.731   335  1931 I CCodecConfig: query failed after returning 20 values (BAD_INDEX)
01-15 17:19:18.731   335  1929 I MediaCodec: MediaCodec will operate in async mode
01-15 17:19:18.731   335  1931 I MediaCodec: MediaCodec will operate in async mode
01-15 17:19:18.731   335  1928 D MediaCodec: keep callback message for reclaim
01-15 17:19:18.732   335  1930 D MediaCodec: keep callback message for reclaim
01-15 17:19:18.732   282  1933 D C2FFMPEGVideoDecodeComponent: onInit
01-15 17:19:18.745   282  1933 I FFMPEG  : FFMPEG initialized: 5.1.2 (KonstaKANG)
01-15 17:19:18.756   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb2450
01-15 17:19:18.775   335  1931 W Codec2Client: query -- param skipped: index = 1342179345.
01-15 17:19:18.775   335  1931 W Codec2Client: query -- param skipped: index = 2415921170.
01-15 17:19:18.775   335  1931 W Codec2Client: query -- param skipped: index = 1610614798.
01-15 17:19:18.777   343   394 D BufferPoolAccessor2.0: bufferpool2 0xb4000071ae98ee48 : 0(0 size) total buffers - 0(0 size) used buffers - 50/54 (recycle/alloc) - 4/53 (fetch/transfer)
01-15 17:19:18.777   343   394 D BufferPoolAccessor2.0: Destruction - bufferpool2 0xb4000071ae98ee48 cached: 0/0M, 0/0% in use; allocs: 54, 93% recycled; transfers: 53, 92% unfetched
01-15 17:19:18.778   335  1931 D CCodecBufferChannel: [c2.android.aac.decoder#973] Created input block pool with allocatorID 16 => poolID 18 - OK (0)
01-15 17:19:18.780   335  1423 D BufferPoolAccessor2.0: bufferpool2 0xef643628 : 0(0 size) total buffers - 0(0 size) used buffers - 51/59 (recycle/alloc) - 8/112 (fetch/transfer)
01-15 17:19:18.781   335  1252 D BufferPoolAccessor2.0: Destruction - bufferpool2 0xef643628 cached: 0/0M, 0/0% in use; allocs: 59, 86% recycled; transfers: 112, 93% unfetched
01-15 17:19:18.781   335  1931 I CCodecBufferChannel: [c2.android.aac.decoder#973] Created output block pool with allocatorID 16 => poolID 27 - OK
01-15 17:19:18.784   335  1931 D CCodecBufferChannel: [c2.android.aac.decoder#973] Configured output block pool ids 27 => OK
01-15 17:19:18.787   282  1933 D C2FFMPEGVideoDecodeComponent: initDecoder: 0xb40000717de55650 [h264], 1920 x 1080, video/avc, use-drm-prime = 1
01-15 17:19:18.788   335  1932 W Codec2Client: query -- param skipped: index = 1342179345.
01-15 17:19:18.789   335  1932 W Codec2Client: query -- param skipped: index = 2415921170.
01-15 17:19:18.789   335  1932 W Codec2Client: query -- param skipped: index = 1610614798.
01-15 17:19:18.792   335  1932 D CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] Created input block pool with allocatorID 16 => poolID 19 - OK (0)
01-15 17:19:18.794   343  1934 D SimpleC2Component: Using output block pool with poolID 27 => got 27 - 0
01-15 17:19:18.794   335  1932 D CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] Query output surface allocator returned 0 params => BAD_INDEX (6)
01-15 17:19:18.795   335  1932 I CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] Created output block pool with allocatorID 18 => poolID 17 - OK
01-15 17:19:18.795   335  1932 D CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] Configured output block pool ids 17 => OK
01-15 17:19:18.797   335  1932 D Codec2-OutputBufferQueue: C2SurfaceSyncMemory created 20(20)
01-15 17:19:18.797   335  1932 D Codec2-OutputBufferQueue: remote graphic buffer migration 0/0
01-15 17:19:18.797   335  1932 D Codec2Client: setOutputSurface -- generation=343041 consumer usage=0x900 sync
01-15 17:19:18.803   282   696 D C2BqBuffer: local generation change 343041 , bqId: 8104603287553 migrated buffers # 0
01-15 17:19:18.803   335  1932 D Codec2Client: Surface configure completed
01-15 17:19:18.792   335   335 I CodecLooper: type=1400 audit(0.0:155): avc: denied { use } for path="/dev/ashmem8f56a513-e4f0-4c94-bc5d-cfbd8aaad2a7" dev="tmpfs" ino=446 scontext=u:r:init:s0 tcontext=u:r:mediaserver:s0 tclass=fd permissive=1
01-15 17:19:18.823   343  1934 I C2SoftAacDec: Reconfiguring decoder: 0->48000 Hz, 0->2 channels
01-15 17:19:18.830   335  1929 W ResourceManagerService: Ignoring request to add new resource entry with value <= 0
01-15 17:19:18.836   335  1931 D CCodecConfig: c2 config diff is   c2::u32 raw.channel-mask.value = 12
01-15 17:19:18.837   335  1931 D CCodecBuffers: [c2.android.aac.decoder#973:Output[N]] popFromStashAndRegister: at 0us, output format changed to AMessage(what = 0x00000000) = {
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-drc-album-mode = 0
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-drc-boost-level = 127
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-drc-cut-level = 127
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-drc-effect-type = 3
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-drc-output-loudness = -1
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-encoded-target-level = -1
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-max-output-channel_count = 8
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t aac-target-ref-level = 64
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t channel-count = 2
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t channel-mask = 12
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t max-output-channel-count = 8
01-15 17:19:18.837   335  1931 D CCodecBuffers:   string mime = "audio/raw"
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t sample-rate = 48000
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t encoder-delay = 0
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t encoder-padding = 0
01-15 17:19:18.837   335  1931 D CCodecBuffers:   int32_t android._config-pcm-encoding = 2
01-15 17:19:18.837   335  1931 D CCodecBuffers: }
01-15 17:19:18.835     0     0 W audit   : audit_lost=6 audit_rate_limit=5 audit_backlog_limit=64
01-15 17:19:18.843     0     0 E audit   : rate limit exceeded
01-15 17:19:18.847   282   696 D C2Store : Using ION
01-15 17:19:18.848   282  1933 D SimpleC2Component: Using output block pool with poolID 17 => got 17 - 0
01-15 17:19:18.853   309   309 W AudioFlinger: createTrack_l(): mismatch between requested flags (00000008) and output flags (00000002)
01-15 17:19:18.854   309   309 D AudioFlinger: Client defaulted notificationFrames to 12288 for frameCount 24576
01-15 17:19:18.857   309   309 D AF::TrackHandle: OpPlayAudio: track:57 usage:1 not muted
01-15 17:19:18.873   282  1933 D C2FFMPEGVideoDecodeComponent: openDecoder: begin to open ffmpeg decoder(h264) now
01-15 17:19:18.883   335   335 D AudioTrack: getTimestamp_l(10): device stall time corrected using current time 223371276365
01-15 17:19:18.911   282  1933 I FFMPEG  : [h264_v4l2m2m @ 0xb40000717de55650] Using device /dev/video10
01-15 17:19:18.911   282  1933 I FFMPEG  : [h264_v4l2m2m @ 0xb40000717de55650] driver 'bcm2835-codec' on card 'bcm2835-codec-decode' in mplane mode
01-15 17:19:18.911   282  1933 I FFMPEG  : [h264_v4l2m2m @ 0xb40000717de55650] requesting formats: output=H264 capture=YU12
01-15 17:19:18.948   282  1933 D C2FFMPEGVideoDecodeComponent: openDecoder: open ffmpeg video decoder(h264) success, caps = 00060020
01-15 17:19:18.962   479   734 D CoreBackPreview: Window{c6cc9ab u0 Splash Screen com.android.gallery3d EXITING}: Setting back callback null
01-15 17:19:18.962   309   434 D AudioFlinger: mixer(0xb400007f362f39a0) throttle end: throttle time(38)
01-15 17:19:18.969   479   531 W InputManager-JNI: Input channel object 'c6cc9ab Splash Screen com.android.gallery3d (client)' was disposed without first being removed with the input manager!
01-15 17:19:19.010   661   702 W FrameTracker: Missed App frame:JANK_APP_DEADLINE_MISSED, 17897, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing HWUI jank callback for vsyncId: 17897, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed App frame:UNKNOWN: 74, 17970, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed SF frame:UNKNOWN: 74, 17970, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing HWUI jank callback for vsyncId: 17970, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing HWUI jank callback for vsyncId: 17999, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed App frame:UNKNOWN: 328, 18064, 71062308, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed App frame:UNKNOWN: 74, 18100, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed SF frame:UNKNOWN: 74, 18100, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing HWUI jank callback for vsyncId: 18100, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missed App frame:UNKNOWN: 72, 18203, 0, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing HWUI jank callback for vsyncId: 18203, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.010   661   702 W FrameTracker: Missing SF jank callback for vsyncId: 18210, CUJ=J<SPLASHSCREEN_EXIT_ANIM>
01-15 17:19:19.011   661   692 V PerfettoTrigger: Not triggering com.android.telemetry.interaction-jank-monitor-39 - not enough time since last trigger
01-15 17:19:19.039   282  1933 D C2FFMPEGVideoDecodeComponent: WorkQueue: queue full, output delay set to 18
01-15 17:19:19.042   335  1932 D CCodecConfig: c2 config diff is   c2::u32 output.delay.value = 18
01-15 17:19:19.043   335  1932 D Codec2-OutputBufferQueue: set max dequeue count 25 from update
01-15 17:19:19.057   282  1933 D C2FFMPEGVideoDecodeComponent: WorkQueue: queue full, output delay set to 34
01-15 17:19:19.058   335  1932 D CCodecConfig: c2 config diff is   c2::u32 output.delay.value = 34
01-15 17:19:19.059   335  1932 D Codec2-OutputBufferQueue: set max dequeue count 41 from update
01-15 17:19:19.119   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb1550
01-15 17:19:19.120   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb0ef0
01-15 17:19:19.120   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb0530
01-15 17:19:19.120   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb0530
01-15 17:19:19.120   312   404 W TransactionTracing: Could not find layer handle 0xb4000076cbbb0ef0
01-15 17:19:19.845   306   306 I WifiHAL : Creating message to get link statistics; iface = 8
01-15 17:19:19.845   306   306 D WifiHAL : error code -95 mapped to -3
01-15 17:19:19.846   479   594 E WifiVendorHal: getWifiLinkLayerStats_1_6_Internal(l.1199) failed {.code = ERROR_NOT_SUPPORTED, .description = }
01-15 17:19:20.009   335  1929 D CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] buffers after EOS ignored (0 us)
01-15 17:19:20.009   335  1929 D CCodecBufferChannel: [c2.ffmpeg.h264.decoder#19] buffers after EOS ignored (0 us)

@goffioul Any thoughts should the V4L2 API even work with this stagefright plugin?

Pi's stateless H.265 hw decoder seems to be entirely different story as it's not even supported on mainline FFmpeg but requires ton of external patches (https://github.com/jc-kynesim/rpi-ffmpeg) and apparently produces buffers in SAND format(?).

H.264 hardware video decoding and encoding is now somewhat working on my LineageOS 20 builds and Raspberry Vanilla sources (https://github.com/raspberry-vanilla) with AOSP v4l2_codec2 anyway (decoding still produces random glitches/green tint on occasions).

acrolink commented 1 year ago

@KonstaT

Great news, thank you very much. Should I just go ahead, download and test? Is it ready?

goffioul commented 1 year ago

@KonstaT From the logs, it seems you're not using the latest version if my repo (or patch set), so you may to update that. Regarding your question, I don't have enough knowledge of v4l2 codec integration in ffmpeg to answer it. The code now uses the generic HW accel API from FFMPEG, so if the v4l2_m2m codec adheres to it, it should work, in theory...

KonstaT commented 1 year ago

@goffioul Thanks. Yeah, I had some strategic reverts for testing purposes. Didn't seem to make any difference, though. Wouldn't be too surprising if I have some configuration issue on FFmpeg side either.

KonstaT commented 1 year ago

Managed to get both hevc_v4l2request hwaccel for stateless H.265 hw decoding and h264_v4l2m2m for stateful H.264 hw decoding working after merging jc-kynesim's FFmpeg branch! https://twitter.com/KonstaKANG/status/1616054277629218819

Only hevc_v4l2request is actually listed as hwaccel on FFmpeg. h264_v4l2m2m is just a decoder. H.265 hw decoding seems fine at 1080p30 (haven't found test clips with higher frame rate or 4k). H.264 hw decoding seems fine at 1080p30 as well but can't quite keep up with 1080p60. h264_v4l2m2m actually performs worse than FFmpeg's software decoder that can also do 1080p60. :o FFmpeg's h264 software decoder also outperforms Android's own h264 software decoder as that can't keep up with 1080p60 either.

@goffioul Couple of questions about the libstagefright integration.

I see you have have the plugin in vendor partition. I had to move ffmpeg and the plugin to system partition as libstagefright is framework only. Would making libstagefright vendor available allow moving ffmpeg/plugin to vendor as well or how you've managed that? It looked like libstagefright has other dependencies as well so I didn't even bother trying yet.

What patches to libstagefright at frameworks/av are relevant if I just want to use the codec2 plugin and none of the old omx stuff?

Also somewhat unrelated, but what's currently the place to keep up with latest developments with Android on x86? Android-x86, BlissOS, Waydroid, or something else?

goffioul commented 1 year ago

I you only want the codec2 stuff, you might as well try without any of the /framework/av patches and see where this gets you (that also assumes you're not interested in the FFMPEG based extractor as well).

The Android on x86 community is unfortunately quite fragmented and small. The original developer hasn't been active lately and the mailing lists are more or less in limbo. There's an android-x86-dev channel on BlissLabs slack channel, where some of us are active. I know there's also the discord and telegram community, but I don't use that personally.

hmtheboy154 commented 1 year ago

Also somewhat unrelated, but what's currently the place to keep up with latest developments with Android on x86? Android-x86, BlissOS, Waydroid, or something else?

Telegram, we chat there a lot If you want private, then we use Slack which Micheal mentioned it

rsglobal commented 11 months ago

Support for HW video codecs added:

https://github.com/GloDroidCommunity/pine64-pinephone/commit/1fce8f8fd3e8d089b2dddb27910314fc946bebd8

BTW, if somebody wants to see updates for OPI3, OPI-Plus2E, and other Sunxi devices that were supported previously, open an issue for that.

GloDroid / glodroid_manifest

V4L2 Hardware Codecs support #152