Android AHardwareBuffer support for non-standard formats

marius-pelegrin-arm commented 1 year ago

What it's about

A Vulkan application on Android can use AHardwareBuffers with a "non-standard" format, (ie. not listed in the official documentation) and create textures from it in a perfectly safe and defined way. However, it is an undefined behavior for an application to create such a buffer itself - the buffer is often created by vendor applications that know what's happening "under the hood", as for the images yielded by the camera. Thus, when capturing an application that uses externally created AHardwareBuffers with non-standard formats, the same AHardwareBuffers cannot be created at replay time.

The current solution in GFXReconstruct is to completely ignore the issue and still create the buffer with an unauthorized format (which still works on most devices as the format is still known "internally" by Android) and as the format is not recognized in GetHardwareBufferFormatBpp, the buffer is not filled and used as-is by the following Vulkan commands, resulting in textures with undefined content.

The issue

Of course, undefined behavior is an issue in itself, but not filling the buffer also leads to artifacts and thus different results than the original application captured. Finally, absolutely no warning/error/message is displayed by GFXReconstruct.

Proposed solution

I think it is impossible to have at the same time a replay with the exact same behavior and output as the application because it is impossible to know directly what is inside an AHardwareBuffer with a non-standard format. What I think should be a good option is to give the choice to the user:

Having a replay with a behavior as close as possible to the application. This is what GFXReconstruct partly does. The AHardwareBuffer could be created as it is now (it will works on most devices), left empty, and used as-is by the future Vulkan commands, that thus remains the same. The differences would be that as this non-standard format can be detected at capture time, there is no need to write a FillMemoryCommand, and that a warning could be issued at both capture and replay time to signal that the replay will have a different, undefined output.
Having a replay with an output as close as possible to the application. At capture time, GFXReconstruct could retrieve the data in a texture created from such a buffer (by format conversion on a copy, memory mapping or whatever) and save an AHardwareBuffer creation command with a standard format "close" to the format of the Vulkan texture. This will very slightly alter the behavior, as the Vulkan textures at replay time will be created from an AHardwareBuffer with a different format. But at least, the content of the textures will be the exact same.

One of the two option could be the default behavior and the other a simple option passed through and environment variable at capture time.

bradgrantham-lunarg commented 1 year ago

marius-pelegrin-arm commented 1 year ago

Indeed I had not seen this PR, thanks. The issue is that it only partially solves the issue for one known non-standard format, I think a "generic" solution would be preferable.

bradgrantham-lunarg commented 1 year ago

Sorry, I had not meant that #884 was meant to solve this issue. I only wanted to note that we had seen it before and are aware of it.

bradgrantham-lunarg commented 7 months ago

Some more detail:

In GFXReconstruct capture, we query the AHB format, lock the pixels, and dump them out into a FillMemoryCommand for recreation of the AHB in replay.
In GFXReconstruct replay, we create an AHardwareBuffer using AHardwareBuffer_allocate
AHardwareBuffer_allocate calls AHardwareBuffer_isValidDescriptionpretty early in the function to validate the format and returns with BAD_VALUE if that returned false.
AHardwareBuffer_isValidDescription calls AHardwareBuffer_isValidPixelFormatpretty early in the function and that returns false if that fails.

AHardwareBuffer_isValidPixelFormat fails if the format isn't one of the following "standard" formats:

AHARDWAREBUFFER_FORMAT_R8_UNORM
AHARDWAREBUFFER_FORMAT_R16_UINT
AHARDWAREBUFFER_FORMAT_R16G16_UINT
AHARDWAREBUFFER_FORMAT_R8G8B8A8_UNORM
AHARDWAREBUFFER_FORMAT_R8G8B8X8_UNORM
AHARDWAREBUFFER_FORMAT_R5G6B5_UNORM
AHARDWAREBUFFER_FORMAT_R8G8B8_UNORM
AHARDWAREBUFFER_FORMAT_R16G16B16A16_FLOAT
AHARDWAREBUFFER_FORMAT_R10G10B10A2_UNORM
AHARDWAREBUFFER_FORMAT_R10G10B10A10_UNORM
AHARDWAREBUFFER_FORMAT_BLOB
AHARDWAREBUFFER_FORMAT_D16_UNORM
AHARDWAREBUFFER_FORMAT_D24_UNORM
AHARDWAREBUFFER_FORMAT_D24_UNORM_S8_UINT
AHARDWAREBUFFER_FORMAT_D32_FLOAT
AHARDWAREBUFFER_FORMAT_D32_FLOAT_S8_UINT
AHARDWAREBUFFER_FORMAT_S8_UINT
AHARDWAREBUFFER_FORMAT_Y8Cb8Cr8_420
AHARDWAREBUFFER_FORMAT_B8G8R8A8_UNORM
AHARDWAREBUFFER_FORMAT_YV12
AHARDWAREBUFFER_FORMAT_Y8
AHARDWAREBUFFER_FORMAT_Y16
AHARDWAREBUFFER_FORMAT_RAW16
AHARDWAREBUFFER_FORMAT_RAW10
AHARDWAREBUFFER_FORMAT_RAW12
AHARDWAREBUFFER_FORMAT_RAW_OPAQUE
AHARDWAREBUFFER_FORMAT_IMPLEMENTATION_DEFINED
AHARDWAREBUFFER_FORMAT_YCbCr_422_SP
AHARDWAREBUFFER_FORMAT_YCrCb_420_SP
AHARDWAREBUFFER_FORMAT_YCbCr_422_I
AHARDWAREBUFFER_FORMAT_YCbCr_P010

A particular example of a format used in a title is returned from AHardwareBuffer_describe as format 0x130. That's not any of the list above, so we can't create it and populate it even though we don't know and don't need to know the specifics of the format. This limitation exists at least through Android 13 but may have been removed in Android 14 as of https://android.googlesource.com/platform/frameworks/native/+/d09232c05f57fafabe71c2797d797699e9a67266.

One possible workaround for this issue is to use a commandbuffer to convert the non-standard format to one of the standard formats in capture and then dump that to the file.

bradgrantham-lunarg commented 7 months ago

Additional (sanitized) context from an internal email thread:

The game uses the buffer in order to render a video in the background of the starting menus. When replaying we fail to render that video completely. Instead we draw a green background while the menus are correct. However the actual game records and replays without problems (as long as the hacky patch is in place where it treats the format as the standard yuv ahb format so that we don't crash at the beginning). All of the [vendor's] formats appear to be yuv variations, so it would make sense if all these formats are used to render videos or feed input from other sources like the phone's camera.

Even if we manage to discover the secret handshake it takes to create buffers with these formats, we will still fail to render the video correctly as we currently only attempt to copy the content through the host and in this case the buffers are not host visible. However, I'm not sure if attempting to actually copy/convert the buffer is going to work either as I'm not sure how the game updates the buffers' content (through vulkan commands or in some other way)

So in this case even if we could create a buffer, it's not clear we can read the data either.

LunarG / gfxreconstruct