search

google / amber

Amber is a multi-API shader test framework
Apache License 2.0
198 stars 65 forks source link
dawn testing vulkan vulkan-compute-shaders

readme

Amber

Amber is a multi-API shader test framework.

Amber lets you capture and communicate shader bugs with the fluidity and ease of a scripting flow:

Amber is influenced by Talvos and VkRunner. The VkScript syntax matches the format used by VkRunner.

This is not an officially supported Google product.

Writing Amber Tests

Working with Amber involves writing input test files. Some example files can be see in the tests/cases folder.

The main input format is Amberscript. New features are added to AmberScript as Amber is enhanced. This is the preferred format in which new script files are written.

Clear test as AmberScript

SHADER vertex vtex_shader PASSTHROUGH
SHADER fragment frag_shader GLSL
#version 430

layout(location = 0) in vec4 color_in;
layout(location = 0) out vec4 color_out;

void main() {
  color_out = color_in;
}
END

BUFFER img_buf FORMAT B8G8R8A8_UNORM

PIPELINE graphics my_pipeline
  ATTACH vtex_shader
  ATTACH frag_shader

  FRAMEBUFFER_SIZE 256 256
  BIND BUFFER img_buf AS color LOCATION 0
END

CLEAR my_pipeline
EXPECT img_buf IDX 0 0 SIZE 256 256 EQ_RGBA 0 0 0 0

The VkScript format is supported for historic reasons. It is based off, and very closely matches, the format accepted by VkRunner. There are no new features being added to VkScript, it is for historical use.

Clear test as VkScript

[require]
VK_KHR_get_physical_device_properties2

[vertex shader passthrough]

[fragment shader]
#version 430

layout(location = 0) in vec4 color_in;
layout(location = 0) out vec4 color_out;

void main() {
  color_out = color_in;
}

[test]
clear
relative probe rect rgba (0.0, 0.0, 1.0, 1.0) (0, 0, 0, 0)

Requirements

Building

git clone https://github.com/google/amber.git
cd amber
./tools/git-sync-deps
mkdir -p out/Debug
cd out/Debug
cmake -GNinja ../..
ninja

Alternatives:

Android

Android plain executable

It is possible to obtain a plain executable for Android, as opposed to an APK, with the following:

git clone https://github.com/google/amber.git
cd amber
./tools/git-sync-deps

./tools/update_build_version.py . samples/ third_party/
./tools/update_vk_wrappers.py . .

mkdir build
cd build
mkdir app
mkdir libs
${ANDROID_NDK_HOME}/ndk-build -C ../samples NDK_PROJECT_PATH=. NDK_LIBS_OUT=`pwd`/libs NDK_APP_OUT=`pwd`/app

The list of target ABIs can be configured in samples/jni/Application.mk by editing the APP_ABI entry:

APP_ABI := arm64-v8a armeabi-v7a x86 x86_64

The resulting executable will be produced as build/app/local/<abi>/amber_ndk. This executable can be run via the adb shell on your device, e.g. under /data/local/tmp (/sdcard is generally not suitable because it is mounted with a non-executable flag). Also, vulkan layers may not be available to this executable as it is not an app, so make sure to use the -d flag to disable Vulkan layers:

adb push build/app/local/<abi>/amber_ndk /data/local/tmp
adb shell
# Now on device shell
cd /data/local/tmp
./amber_ndk -d <shader-test-files>

ChromeOS plain executable (not officially supported)

It is possible to obtain produce a cross compiled amber binary for ChromeOS. Start with the standard amber checkout

git clone https://github.com/google/amber.git
cd amber
./tools/git-sync-deps
./tools/update_build_version.py . samples/ third_party/
./tools/update_vk_wrappers.py . .

Then add the cmake cross compiling variable to the root amber /CMakeLists.txt

The example ChromeOS platform used here is trogdor.

# Top of CMakeLists.txt file 
cmake_minimum_required(VERSION 3.13)

set(CMAKE_SYSTEM_NAME Linux)
# The example processor here is 64 bit arm
set(CMAKE_SYSTEM_PROCESSOR aarch64)

set(CMAKE_SYSROOT "$ENV{HOME}/chromium/src/build/cros_cache/chrome-sdk/symlinks/trogdor+16074.0.0-1064250+sysroot_chromeos-base_chromeos-chrome.tar.xz/")

set(tools "$ENV{HOME}/chromium/src/build/cros_cache/chrome-sdk/symlinks/trogdor+16074.0.0-1064250+target_toolchain")
set(CMAKE_C_COMPILER ${tools}/bin/aarch64-cros-linux-gnu-gcc)
set(CMAKE_CXX_COMPILER ${tools}/bin/aarch64-cros-linux-gnu-g++)

set(CMAKE_FIND_ROOT_PATH_MODE_PROGRAM NEVER)
set(CMAKE_FIND_ROOT_PATH_MODE_LIBRARY ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_INCLUDE ONLY)
set(CMAKE_FIND_ROOT_PATH_MODE_PACKAGE ONLY)
# To avoid having the host try to execute a cross compiled binary 'asm_offset'
set(USE_GAS OFF)

We statically link c++ to avoid missing libstdc++.so.X

Replace these lines

  if (NOT ${AMBER_ENABLE_SHARED_CRT})
    # For MinGW cross compile, statically link to the C++ runtime.
    # But it still depends on MSVCRT.dll.
    if (${CMAKE_SYSTEM_NAME} STREQUAL "Windows")
      if (${CMAKE_CXX_COMPILER_ID} STREQUAL "GNU")
    set_target_properties(${TARGET} PROPERTIES LINK_FLAGS
      -static
      -static-libgcc
      -static-libstdc++)
      endif()
    endif()
  endif()

With this line.

    set_target_properties(${TARGET} PROPERTIES LINK_FLAGS -static-libstdc++)

Finally we can build and deploy the amber binary.

mkdir -p out/Debug
cd out/Debug
cmake -GNinja ../..  -DAMBER_USE_LOCAL_VULKAN=1  
ninja amber
# Copy over the amber binary to the DUT (note the root disk partion has limited space)
scp amber device:/root/amber
# An example of how to copy over some amber scripts
scp ../../tests/cases/* device:/root/

Now by ssh-ing into the DUT you can locally run any amber script. Also, vulkan layers may not be available to this executable, so make sure to use the -d flag to disable Vulkan layers.

Optional components

The components which build up Amber can be enabled or disabled as needed. Any option with _SKIP_ in the name is on by default, any with _USE_ is off by default.

The available flags which can be defined are:

cmake -DAMBER_SKIP_TESTS=True -DAMBER_SKIP_SPIRV_TOOLS=True -GNinja ../..

DXC

DXC can be enabled in Amber by adding the -DAMBER_USE_DXC=true flag when running cmake.

Build Bots

There are a number of build bots to verify Amber continues to compile and run on the various targets. Due to bot limitations, the integration tests are not being run on the bots, just the unit tests.

Backends

Amber is designed to run against different graphics APIs. Amber will build if no graphics API is found, but will only allow verifying the syntax of the amber script files.

Currently the Vulkan and Dawn graphics APIs are supported.

Using Vulkan as a backend

A Vulkan implementation is found by CMake in the following priority order:

Using Dawn as a backend

We assume you have built Dawn from source, and have access to both the source and build trees. To build a Dawn backend for Amber, set the following CMake variables when configuring Amber:

Using SwiftShader as a backend

SwiftShader, if available, can be used by by exporting the VK_ICD_FILENAMES environment variable and using it directly. If SwiftShader is not installed it can be built with Amber by setting AMBER_ENABLE_SWIFTSHADER during the configure step of CMake.

mkdir out/sw
cd out/sw
cmake -GNinja -DAMBER_ENABLE_SWIFTSHADER=TRUE ../..
ninja
export VK_ICD_FILENAMES=$PWD/Linux/vk_swiftshader_icd.json
./amber -d -V    # Should see SwiftShader listed as device
./amber -d ../../tests/cases/clear.amber

Amber Samples

The build will generate an out/Debug/amber executable which can be used to run amber scripts. The script can be used as out/Debug/amber <path to amber file>.

out/Debug/amber tests/cases/clear.amber

The sample app returns a value of 0 on success or non-zero on error. Any issues encountered should be displayed on the console.

Run out/Debug/amber -h to see a description of the program's command line options.

Example AmberScript files can be found in the tests/cases directory in this repository. Also the Vulkan Conformance Test Suite contains many real-world examples in its external/vulkancts/data/vulkan/amber subdirectory.

By default, out/Debug/amber supports saving the output image into '.png' file. You can disable this by passing -DAMBER_SKIP_LODEPNG=true to cmake.

The image_diff program will also be created. This allows comparing two images using the Amber buffer comparison methods.

Contributing

Please see the CONTRIBUTING and CODE_OF_CONDUCT files on how to contribute to Amber.

References