Did I input the wrong argument?

[OkazakiNaoki]

I saw your reply in DerPopo/UABE which is talking about how to extract shader of Unity. I just run the unity_asset_extractor.py with the asset file I want to see and then show up error. File "unity_asset_extractor.py", line 633, in <module> main() File "unity_asset_extractor.py", line 628, in main analyse(open(file, 'rb')) File "unity_asset_extractor.py", line 592, in analyse assert(os.fstat(file.fileno()).st_size == file_len) AssertionError

Feed asset file to unity_asset_extractor.py and get shader.decompressed right? Then feed shader.decompressed to extract_unity55_shaders.py. Did I misunderstand anything?

Also try unity_asset_bundle_extractor.py, still get the error. File "unity_asset_bundle_extractor.py", line 228, in <module> main() File "unity_asset_bundle_extractor.py", line 223, in main analyse(open(file, 'rb')) File "unity_asset_bundle_extractor.py", line 176, in analyse assert(data_header.read(16) == b'\0' * 16) AttributeError: 'bytes' object has no attribute 'read'

Yup... by the way it's 5.6 version asset.

[DarkStarSword]

The asserts are there to catch variations I haven't seen before that may not be being parsed correctly - usually that means a new version, but I added support for 5.6 through 2017.3 last week, so that should have worked. Can you attach an example of an asset file this is failing on?

General usage is fairly simple - run unity_asset_extractor.py on any of the *_Data/*.assets or *_Data/Resources/* files and it will extract all shader assets (type 48) under the 'extracted' directory (if you need other types you can hook up an extractor in the script - extract_raw will work with any type since it doesn't make any attempt to interpret the contained data).

Feed those files into extract_unity55_shaders.py (or previous versions depending on which Unity version you are working with) and it will extract the actual shaders under ShaderCRCs (d3d9), ShaderFNVs (d3d11) and ShaderGL - they will be in directories with the shader name from Unity, but the filename is based on a hash matching that used by Helix Mod (d3d9), 3DMigoto (d3d11) or the OpenGL 3D Vision Wrapper.

For d3d11 shaders you will need the latest cmd_Decompiler from the 3DMigoto project to disassemble and decompile them (the MS disassembler corrupts floating point values in shaders, and there is no other HLSL decompiler) - this should be placed in the 3d-fixes directory before running extract_unity55_shaders.py. If you don't have this it will still extract the shader binaries and place the metadata in a text file next to it, but will not transform the shaders into human readable form.

Some of the metadata from Unity will be attached to the extracted shaders as comments - of particular interest for our work in 3D Vision modding is the resource bind info and constant buffer descriptions. In cases where multiple distinct shaders were binary identical, the metadata from all matching shaders found in a single run of extract_unity*_shaders will be combined together (our use case is locating the metadata for a given shader binary, so in cases where they are binary identical we cannot determine which one it was, so we want the metadata from all possible matches). The new format introduced in Unity 5.5+ has a lot more metadata that is not yet included (pass -vv to the script to see this), but could be if there was a need for it.

Since you noted unity_asset_bundle_extractor.py - that's specifically for asset bundle files (and only the most recent UNITYFS versions of these), which are fairly rare in practice, but sometimes used for things like DLC. You would use it on files typically (though not always) found under *_Data/SteamingAssets/Bundles/*, and it works exactly the same as unity_asset_extractor.py (in fact, it will call into unity_asset_extractor if it finds any asset files inside the bundle).

For reference, here's an example of using these scripts to extract some shaders from Subnautica, which is Unity 5.6:

$ ~/3d-fixes/get_unity_version.sh Subnautica
5.6.2.0         Subnautica/Subnautica

$ ~/3d-fixes/unity_asset_extractor.py Subnautica_Data/sharedassets3.assets
Analysing Subnautica_Data/sharedassets3.assets...
Header len (?): 82060 (0x0001408c)
File Version: 17
Data offset: 0x140a0 (82080)
Unity version: 5.6.2p4UnknownWorlds
Type table length: 132
Unknown value: 0x13
Has data structure descriptions embedded in type table: 0
     ...
     4:  48 -1 a70f7abc 6586fb35 b3a0641a a81e9375
     ...
Num resources: 3789
   Resource 1: offset: 0x00000000, size:  10248, b'', type: 150
   Resource 2: offset: 0x00002808, size:    148, b'dev_measuregeneric01grey80', type: 21
   Resource 3: offset: 0x000028a0, size:    280, b'FindSub', type: 21
   Resource 4: offset: 0x000029b8, size:    744, b'Sand02', type: 21
   Resource 5: offset: 0x00002ca0, size:    564, b'EndSequenceWarpScreenFX', type: 21
   ...
   Resource 514: offset: 0x00677910, size:  11592, b'', type: 48
Dumping 'extracted/sharedassets3/0x0068b9b0.a70f7abc6586fb35b3a0641aa81e9375.shader.raw'...
   Resource 515: offset: 0x0067a658, size:   4532, b'', type: 48
Dumping 'extracted/sharedassets3/0x0068e6f8.a70f7abc6586fb35b3a0641aa81e9375.shader.raw'...
   Resource 516: offset: 0x0067b810, size:   3912, b'', type: 48
Dumping 'extracted/sharedassets3/0x0068f8b0.a70f7abc6586fb35b3a0641aa81e9375.shader.raw'...
   ...

$ cd extracted/sharedassets3
$ ~/3d-fixes/extract_unity55_shaders.py 0x0068b9b0.a70f7abc6586fb35b3a0641aa81e9375.shader.raw
Processing 0x0068b9b0.a70f7abc6586fb35b3a0641aa81e9375.shader.raw...
 Name: UWE/Particles/WBOIT ScreenDrops
 Num Platforms       : 2
  Platform d3d11 (4):
   Num shaders         : 2
  Platform glcore (15):
   Num shaders         : 2
Extracting ./ShaderGL/UWE_Particles_WBOIT ScreenDrops/fp/Pixel_5bf3a23c.glsl...
Extracting ./ShaderGL/UWE_Particles_WBOIT ScreenDrops/vp/Vertex_c824565c.glsl...

Extracting ./ShaderFNVs/UWE_Particles_WBOIT ScreenDrops/7ee4cd19d8e0a77a-ps.bin
Disassembling (Flugan) 7ee4cd19d8e0a77a-ps.bin...
  -> 7ee4cd19d8e0a77a-ps.asm
Decompiling 7ee4cd19d8e0a77a-ps.bin...
    creating HLSL representation
  -> 7ee4cd19d8e0a77a-ps.hlsl
Extracting ./ShaderFNVs/UWE_Particles_WBOIT ScreenDrops/9a9407e0c21e8169-vs.bin
Disassembling (Flugan) 9a9407e0c21e8169-vs.bin...
  -> 9a9407e0c21e8169-vs.asm
Decompiling 9a9407e0c21e8169-vs.bin...
    creating HLSL representation
  -> 9a9407e0c21e8169-vs.hlsl

$ cd ShaderFNVs/UWE_Particles_WBOIT\ ScreenDrops/
$ cat 7ee4cd19d8e0a77a-ps.txt
// 3DMigoto: 7ee4cd19d8e0a77a |    Unity headers extracted from UWE_Particles_WBOIT ScreenDrops
//    Shader "UWE/Particles/WBOIT ScreenDrops" {
//      SubShader 1/1 {
//        Tags { "IGNOREPROJECTOR"="true" "QUEUE"="Transparent+101" "RenderType"="Transparent" }
//        Pass 1/1 {
//          Tags { "IGNOREPROJECTOR"="true" "QUEUE"="Transparent+101" "RenderType"="Transparent" }
//          ZWrite Off
//          Program "fp" {
//            SubProgram "d3d11" {
//            }
//          }
//        }
//      }
//    }
//
// Unity 5.3 headers extracted from UWE_Particles_WBOIT ScreenDrops:
//   API d3d11
//   Shader model ps_4_0
//   undeciphered1: 201609010 122 1 7
//   undeciphered1a: 10
//   undeciphered2: 1 4 1 4 0 0
//   undeciphered3: 0 0
//   ConstBuffer "$Globals" 464
//   Float 32 [_UweFogEnabled]
//   Float 36 [_UweAboveWaterFogStartDistance]
//   Vector 48 [_UweFogVsLightDirection] 3
//   Vector 80 [_UweFogLightColor]
...
//   Float 448 [_WBOIT_WeightToggle]
//   Float 452 [_WBOIT_WeightSharpness]
//   Matrix 144 [_UweCameraToVolumeMatrix]
//   SetTexture 0 [_MainTex] 2D 3
//   SetTexture 1 [_UweExtinctionTexture] 2D 0
//   SetTexture 2 [_UweScatteringTexture] 2D 1
//   SetTexture 3 [_UweEmissiveTexture] 2D 2
//   BindCB "$Globals" 0
//
// Headers extracted with DarkStarSword's extract_unity55_shaders.py
// https://github.com/DarkStarSword/3d-fixes

//
// Generated by Microsoft (R) D3D Shader Disassembler
//
//   using 3Dmigoto v1.2.71 on Sun Jan 28 16:41:05 2018
//
//
// Input signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_POSITION              0   xyzw        0      POS   float       
// COLOR                    0   xyzw        1     NONE   float   xyzw
// TEXCOORD                 0   xyzw        2     NONE   float   xy  
// TEXCOORD                 1   xyzw        3     NONE   float   xyz 
// TEXCOORD                 2   xyzw        4     NONE   float       
// TEXCOORD                 3   xyzw        5     NONE   float    y  
//
//
// Output signature:
//
// Name                 Index   Mask Register SysValue  Format   Used
// -------------------- ----- ------ -------- -------- ------- ------
// SV_Target                0   xyzw        0   TARGET   float   xyzw
// SV_Target                1   xyzw        1   TARGET   float   xyzw
// SV_Target                2   xyzw        2   TARGET   float   xyzw
//
ps_4_0
dcl_constantbuffer cb0[29], immediateIndexed
dcl_sampler s0, mode_default
dcl_sampler s1, mode_default
dcl_sampler s2, mode_default
dcl_sampler s3, mode_default
dcl_resource_texture2d (float,float,float,float) t0
dcl_resource_texture2d (float,float,float,float) t1
dcl_resource_texture2d (float,float,float,float) t2
dcl_resource_texture2d (float,float,float,float) t3
dcl_input_ps linear v1.xyzw
dcl_input_ps linear v2.xy
dcl_input_ps linear v3.xyz
dcl_input_ps linear v5.y
dcl_output o0.xyzw
dcl_output o1.xyzw
dcl_output o2.xyzw
dcl_temps 10
add r0.x, v5.y, -cb0[27].y
add r0.x, r0.x, -cb0[27].z
add r0.x, r0.x, cb0[27].x
div r0.x, r0.x, cb0[27].x
add_sat r0.x, -r0.x, v1.w
mul r0.w, r0.x, v1.w
add r1.xyzw, cb0[22].xyzw, cb0[22].xyzw
...

And yeah - where it says "unity 5.3 headers extracted..." is correct - Unity 5.3 introduced a new binary header format that is still present in Unity 5.5+ shaders, even though it is largely redundant given the 5.5+ headers say the same thing. Earlier versions of Unity just used a plaintext format for the metadata that is included instead.

[OkazakiNaoki]

Ok. I did it correctly now. I just want to see if shader is still readable after decompile/disassemble. So...it is basically no way to get shader source. Thank you.

[DarkStarSword]

If it's an OpenGL shader you will get something close - not the exact Cg source from Unity, but a translation of that to GLSL, which is pretty close.

But DirectX shaders are compiled to bytecode, so will be much the same as disassembling/decompiling a C program - it's still possible to examine them and work out what they are doing, but much harder than reading source.

If you're looking for the source for any standard Unity shaders, don't forget that they are available from the Unity download page under "Older versions of Unity" -> "Downloads (Win)" -> "Built in Shaders".

[OkazakiNaoki]

Umm...then do you have GLSL shader asset sample? (Or I just need to change Unity build option and it will come out with GLSL version? I mean do some experiment myself.) I would like to see how close it is.

The reason why I curious is that I saw a reply at Unity forum which said : "As nvidia is not supporting CG anymore, latest unity versions actually compile shaders using HLSL compiler and transform the resulting bytecode to GLSL."

Then does it mean that the compile order is Cg->HLSL bytecode->GLSL ? How come GLSL code is more readable than HLSL since it's transformed from HLSL bytecode. Or that reply is old information and no longer correct?

[DarkStarSword]

I haven't looked at an OpenGL shader from Unity in ages, but based on this, I'd say you're probably right - although this is clearly translated from bytecode before the reflection information has been stripped. This is from Hidden/Internal-DeferredReflections:

#ifdef VERTEX
#version 150
#extension GL_ARB_explicit_attrib_location : require
#extension GL_ARB_shader_bit_encoding : enable

uniform     vec4 _ProjectionParams;
uniform     vec4 hlslcc_mtx4x4unity_ObjectToWorld[4];
uniform     vec4 hlslcc_mtx4x4unity_MatrixV[4];
uniform     vec4 hlslcc_mtx4x4unity_MatrixVP[4];
uniform     float _LightAsQuad;
in  vec4 in_POSITION0;
in  vec3 in_NORMAL0;
out vec4 vs_TEXCOORD0;
out vec3 vs_TEXCOORD1;
vec4 u_xlat0;
vec4 u_xlat1;
vec4 u_xlat2;
void main()
{
    u_xlat0 = in_POSITION0.yyyy * hlslcc_mtx4x4unity_ObjectToWorld[1];
    u_xlat0 = hlslcc_mtx4x4unity_ObjectToWorld[0] * in_POSITION0.xxxx + u_xlat0;
    u_xlat0 = hlslcc_mtx4x4unity_ObjectToWorld[2] * in_POSITION0.zzzz + u_xlat0;
    u_xlat0 = u_xlat0 + hlslcc_mtx4x4unity_ObjectToWorld[3];
    u_xlat1 = u_xlat0.yyyy * hlslcc_mtx4x4unity_MatrixVP[1];
    u_xlat1 = hlslcc_mtx4x4unity_MatrixVP[0] * u_xlat0.xxxx + u_xlat1;
    u_xlat1 = hlslcc_mtx4x4unity_MatrixVP[2] * u_xlat0.zzzz + u_xlat1;
    u_xlat1 = hlslcc_mtx4x4unity_MatrixVP[3] * u_xlat0.wwww + u_xlat1;
    gl_Position = u_xlat1;
    u_xlat1.y = u_xlat1.y * _ProjectionParams.x;
    u_xlat2.xzw = u_xlat1.xwy * vec3(0.5, 0.5, 0.5);
    vs_TEXCOORD0.zw = u_xlat1.zw;
    vs_TEXCOORD0.xy = u_xlat2.zz + u_xlat2.xw;
    u_xlat1.xyz = u_xlat0.yyy * hlslcc_mtx4x4unity_MatrixV[1].xyz;
    u_xlat1.xyz = hlslcc_mtx4x4unity_MatrixV[0].xyz * u_xlat0.xxx + u_xlat1.xyz;
    u_xlat0.xyz = hlslcc_mtx4x4unity_MatrixV[2].xyz * u_xlat0.zzz + u_xlat1.xyz;
    u_xlat0.xyz = hlslcc_mtx4x4unity_MatrixV[3].xyz * u_xlat0.www + u_xlat0.xyz;
    u_xlat1.xyz = u_xlat0.xyz * vec3(-1.0, -1.0, 1.0);
    u_xlat0.xyz = (-u_xlat0.xyz) * vec3(-1.0, -1.0, 1.0) + in_NORMAL0.xyz;
    vs_TEXCOORD1.xyz = vec3(_LightAsQuad) * u_xlat0.xyz + u_xlat1.xyz;
    return;
}

#endif
#ifdef FRAGMENT
#version 150
#extension GL_ARB_explicit_attrib_location : require
#extension GL_ARB_shader_bit_encoding : enable

uniform     vec3 _WorldSpaceCameraPos;
uniform     vec4 _ProjectionParams;
uniform     vec4 _ZBufferParams;
uniform     vec4 hlslcc_mtx4x4unity_CameraToWorld[4];
uniform     vec4 unity_SpecCube0_BoxMax;
uniform     vec4 unity_SpecCube0_BoxMin;
...

That looks pretty similar to 3DMigoto's decompiler output from when reflection information is present - you can see that the matrix mul() has been broken up into individual multiplications and additions for example.

I definitely remember the GLSL in older versions of Unity was much closer to the Cg source code, but our focus has mostly been on the DX shaders so I haven't paid a huge amount of attention to when that changed.

[OkazakiNaoki]

I tried 2017 version today and try to make some example assets file to extract OpenGL shader code. And then I get pretty much the same result you paste. What I don't understand is that why so many output? pixel and vertex 36 each?

Most important...I don't see any logic in those codes. I guess Unity wins?

[DarkStarSword]

What I don't understand is that why so many output? pixel and vertex 36 each?

The pre-processor directives in the Cg source code lead to many permutations of different shaders - The "Keywords" attribute in the headers tells you which permutation of pre-processor defines were active for that specific shader. 36 is nothing - it is not uncommon for a single Unity shader to explode into hundreds of actual shaders, and in some games the "Standard" and "Hidden/Post FX/Uber Shader" explode into the thousands.

For the OpenGL shaders you should note that the fp directory is actually synthesized by the extraction tools - it contains the same shaders as the vp directory, but with #define FRAGMENT at the top instead of #define VERTEX, and the hashes in the filenames will be different as a result.

Most important...I don't see any logic in those codes.

It is there - I analyse these all the time, but it is a lot harder to follow than just reading Cg - you really have to be familiar with assembly to be able to decipher these (e.g. you should be able to easily spot three row-major matrix multiplications in the shader I pasted above), and best if you can go in and experiment with changing them to observe what happens:

• For DX11 shaders use 3DMigoto for live experiments, removing the nvapi.dll if you are using AMD.
• OpenGL shaders might still work with Helifax' OpenGL 3D Vision Wrapper, but I don't have much experience with it, no idea if it will work if you don't have 3D Vision and if there have been any changes to how Unity munges the shaders in the last couple of years there is a possibility the hashes may no longer match - OpenGL was only ever added as an experiment for these scripts, though Helifax was able get a Unity game running in OpenGL working with 3D Vision, so it was a successful experiment.
• For DX9 shaders use Helix Mod, though I would not recommend this.