search

umautobots / GTAVisionExport

Code to export full segmentations from GTA
MIT License
126 stars 42 forks source link

Frame captured with no depth and stencil #12

Closed JiamingSuen closed 7 years ago

JiamingSuen commented 7 years ago

Hi there, I managed to compile the project and get it working(well partially) in the game. Frame capture is working fine but there's no depth and stencil in the result tiff image. The depth and stencil layer in tiff are just zeros and 255, while color frames are perfectly fine. GTAVisionNative.asi plugin is working normally with proper size raw file captured with 'L' key. I put the native asi file in GTA root folder and other compile output from 'managed' project in 'scripts' folder. I suspect the problem is caused by some linking problem with 'native' and 'managed', but the build process didn't generate any error. Does GTAVisionUtils.dll include GTAVisionNative.asi already? or these files should be put in some other order?

Everything seems to be well in log files, this is content in ScriptHookVDotNet2.log:

[17:23:24] [DEBUG] Created script domain 'ScriptDomain_3BFA16BC' with v2.10.3.
[17:23:24] [DEBUG] Loading scripts from 'C:\Program Files (x86)\Steam\steamapps\common\Grand Theft Auto V\scripts' into script domain 'ScriptDomain_3BFA16BC' ...
[17:23:24] [INFO] Loading assembly 'AWSSDK.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'AWSSDK.dll'.
[17:23:25] [INFO] Loading assembly 'BitMiracle.LibTiff.NET.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'BitMiracle.LibTiff.NET.dll'.
[17:23:25] [INFO] Loading assembly 'gdal_csharp.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'gdal_csharp.dll'.
[17:23:25] [INFO] Loading assembly 'GTAVisionExport.dll' ...
[17:23:25] [DEBUG] Found 1 script(s) in 'GTAVisionExport.dll'.
[17:23:25] [INFO] Loading assembly 'GTAVisionUtils.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'GTAVisionUtils.dll'.
[17:23:25] [INFO] Loading assembly 'INIFileParser.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'INIFileParser.dll'.
[17:23:25] [INFO] Loading assembly 'MathNet.Numerics.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'MathNet.Numerics.dll'.
[17:23:25] [INFO] Loading assembly 'Microsoft.Extensions.DependencyInjection.Abstractions.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.DependencyInjection.Abstractions.dll'.
[17:23:25] [INFO] Loading assembly 'Microsoft.Extensions.Logging.Abstractions.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.Logging.Abstractions.dll'.
[17:23:25] [INFO] Loading assembly 'Microsoft.Extensions.Logging.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.Logging.dll'.
[17:23:25] [INFO] Loading assembly 'NativeUI.dll' ...
[17:23:25] [DEBUG] Found 1 script(s) in 'NativeUI.dll'.
[17:23:25] [INFO] Loading assembly 'Npgsql.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'Npgsql.dll'.
[17:23:25] [INFO] Loading assembly 'ogr_csharp.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'ogr_csharp.dll'.
[17:23:25] [INFO] Loading assembly 'osr_csharp.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'osr_csharp.dll'.
[17:23:25] [INFO] Loading assembly 'SharpDX.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'SharpDX.dll'.
[17:23:25] [INFO] Loading assembly 'SharpDX.Mathematics.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'SharpDX.Mathematics.dll'.
[17:23:25] [INFO] Loading assembly 'System.Runtime.InteropServices.RuntimeInformation.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'System.Runtime.InteropServices.RuntimeInformation.dll'.
[17:23:25] [INFO] Loading assembly 'System.Threading.Tasks.Extensions.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'System.Threading.Tasks.Extensions.dll'.
[17:23:25] [INFO] Loading assembly 'VAutodrive.dll' ...
[17:23:25] [DEBUG] Found 6 script(s) in 'VAutodrive.dll'.
[17:23:25] [INFO] Loading assembly 'VCommonFunctions.dll' ...
[17:23:25] [DEBUG] Found 2 script(s) in 'VCommonFunctions.dll'.
[17:23:25] [INFO] Loading assembly 'YamlDotNet.dll' ...
[17:23:25] [DEBUG] Found 0 script(s) in 'YamlDotNet.dll'.
[17:23:25] [DEBUG] Starting 10 script(s) ...
[17:23:25] [DEBUG] Instantiating script 'GTAVisionExport.VisionExport' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'GTAVisionExport.VisionExport'.
[17:23:26] [DEBUG] Instantiating script 'NativeUI.BigMessageThread' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'NativeUI.BigMessageThread'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.VehicleMoversHandling' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.VehicleMoversHandling'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.MenuHandling' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.MenuHandling'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.KeyHandling' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.KeyHandling'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.Autopilot' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.Autopilot'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.SeatHandling' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.SeatHandling'.
[17:23:26] [DEBUG] Instantiating script 'VAutodrive.WalkHandling' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'VAutodrive.WalkHandling'.
[17:23:26] [DEBUG] Instantiating script 'Cyron43.GtaV.Common.NotificationTask' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'Cyron43.GtaV.Common.NotificationTask'.
[17:23:26] [DEBUG] Instantiating script 'Cyron43.GtaV.Common.PerFrameFunctions' in script domain 'ScriptDomain_3BFA16BC' ...
[17:23:26] [DEBUG] Started script 'Cyron43.GtaV.Common.PerFrameFunctions'.

I would like to provide other log files if needed. Thanks!

muaz-git commented 7 years ago

I think it should work fine. If you look in to this function, it takes byte arrays for color, depth and stencil buffers and save them in one tiff file. For debugging you can try modifying this function and save depth and stencil buffers in separate files. Also note that you can't interpret/use these buffers as they are, you need to transform them as pointed in the section (III B) Technical Approach -Internal Engine Buffers of paper.

JiamingSuen commented 7 years ago

I managed to capture the depth buffer by turning MSAA off in the graphics settings, but the stencil page of tiff file is still all black. The captured depth looks like this, with bottom left corner all zero for unknown reason(it's width is not equal to the map). image This is the captured color: image

@muaz-git What do you mean by "transform"? Do you mean the depth buffer is stored as the logarithm of the actual depth? I'm not sure if the linearization has already been done by the native plugin. I would like to post the captured tiff file if needed. @muaz-git What's your process on this project, did you successfully captured all the data? It would be nice if you can share your configuration/dll/environment etc.. Thanks for all your help!

I tried tweak with other graphics settings but nothing helped with capturing stencil. I'll try to dig more.

muaz-git commented 7 years ago

@JiamingSuen Try turning off HUD for map. I guess no linearization has been done in native part. Follow this tutorial to linearize the depth, I haven't tried it yet but I guess it should work. I looked in to the code the other day and it seems like this part of code is saving projection matrix from where you can fetch z_near and z_far (which you need to linearize the depth map.) In order to correctly interpret stencil map, you need to read only bottom 4 bits of each byte (pixel) to fetch each object ID (Note: These are only object IDs and you may need use pseudo coloring to visualize them). I am still working on the pipeline of completely automatic annotation. I will share all my stuff once I am done.

JiamingSuen commented 7 years ago

By turning off HUD and Radar, stencil is successfully captured. Thanks a lot @muaz-git !! I'll be continue working on depth linearization, looking forward to your result on semantic tag! I think we can work together to publish a complete guide for this project after the post processing, which is really needed for others.

tyzaizl commented 7 years ago

@JiamingSuen Hi, how did you visualize the "color.raw" and "depth.raw" images in matlab? I got the raw files but failed to convert to normal format. Thanks.

JiamingSuen commented 7 years ago

@tyzaizl The native plugin is not supposed to be used directly according to the paper, use the managed plugin instead. After compile and some tweaking you should get a tiff file with 5 color pages, 1 depth page, 1 stencil page. Use the following python code to read the tiff image:

from PIL import Image, ImageFile, TiffImagePlugin
import matplotlib.pyplot as plt
import numpy as np
ImageFile.LOAD_TRUNCATED_IMAGES = True
TiffImagePlugin.READ_LIBTIFF = True

im = Image.open('./info36.tiff') # with your own image path.
im.seek(1) # use seek() to switch between pages
tyzaizl commented 7 years ago

@JiamingSuen Thank you for replay, It seems I just got raw files by native plugin. When I came to managed plugin, something confused me:

  1. when I copied ScriptHookVDotNet2.dll and ScriptHookVDotNet.asi into GTAV directory, GTAV will be crashed. I don't konw why. I tried ScriptHookVDotNet v2.10.3, v2.10.2 and v2.10.1, and all failed. crash

  2. how to connect to sql dataset, in managed plugin README.md it said that,

"In order the connect to the database the managed plugins needs to know your database information. This is provided in an ini file that looks like the following: [Database] ConnectionString= "

what kind of ini file should be created and where should this file located? directly in GTAV directory?

  1. if press "N" can generate tiff files, where are these files located?

here are the log files: ScriptHookVDotNet2.log ScriptHookV.log GTANativePlugin.log

tyzaizl commented 7 years ago

It seems that can't update NuGet packages. This time I succeed to start GTAV, but when I pressed "N" or "n" nothing happened. here are the log files: ScriptHookVDotNet2.log [15:34:49] [DEBUG] Created script domain 'ScriptDomain_679CB73C' with v2.10.2. [15:34:49] [DEBUG] Loading scripts from 'D:\tools\Grand Theft Auto V\scripts' into script domain 'ScriptDomain_679CB73C' ... [15:34:49] [INFO] Loading assembly 'AWSSDK.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'AWSSDK.dll'. [15:34:50] [INFO] Loading assembly 'BitMiracle.LibTiff.NET.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'BitMiracle.LibTiff.NET.dll'. [15:34:50] [INFO] Loading assembly 'FreeImageNET.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'FreeImageNET.dll'. [15:34:50] [INFO] Loading assembly 'gdalconst_csharp.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'gdalconst_csharp.dll'. [15:34:50] [INFO] Loading assembly 'gdal_csharp.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'gdal_csharp.dll'. [15:34:50] [INFO] Loading assembly 'GTAVisionExport.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'GTAVisionExport.dll'. [15:34:50] [INFO] Loading assembly 'GTAVisionUtils.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'GTAVisionUtils.dll'. [15:34:50] [INFO] Loading assembly 'INIFileParser.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'INIFileParser.dll'. [15:34:50] [INFO] Loading assembly 'MathNet.Numerics.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'MathNet.Numerics.dll'. [15:34:50] [INFO] Loading assembly 'Microsoft.Extensions.DependencyInjection.Abstractions.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.DependencyInjection.Abstractions.dll'. [15:34:50] [INFO] Loading assembly 'Microsoft.Extensions.Logging.Abstractions.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.Logging.Abstractions.dll'. [15:34:50] [INFO] Loading assembly 'Microsoft.Extensions.Logging.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'Microsoft.Extensions.Logging.dll'. [15:34:50] [INFO] Loading assembly 'NativeUI.dll' ... [15:34:50] [DEBUG] Found 1 script(s) in 'NativeUI.dll'. [15:34:50] [INFO] Loading assembly 'Npgsql.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'Npgsql.dll'. [15:34:50] [INFO] Loading assembly 'ogr_csharp.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'ogr_csharp.dll'. [15:34:50] [INFO] Loading assembly 'osr_csharp.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'osr_csharp.dll'. [15:34:50] [WARNING] Skipped assembly 'ScriptHookVDotNet2.dll'. Please remove it from the directory. [15:34:50] [INFO] Loading assembly 'SharpDX.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'SharpDX.dll'. [15:34:50] [INFO] Loading assembly 'SharpDX.Mathematics.dll' ... [15:34:50] [DEBUG] Found 0 script(s) in 'SharpDX.Mathematics.dll'. [15:34:50] [INFO] Loading assembly 'System.Runtime.InteropServices.RuntimeInformation.dll' ... [15:34:51] [DEBUG] Found 0 script(s) in 'System.Runtime.InteropServices.RuntimeInformation.dll'. [15:34:51] [INFO] Loading assembly 'System.Threading.Tasks.Extensions.dll' ... [15:34:51] [DEBUG] Found 0 script(s) in 'System.Threading.Tasks.Extensions.dll'. [15:34:51] [INFO] Loading assembly 'VAutodrive.dll' ... [15:34:51] [DEBUG] Found 0 script(s) in 'VAutodrive.dll'. [15:34:51] [INFO] Loading assembly 'VCommonFunctions.dll' ... [15:34:51] [DEBUG] Found 0 script(s) in 'VCommonFunctions.dll'. [15:34:51] [INFO] Loading assembly 'YamlDotNet.dll' ... [15:34:51] [DEBUG] Found 0 script(s) in 'YamlDotNet.dll'. [15:34:51] [DEBUG] Starting 1 script(s) ... [15:34:51] [DEBUG] Instantiating script 'NativeUI.BigMessageThread' in script domain 'ScriptDomain_679CB73C' ... [15:34:51] [DEBUG] Started script 'NativeUI.BigMessageThread'.

ScriptHookV.log // GTA V SCRIPT HOOK (build Sep 3 2017, v1.0.1180.2) // (C) Alexander Blade 2015-2017 [15:33:50] INIT: Started [15:33:50] INIT: Success, game version is VER_1_0_1180_2_NOSTEAM [15:33:50] INIT: Registering script 'ScriptHookVDotNet.asi' (0x00007FF85DEE1380) [15:33:58] INIT: Pool 1 extended [15:33:58] INIT: Pool 2 extended [15:33:58] INIT: Pool 3 extended [15:33:58] INIT: Pool 4 extended [15:34:04] INIT: GtaThread collection size 189 [15:34:04] INIT: wnd proc 0x00000000FFFF0195 [15:34:04] INIT: IDXGISwapChain 0x00007FF860110AA0 (0x00007FF8600B3610) [15:34:04] INIT: IDXGISwapChain set [15:34:04] INIT: DX init started [15:34:04] INIT: DX feature level B000 [15:34:04] INIT: DX init succeeded [15:34:47] CORE: Requesting thread creation [15:34:47] CORE: Creating threads [15:34:49] CORE: Started control thread, id 31 active 1 [15:34:49] CORE: Started thread 'ScriptHookVDotNet.asi' (0x00007FF85DEE1380), id 32 active 1 [15:34:49] CORE: Launching main() for 'ScriptHookVDotNet.asi' (0x00007FF85DEE1380), id 32

JiamingSuen commented 7 years ago

@tyzaizl you should have a look at the code in VisionExport.cs and modify according to your needs.

JiamingSuen commented 7 years ago

Summary of my steps making the managed plugin work:

  1. If you met #6 (d3d11_4.h is missing), another way to resolve this is to install the latest Windows 10 SDK within the Visual Studio installer(to get DirectX SDK).
  2. Make sure the game is started with all resolution dimension larger than 1000 or GTA will crash during loading asi plugin.
  3. If you don't want to use database, uncomment "N" key to save tiff file directly to your disk. https://github.com/umautobots/GTAVisionExport/blob/master/managed/GTAVisionExport/VisionExport.cs#L579 Modify the paths in VisionExport.cs to assign your own file saving location.
  4. Turn off MSAA, HUD and Radar in GTA settings.
  5. You are supposed to be all set for capture. Make sure ScriptHookVDotNet2.log looks like the one I attached at the beginning, press PageUp to enable the plugin, use N key to start capture.
racinmat commented 7 years ago

@tyzaizl the ini file should be placed in the scripts directory.

TommyAnqi commented 7 years ago

@racinmat could you explain more details about the ini file? Thanks a lot

racinmat commented 7 years ago

@TommyAnqi I have pull request with update README information. I described there the ini file, you can see it here. https://github.com/racinmat/GTAVisionExport/blob/master/managed/README.md Hope it helps.

IanKirwan commented 6 years ago

@JiamingSuen Hiya, I'm getting garbled depth images out (example below). I have MSAA turned off, and for reflections too. Window resolution 1280x720, all of the graphics settings turned down to 'normal'. Any clues as to how I might resolve this? PS stencil looks like its garbled too.

45871

racinmat commented 6 years ago

@IanKirwan this is probably done using incorrect tiff reader. In this repo, tiff flags are heavily used, and some readers can'T deal with it. It's quite possible that it is correctly stored, but incorrectly visualized.

IanKirwan commented 6 years ago

Hi @racinmat, The image above is in the following zip file. If you have the tools could you check it? 45871.zip I used the following code to view it. It treats the depth map as a 1280x720x1 array and scales the values to between 0-255:

from PIL import Image, ImageFile, TiffImagePlugin
import numpy as np
import cv2
ImageFile.LOAD_TRUNCATED_IMAGES = True
TiffImagePlugin.READ_LIBTIFF = True

im = Image.open('D:/Snapshots/Output/45871.tiff') # with your own image path.
im.seek(1) # use seek() to switch between pages
image = np.array(im.getdata()).reshape(im.size[1], im.size[0], 1)
maximg = np.max(image)
imageint = (image/maximg)*255

print(im.size[0])
print(im.size[1])
cv2.imshow('img',imageint)
cv2.waitKey(0)

Do you know if it matters whether the graphics settings are borderless window? full screen? What GTAV graphics settings are you using? I'm starting to get a feeling that the native code is not capturing the depth correctly, at least with my build of it.

IanKirwan commented 6 years ago

If someone could zip up a working (for depth maps) 64bit GTAVisionNative.asi and GTAVisionNative.lib and post it here that would be a huge help. Many thanks in advance.

racinmat commented 6 years ago

@IanKirwan Hi, sorry for the long pause, I didn't have time earlier. I am using windowed mode, not borderless. All my in-game graphics settings I am using are described in the readme, here: https://github.com/umautobots/GTAVisionExport/tree/master/managed#in-game-settings

I think it matters if you are using fullscrenn, borderless or wendowed, but I am not sure. When I am using windows mode, everything works for me fine.

I tried to visualize your data, and it really seems messed up. I tried source code which works for my tiff files exported from GTA. But I had no success, depth and stencil seems to be weird for me too.

But as I too had issues with tiff files, what worked for me was saving data into multiple tiff files instead of one file. So I have 1 tiff file to RGB, 1 file for depth, and 1 file for stencil. Then I was able to load data correctly with all classic python tiff libraries.

Saving into multiple tiffs can be seen here, in my fork: https://github.com/racinmat/GTAVisionExport/blob/multiple_tiffs/managed/GTAVisionUtils/ImageUtils.cs

IanKirwan commented 6 years ago

Hi @racinmat No problem. We kind of got to the bottom of the issue, see issue 28. The problem seems to be that the depth buffer is actually much larger than the screen resolution for some machine setups. It gets garbled as the code tries to squash it into the tiff with smaller dimensions. I'm working on a mod to the native code to resample the buffer to fit when it is over sized. Many thanks for the details, though I had followed those instructions previously. My machine at work, works fine with borderless window at 1280x720 btw. I'm look forward to seeing your mods integrated into the master branch. Kind regards Ian

dominikandreas commented 6 years ago

During my experiments I even noticed the buffer sizes change during a single session without any changes to the graphics settings (changes in weather can cause this apparently). So it may be necessary to implement some dynamic handling of this. Shouldn't it be possible to resize it as a 2d texture while it's still residing on the GPU? This would probably be the easiest and fastest option (although my expertise is limited in that regard). Is anyone familiar enough with the directx APIs in order to implement something like this?

On Sat, Feb 10, 2018, 12:50 IanKirwan notifications@github.com wrote:

Hi @racinmat https://github.com/racinmat No problem. We kind of got to the bottom of the issue, see issue 28 https://github.com/umautobots/GTAVisionExport/issues/28. The problem seems to be that the depth buffer is actually much larger than the screen resolution for some machine setups. It gets garbled as the code tries to squash it into the tiff with smaller dimensions. I'm working on a mod to the native code to resample the buffer to fit when it is over sized. Many thanks for the details, though I had followed those instructions previously. My machine at work, works fine with borderless window at 1280x720 btw. I'm look forward to seeing your mods integrated into the master branch. Kind regards Ian

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/umautobots/GTAVisionExport/issues/12#issuecomment-364646231, or mute the thread https://github.com/notifications/unsubscribe-auth/AM5gMLMTWmI9Kv-TwOyyQu2rbIkcS1Tjks5tTYKagaJpZM4P728w .

IanKirwan commented 6 years ago

I'm trying code up a simple resample (down sample) that checks the screen size against the caputured depth buffer size and only does the resample if over sized. It doesnt work yet, as it crashes. It's a bit ugly, using mixed data types and your suggestion sounds like the smarter way to do it, though I have never done any Windows app programming before and my knowledge of c++ is from college 20 years ago. Still I might look into it.

Anyway, for now my buggy mod to export.cpp looks like this so far:

int screenResX;
int screenResY;
GRAPHICS::GET_SCREEN_RESOLUTION(&screenResX, &screenResY);
if (screenResX >= src_desc.Width)
{
    for (int x = 0; x < src_desc.Width; ++x)
    {
        for (int y = 0; y < src_desc.Height; ++y)
        {
            const float* src_f = (const float*)((const char*)src_map.pData + src_map.RowPitch*y + (x * 8));
            unsigned char* dst_p = &dst[src_desc.Width * 4 * y + (x * 4)];
            unsigned char* stencil_p = &stencil[src_desc.Width * y + x];
            memmove(dst_p, src_f, 4);
            memmove(stencil_p, src_f + 1, 1);
        }
    }
}
else
{
    // resize, for when depth map is bigger than screen image.
    float scale = ((float) src_desc.Width) / ((float) screenResX);
    dst = vector<unsigned char>(screenResY * screenResX * 4);
    for (int x = 0; x < screenResX; ++x)
    {
        for (int y = 0; y < screenResY; ++y)
        {
            int scaledX = int(x*scale);
            int scaledY = int(y*scale);
            const float* src_f = (const float*)((const char*)src_map.pData + src_map.RowPitch*scaledY + (scaledX * 8));
            unsigned char* dst_p = &dst[src_desc.Width * 4 * x + (y * 4)];
            unsigned char* stencil_p = &stencil[src_desc.Width * scaledY + scaledX];
            memmove(dst_p, src_f, 4);
            memmove(stencil_p, src_f + 1, 1);
        }
    }
}
IanKirwan commented 6 years ago

OK my (fixed) code mod appears to be working (and miraculously fixed stencil too) on this machine, both with standalone native and combined native/managed. It would be good if someone else could try this with their machine before I do a pull request. Here's the modified export.cpp unpack_depth() function:

static void unpack_depth(ID3D11Device* dev, ID3D11DeviceContext* ctx, ID3D11Resource* src, vector<unsigned char>& dst, vector<unsigned char>& stencil)
{
    // static void GET_SCREEN_RESOLUTION(int* x, int* y) { invoke<Void>(0x888D57E407E63624, x, y); } // 0x888D57E407E63624 0x29F3572F
    HRESULT hr = S_OK;
    int screenResX;
    int screenResY;

    ComPtr<ID3D11Texture2D> src_tex;
    GRAPHICS::GET_SCREEN_RESOLUTION(&screenResX, &screenResY);

    hr = src->QueryInterface(__uuidof(ID3D11Texture2D), &src_tex);
    if (hr != S_OK) throw std::system_error(hr, std::system_category());
    D3D11_TEXTURE2D_DESC src_desc;
    src_tex->GetDesc(&src_desc);
    assert(src_desc.Format == DXGI_FORMAT_R32G8X24_TYPELESS);
    D3D11_MAPPED_SUBRESOURCE src_map = { 0 };
    hr = ctx->Map(src, 0, D3D11_MAP_READ, 0, &src_map);
    if (hr != S_OK) throw std::system_error(hr, std::system_category());
    if (dst.size() != src_desc.Height * src_desc.Width * 4) dst = vector<unsigned char>(src_desc.Height * src_desc.Width * 4);
    if (stencil.size() != src_desc.Height * src_desc.Width) stencil = vector<unsigned char>(src_desc.Height * src_desc.Width);

    if (screenResX >= src_desc.Width)
    {
        for (int x = 0; x < src_desc.Width; ++x)
        {
            for (int y = 0; y < src_desc.Height; ++y)
            {
                const float* src_f = (const float*)((const char*)src_map.pData + src_map.RowPitch*y + (x * 8));
                unsigned char* dst_p = &dst[src_desc.Width * 4 * y + (x * 4)];
                unsigned char* stencil_p = &stencil[src_desc.Width * y + x];
                memmove(dst_p, src_f, 4);
                memmove(stencil_p, src_f + 1, 1);
            }
        }
    }
    else
    {
        // resize, for when depth map is bigger than screen image.
        float scale = ((float) src_desc.Width) / ((float) screenResX);
        dst = vector<unsigned char>(screenResY * screenResX * 4);
        for (int x = 0; x < screenResX; ++x) // screenResX
        {
            for (int y = 0; y < screenResY; ++y) //screenResY
            {
                int scaledX = int(x*scale);
                int scaledY = int(y*scale);
                const float* src_f = (const float*)((const char*)src_map.pData + int(src_map.RowPitch*scaledY + (scaledX * 8)));
                unsigned char* dst_p = &dst[screenResX * 4 * y + (x * 4)];
                unsigned char* stencil_p = &stencil[screenResX * y + x];
                memmove(dst_p, src_f, 4);
                memmove(stencil_p, src_f + 1, 1);
            }
        }
    }
    ctx->Unmap(src, 0);
}
IanKirwan commented 6 years ago

See pull request 30