Indiana Jones And The Emporer's Tomb missing textures

[Radio30]

Hi, I've just been using d3d8to9 for the first time and am getting missing textures through out the game:

Before: http://imgur.com/zujLRLe After: http://imgur.com/ZO0PU7B

Also the in-game movies are just a black screen with audio, the game never crashes and works fine otherwise. Is there a log file somewhere i'm missing that might tell me whats causing the missing textures? cheers

[elishacloud]

I purchased the game on GOG and was able to see the same issue of the missing textures. When I try other converters, like the one here, it works fine. I believe this is the same issue I reported on Hitman 2 Silent Assassin.

BTW: this game is on sale for $1.49 on GOG until June 27, if someone wants to try and reproduce this issue.

[elishacloud]

Ok, I figured out why the textures are missing. It seems that Indiana Jones and the Emperor's Tomb (indy.exe) is calling CreateTexture() 468 times when the game first starts. After that it tries calling SetTexture(). From what I can tell it is looping and calling SetTexture() around 192 times or so each loop.

However the issue is that around 42 of the times each loop SetTexture() is called with a nullptr. So somehow Direct3DTexture8 is getting deleted between the time it is created and the time indy.exe calls the SetTexture() API.

A few things to note about the SetTexture() API calls:

1. Stage = 0, 1, 2, 3 are always called in order.
2. Stage = 1, 2 and 3 seem to fail for the first 10 to 14 times each loop, then they start working for the rest of the loop.
3. Stage = 0 always works.
4. Stage = 1 is always a D3DRTYPE_CUBETEXTURE, whereas all the other stages are always D3DRTYPE_TEXTURE.

I tested this with Hitman 2 Silent Assassin and got the same results where SetTexture() is called with a nullptr so it seems to be the same issue.

I added a few extra log entries and attached them if someone wants to look at it.

Here is an example:

.....
16140 01:18:59.040 Set Texture, Stage=0 Type=3 addr=09525DC8
16140 01:18:59.040 Set Texture, Stage=1 Type=0 addr=null
16140 01:18:59.040 Set Texture, Stage=2 Type=0 addr=null
16140 01:18:59.040 Set Texture, Stage=3 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=0 Type=3 addr=09525738
16140 01:18:59.043 Set Texture, Stage=1 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=2 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=3 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=0 Type=3 addr=09525B28
16140 01:18:59.043 Set Texture, Stage=1 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=2 Type=0 addr=null
16140 01:18:59.043 Set Texture, Stage=3 Type=0 addr=null
.....
16140 01:18:59.050 Set Texture, Stage=0 Type=3 addr=09527AE0
16140 01:18:59.050 Set Texture, Stage=1 Type=5 addr=0952CD58
16140 01:18:59.050 Set Texture, Stage=2 Type=3 addr=09527AE0
16140 01:18:59.050 Set Texture, Stage=3 Type=3 addr=09527AE0
16140 01:18:59.054 Set Texture, Stage=0 Type=3 addr=09527CD8
16140 01:18:59.054 Set Texture, Stage=1 Type=5 addr=0952CD58
16140 01:18:59.054 Set Texture, Stage=2 Type=3 addr=09527CD8
16140 01:18:59.054 Set Texture, Stage=3 Type=3 addr=09527CD8
16140 01:18:59.057 Set Texture, Stage=0 Type=3 addr=09527AE0
16140 01:18:59.057 Set Texture, Stage=1 Type=5 addr=0952CD58
16140 01:18:59.057 Set Texture, Stage=2 Type=3 addr=09527AE0
16140 01:18:59.057 Set Texture, Stage=3 Type=3 addr=09527AE0
.....

d3d8-indy.zip

[crosire]

You could try disabling the Direct3DTexture8::Release method(s) and see if that solves the visual errors (since that should prevent all textures from ever being deleted).

[elishacloud]

I tried disabling both Direct3DTexture8::Release() and Delete This as well as calling an additional Direct3DTexture8::AddRef(). None of them helped. In addition I added logging for when the Direct3DDevice8, Direct3DTexture8 and Direct3DCubeTexture8 were created and deleted. It seems that all of these objects were created at the beginning of the program, before SetTexture() was called, and deleted at the end of the program after all the SetTexture() calls are completed. Therefore it does not seem to be an issue with d3d8to9 directly deleting the texture objects.

Only 1 instance of Direct3DDevice8 and Direct3DCubeTexture8 are created, and 468 instances of Direct3DTexture8 are created.

I am puzzled why indy.exe is making SetTexture() calls with nullptr.

[crosire]

If they were deleted too early, the game would likely not have passed nullptr but a dangling pointer to the deleted object, so that makes sense. As to the nullptr question: You have to call it that way to unbind a resource from the pipeline. It's normal behavior, so I don't think this is the reason. I much more suspect something in the shader conversion is going wrong or resource slots aren't handled correctly.

[Radio30]

I don't know a lot about this but I've tested running the game whilst having deleted different files from D:\GoG Games\Indiana Jones and the Emperor's Tomb\GameData\shaders\ and i've found that deleting 'StaticOmni0.vso' removes all of the models with the offending textures completely while everything else remains as it should be

[elishacloud]

I looked at this a little bit more today. Thanks corsire for pointing me in the right direction. As far as I can tell what is happening is that Direct3DDevice8::CreateVertexShader is called twice in this game. Both calls complete fine and go through all the disassembling and reassembling and creating a shader handle (using the ShaderMagic). Debugging this function seems to show that it is working fine for both calls. At least I could not find any obvious issues with this function in either call.

However even though the Direct3DDevice8::CreateVertexShader completes fine with both calls only the shader from the first call seems to work. The game seems to play as if the second call is ignored (or all the SetVertexShader calls to that handle are ignored). If I specifically add code to ignore the handle to the second Direct3DDevice8::CreateVertexShader call there is no change in the game play and the same textures are missing.

Interestingly enough I ran the same test with Hitman 2 Silent Assassin and the results are similar. Hitman 2 Silent Assassin calls Direct3DDevice8::CreateVertexShader several times, all complete successfully but only the shader from the first call seems to work.

Other games that I have call Direct3DDevice8::CreateVertexShader multiple times and all their shaders seem to work with d3d8to9. Not sure why with these two games the shader in the first call is the only one that works.

Any ideas on what I should look at next? You mentioned it could be an issue with resource slots. How would I verify that?

[Radio30]

Thank you both for looking into this! The solution is well beyond my comprehension but I'm keen to learn about this more as i go. I'll be so happy to see my favorite game running with all the cool ReShade effects!

[crosire]

Mmh. It might be that texture coordinates aren't properly communicated from the vertex to the pixel shader. But the game simply ignoring a created shader could have different reasons. I don't see how it would be able to detect a "faulty" shader and ignore it. More likely it just creates various shader permutations upfront of which some may only be used in very specific situations (that didn't appear in your tests). As for my previous comment on resource slots: It could be tested whether SetTexture etc. behave correctly by replacing all non-nullptr textures in SetTexture calls with a custom test texture and then see if the issue still persists (now the entire game will be made out of that single texture, but if the objects that were "missing" texture before are still not properly texturized, then we know the error is somewhere else). The most likely place is that the vertex/pixel shader translation does something wrong. The regex used there might mess something up it shouldn't actually touch. So I'd dump all the shaders before (right after dissassembling) and after (right before re-assembling) modification and check if something obvious sticks out.

[elishacloud]

Thanks crosire. That is very useful. I will try your trick with the SetTexture to see if the issue is with resource slots.

BTW: yesterday I tried dumping all the vertex shaders right after dissassembling and right before re-assembling and did a diff on them to verify they did not get messed up. I really don't think it is a problem with the regex used there.

[HerMajestyDrMona]

Have you tried to force an older Pixel Shader version? Edit: Output.PixelShaderVersion = D3DPS_VERSION(1, 4); to: Output.PixelShaderVersion = D3DPS_VERSION(1, 1);

There was a similar problem on Empire Earth II - missing fire and explosions effects, but affecting only Intel HD GPUs. With Nvidia everything worked fine. I found out that setting Pixel Shader version to 1.1 fixed this issue.

EDIT: I was close, but the issue is: Output.VertexShaderVersion = D3DVS_VERSION(1, 1);

after changing to: Output.VertexShaderVersion = D3DVS_VERSION(1, 0);

It seems to work better. No d3d8 verson: http://i.imgur.com/n9pBqM1.png Default d3d8: http://i.imgur.com/MKunmld.png With VertexShaderVersion fix: http://i.imgur.com/DZGRLA7.png

What I would do is an additional config file for d3d8, then something like this:

Output.PixelShaderVersion = D3DPS_VERSION(PixelShaderMajor, PixelShaderMinor);
Output.VertexShaderVersion = D3DVS_VERSION(VertexShaderMajor, VertexShaderMinor);

I have some config file read function, but it's too dirty to create a pull request :D Anyway it can be seen here: https://pastebin.com/raw/aBvf9zUX

[elishacloud]

I tried replacing all non-nullptr textures in SetTexture calls with a custom texture and the same objects were still had missing textures. I guess this means it is not a resource slot issue.

As far as the pixel shader issue HerMajestyDrMona brought up, this game is not using pixel shaders only vertex shaders (sorry for the confusion). I added logging to both CreatePixelShader and SetPixelShader and neither one is called by this game. Only CreateVertexShader and SetVertexShader are called. However I am wondering if we want to change the PixelShaderVersion to always be D3DPS_VERSION(1, 1) rather than D3DPS_VERSION(1, 4) to reduce compatibility issues.

[HerMajestyDrMona]

I edited my previous answer and it seems like setting VertexShaderVersion to 1.0 fixes this issue. I don't recommend changing PixelShaderVersion to 1.1 as default, because it might cause some new problems in games released after ~2004. For example, in Empire Earth II: The Art of Supremacy changing PixelShaderVersion to 1.1 causes bugs where "High Lighting Details" in game don't work. I think it's best to add a configuration file, where it would be possible to change Pixel/Vertex shader versions, disable log, Vsync, etc.

[elishacloud]

You are right that changing VertexShaderVersion to 1.0 solves most of the issues. I would love to find out how you figure that out! But there a few problems with this solution:

1. This casues a new issue that happens now and it seem like one texture is way too dark.
2. VertexShaderVersion 1.0 is not supported by Direct3D9.
3. You don't need to set this to D3DVS_VERSION(1, 0);. Just setting Output.VertexShaderVersion to 0 has the same effect. In other words it seems you can set this to a bogus value and get the same result.

My guess is that when this call is made with an unknown (or unsupported) value the OS has some way to handle it that fixes the shader issues. I am wondering if some other value is not getting initialized right and putting a bogus value in VertexShaderVersion forces the OS to initialize it.

[HerMajestyDrMona]

I found out the "fix" with Pixel Shader versions, by testing EE2 with a program called 3D-Analyze ( https://www.3dfxzone.it/dir/tools/3d_analyze/download/ ). It seems like graphics drivers for Integrated GPU (Intel HD 4600 in my case) have bug with missing/deformed fire textures. This problem doesn't occur on my other GPU (GTX 870M). So then I found out that skipping PixelShaderVersion 1.4 in 3D-Analyze fixed this issue on Intel GPU, but then it was impossible to change Lighting Details in game to "High". After d3d8 was released I decided to use it instead of 3D-Analyze, and that's how I found that this solution works for some issues in other games.

You are right that changing PixelShaderVersion to 1.0 solves most of the issues.

Did you mean VertexShaderVersion? If not, then PixelShaderVersion should be kept on 1.4, because: From my experience objects that use "lightings" don't work when Pixel Shader version is lower than 1.4 (giving from the EE2 example, not sure if it's the same on the other games). Example: http://i.imgur.com/HWB4Wlr.jpg

So basically, you should: Keep PixelShaderVersion on 1.4 Change VertexShaderVersion to 1.0 (or 0 as you mentioned) and see if everything works fine.

But just to make things clear: I don't know much how DX8/DX9 works nor I'm not a good C++ programmer, so those are just ideas from my experience, but I don't have any logical explanation why it happens.

What I would recommend you to do now is: Download 3D Analyze and try to use it together with d3d8to9 (remember to run it as administrator). 3D Analyze uses it's own d3d8.dll. I think it could be done by editing 3D Analyzer's d3d8.dll with a HEX editor, and replacing "d3d8.dll" with for example "new9.dll" (which would be the current d3d8.dll from d8to9 converter). Then try random options in 3D Analyzer's window and see which one fixes the dark textures problem (if any does).

EDIT: Actually I'm not sure how to run d8to9 together with 3D Analzye. It seems that 3D Analyze deletes d3d8.dll when it's present in game folder... So it's hard to figure out what it is doing.

[elishacloud]

I am not an expert at either C++ or Direct3D either. In fact I just started working on both of them this year. A long time ago (+20 years) I used to program in basic (please don't laugh!)

Anyways I looked into setting the Vertext Shader to 1.0 and it looks like if you set this to any number smaller than 1.1 some programs will disable vertex shading altogether. See here for an example. I suspect this is what indy.exe is doing. Thus I don't think this is really a solution, just a work around. Because we don't really want to disable the shaders, instead we want to get them working with Direct3D9.

However, all is not lost. This does point us in the right direction. Because if disabling vertext shaders solves the missing textures then it is likely that the textures issue is related to vertext shaders, which is what crosire said earlier.

If someone has a suggestion for me I am happy to try it out. I am also happy to post log files if someone wants to look at them.

[elishacloud]

The issue is in the Direct3DDevice8::CreateVertexShader function with how the registers are initialized.

1. Removing or remarking out this line here:

   ConstantsCode += "    def c95, 0, 0, 0, 0\n";

2. Changing the c95 to c0 in these lines here, here and here:

   SourceCode.insert(DeclPosition + ConstantsCode.size(), "    mov " + reg + ", c95 /* initialize output register " + reg + " */\n");

   As seen in pull request #27 resolves the issue of missing textures.

[Radio30]

Hi elishacloud, It looks like you've solved this!! As i don't understand how to make these changes would you mind uploading the required edited file(s)? It'd make my week!

[elishacloud]

You can download the updated d3d8to9_device.cpp file here. You can also download the updated source code to the whole project here.

I have also attached an updated compiled d3d8.dll version in case you have any issues compiling this project. You can download it here.

[HerMajestyDrMona]

I didn't want to comment to not spam, but I must write it: Awesome work fixing all those issues! I've been following this and the other threads and it's very satisfying to see that most (if not all) issues are being fixed! I like those regex_replace and PaletteNumber fixes as well. Thank you for spending time on it!

[Radio30]

!!!! Just tested it out, it works amazingly! Thanks so much for uploading the compiled d3d8.dll and all your great work!

[crosire]

Should be fixed now, so closing this.