Can't wrap custom d3d8.dll with d3d8to9

[BC46]

I'm trying to run a game with a d3d8.dll from an older Windows 10 version that fixes some lighting-related issues the game has with the default Windows 10 d3d8.dll. Additionally, I'd like to wrap the older d3d8.dll with d3d8to9 so I can enable AF and AA.

This is what I have in my game folder: dxwrapper.dll dxwrapper.ini with the following config:

[Compatibility]
D3d8to9                    = 1

[d3d9]
AnisotropicFiltering       = 16
AntiAliasing               = 1

This file includes all the other values in the default dxwrapper.ini file, but those have not been changed from their defaults. d3d8.dll from the Stub folder d3d8.ini with the following config:

;; Config file for Stub to dxwrapper
[General]
RealDllPath       = Custom\d3d8.dll
WrapperMode       = d3d8.dll
StubOnly          = 0
LoadFromMemory    = 0

Custom\d3d8.dll. This is the custom d3d8.dll that I want to load instead of the default Windows 10 one.

With all these files and configs, d3d8to9 seems to load since the AF and AA are both working. However, it also seems the default Windows 10 d3d8.dll has loaded instead of the custom one, since the lighting bug is present in the game.

When I change D3d8to9 in dxwrapper.ini from 1 to 0, the exact opposite happens; the custom d3d8.dll loads, which has resulted in the lighting bug being gone, but obviously now d3d8to9 doesn't load as there is no AF nor AA.

It seems like I can only get one or the other to work simultaneously, but I'd like d3d8to9 to wrap the custom d3d8.dll. Am I missing something or is it impossible to get the result I'm trying to achieve here?

I'm using DxWrapper v1.0.6542.21. (latest release)

[mirh]

You should also open another issue about that lighting bug, since it's kinda within the aims of dxwrapper to fix too.

[BC46]

You should also open another issue about that lighting bug, since it's kinda within the aims of dxwrapper to fix too.

I wasn't sure if I was supposed to do this since it's a very specific issue. But I'll do it, thanks for letting me know.

[BC46]

I decided to run the debug version of DxWrapper to see if that would show useful logs.

With the same config as before and D3d8to9 set to 0, it outputs this in the log:

20000 23:38:42.140 Wrapping 'd3d8.dll'...
20000 23:38:42.141 Loading 'Custom\d3d8.dll' as real dll...

The custom d3d8.dll loads correctly and works fine in-game here.

Now with D3d8to9 set to 1, those 2 lines no longer show in the output. It doesn't even mention anything about a d3d8.dll, which leads me to believe that it simply loads the default Windows 10 one.

[dmutlu]

I am able to replicate this behavior on my system. It seems setting "D3d8to9" to true prevents the running of whatever DLL is specified in "RealDLLPath". Attached is two screenshots from ProcExp showing the result of flipping "D3d8to9" between true/false.

D3d8to9 = 1

D3d8to9 = 0

I am a novice when it comes to C++ but I am trying my best to look through the code to see why this is. My guess is maybe the logic/issue involves lines 187 and 435 of Dllmain.cpp.

My two theories:

1. This is not a feature and dxwrapper never passes the DLL specified in Config.RealDLLPath to the D3d8to9 module.
2. This is a feature but there is a conditional statement somewhere incorrectly preventing the passing of the specified RealDLLPath to the D3d8to9 module.

[elishacloud]

I am a novice when it comes to C++ but I am trying my best to look through the code to see why this is.

This is nothing to do with the C++ code. If you convert the game to use d3d9 (using d3d8to9) then it does no good to load a 3rd party d3d8.dll since the game is using d3d9 not d3d8.

Think about it this way, all 3rd party wrappers are hard coded to load the System32 dll files. So a 3rd party d3d8.dll wrapper will load System32 d3d8.dll. It looks like this:

Game -> (dxwrapper) d3d8.dll -> (3rd party) d3d8.dll -> (System32) d3d8.dll

However, when using 'd3d8to9' dxwrapper will load d3d9.dll rather than d3d8.dll like this:

Game -> (dxwrapper) d3d8.dll -> d3d9.dll

If dxwrapper tries to load a 3rd party dll before loading d3d9.dll then there will be no way for the game to use d3d9. See below:

Game -> (dxwrapper) d3d8.dll -> (3rd party) d3d8.dll -> (System32) d3d8.dll

There are two ways around this. The first way around this would be to hook the System32 d3d8.dll functions so that when the game calls them it will go back to dxwrapper, like this:

Game -> (dxwrapper) d3d8.dll -> (3rd party) d3d8.dll -> (System32 - hooked) d3d8.dll -> (dxwrapper) d3d8.dll -> d3d9.dll

The other solution is to modify the 3rd party wrapper to have it load dxwrapper rather than the System32 d3d8.dll, like this:

Game -> (3rd party) d3d8.dll -> (dxwrapper) d3d8.dll -> d3d9.dll

All this to say that it is a lot more work to support loading a 3rd party d3d8.dll and d3d8to9 at the same time. I have done a few custom developments of this (see here and here for examples). In both of these cases I chose to modify the 3rd party dll since that was the easier solution. I have thought about making a feature in dxwrapper to support this but I probably won't get around to it anytime soon.

Note: if you just want the dll to be loaded you can use either the 'LoadCustomDllPath' or the 'LoadPlugins' feature of dxwrapper.

[dmutlu]

Thank you for the response and explanation @elishacloud, I appreciate it. In this specific use case the "3rd-party" dll is just an older version of Microsoft's d3d8.dll from Windows 10 19H1. Post 20H1 Microsoft seems to have replaced the d3d8.dll found in the System32 dir with d3d8thk.dll. This new version seems to carry some regressions in terms of lighting and shadows, specifically for the game Freelancer, which I guess folks will have to live with for now.

[elishacloud]

If you are using Microsoft's d3d8.dll then enabling d3d8to9 will by pass that and use d3d9.dll instead.

Does d3d8to9 work ok with Windows 10 19H1? What if you try the d3d9.dll from Windows 10 19H1? Just drop it in the game folder along with dxwrapper and enabled3d8to9. Does it work then?

Can you give more details on the lighting issue? Maybe this is something we can fix in d3d8to9?

[dmutlu]

Does d3d8to9 work ok with Windows 10 19H1?

Not sure, I am running it in a VM just so I can grab the DLLs. Maybe @BC46 can answer since they originally found the older DLL trick.

What if you try the d3d9.dll from Windows 10 19H1? Just drop it in the game folder along with dxwrapper and enabled3d8to9. Does it work then?

I just tried this, I can see that dxwrapper is loading the older d3d9.dll but I am still seeing the lighting / shading issue.

Can you give more details on the lighting issue? Maybe this is something we can fix in d3d8to9?

Sure, I have attached a zip with two comparison screenshots (and also my dxwrapper log during the older d3d9.dll test). But here is a description of what we are seeing if you do not want to download the zip.

Issue: In specific interior locations lighting, shading, and transparency is broken with d3d9.dll

1. Objects (ships, NPCs, crates, clutter) that interact with scene lighting appear fullbright, having no shading.
2. Some interior level windows lose transparency, becoming opaque (black in color).

20220907.zip

Side Note: Just to correct my past self and for any future readers. I mentioned that d3d8.dll was replaced in later builds of Windows 10, this was an incorrect assumption I made. For some reason I was not thinking about the WoW64 dir which actually stores the d3d8.dll. I guess my brain is just stuck on 64 bit these days...

[elishacloud]

Does d3d8to9 work ok with Windows 10 19H1?

Not sure, I am running it in a VM just so I can grab the DLLs. Maybe @BC46 can answer since they originally found the older DLL trick.

It would be good to know if d3d8to9 had an issue on the older Windows 10 with this game. I think the solution may be to fix d3d8to9 so that it does not have this lighting issue. In fact, does the lighting issue happen when using d3d8to9 on older OS's, like Windows 7, with this game?

BTW: I'm not sure, but I think maybe the change that Microsoft did after Windows 10 19H1 was to redirect Direct3D8 to go to the Driect3D9 drivers.

[BC46]

It would be good to know if d3d8to9 had an issue on the older Windows 10 with this game.

I have a PC running Windows 10 19H1 which I use for compatibility testing. I played Freelancer (the game in question) using just d3d9to9 on it and it ran completely fine without the lighting bug.

Windows 10 version 2004 (20H1), or build 19041 is the first known Windows version where the lighting bug appears. People started reporting this issue shortly after the release date of this build. Players who still use Windows 7 & 8 to play this game can confirm that they've never encountered the issue before. However, these older Windows releases, including Vista and even newer XP service packs do all have a Freelancer DirectX compatibility issue regarding glass reflections on the ship windows. However, it's so minor that it's hardly worth mentioning. As far as I know, dgVoodoo is the only graphics wrapper that fixes both the lighting bug and the glass reflections issue properly on modern Windows versions.

In fact, does the lighting issue happen when using d3d8to9 on older OS's, like Windows 7, with this game?

I currently don't have access to a PC/VM running such older versions of Windows. Though I think it's unlikely that d3d8to9 could be the culprit since the lighting bug seems to show depending on the Windows version/build, regardless of whether d3d8to9 is used.

BTW: I'm not sure, but I think maybe the change that Microsoft did after Windows 10 19H1 was to redirect Direct3D8 to go to the Driect3D9 drivers.

All I know myself is that the DirectX rendering pipeline has been heavily modified since the release of Windows 10 2004, which I'm assuming is the cause of the lighting bug.

[elishacloud]

It would be good to know if d3d8to9 had an issue on the older Windows 10 with this game.

I have a PC running Windows 10 19H1 which I use for compatibility testing. I played Freelancer (the game in question) using just d3d9to9 on it and it ran completely fine without the lighting bug.

Ok, it sounds like d3d8to9 is not causing the issue. Have you tried using d3d8to9 and then using Direct3D9 to Vulkan? This way you can bypass whatever the Windows bug is.

[BC46]

Yes, I tried that a while ago. I tested Freelancer using d3d8to9 + DXVK's d3d9.dll. This actually did fix the lighting bug, as a matter of fact. However, DXVK also introduced several compatibility issues in Freelancer specifically, such as inconsistent stutters and LOD models not always being displayed correctly. Hence I prefer using different solutions to counter the lighting bug.

[elishacloud]

I see. Maybe I should fix the lighting issue in my d3d9 wrapper? The way I have coded the d3d8to9 function is that it goes through the d3d9 wrapper. So any d3d9 fixes I add will also fix d3d8to9.

[BC46]

If you think it's worth the effort, then that would be amazing. I'll provide any additional info and help if needed.

[BC46]

I wanted to do some more research and @elishacloud's info regarding Direct3D8 being redirected to the Direct3D9 drivers after Windows update 19H1 gave me some clues as to where I could look.

First I should mention how Freelancer makes use of DirectX 8's lights. Basically the game uses so-called Thorn scripts to specify what kind of lights should be used in a scene. Inherently these are just compressed Lua scripts. Here's an example of how a light is set in one of the scripts that has been decompressed:

{
    entity_name  =  "ambi_LtG03_Basement_Ohd_Ylw",
    type  =  LIGHT,
    template_name  =  "",
    lt_grp  =  3, srt_grp  =  0, usr_flg  =  0,
    spatialprops  = 
    {
        pos  =  { -5.010058, 0, 1.758013 },
        orient  =  { { -0.105263, -0.008535, -0.994408 },
                   { 0.977260, 0.184208, -0.105029 },
                   { 0.184075, -0.982850, -0.011049 } }
    },
    lightprops  = 
    {
        on  =  Y,
        color  =  { 234, 192, 115 },
        diffuse  =  { 0.623529, 0.576471, 0.376471 },
        specular  =  { 0, 0, 0 },
        ambient  =  { 0.086275, 0.086275, 0.07451 },
        direction  =  { 0, 0, 1 },
        range  =  100,
        cutoff  =  179,
        type  =  L_SPOT,
        theta  =  179,
        atten  =  { 1, 0, 0 }
    }
}

If you look closely at the parameters in the lightprops entry, you can see that they look very similar to the values in the D3D8LIGHT8 struct.

A while ago I talked to another Freelancer modder about the lighting bug. After a small investigation they concluded the light type L_SPOT became broken after the Windows update. So they "fixed" it by going into every Thorn script to change each light with the type L_SPOT to L_POINT. Internally this changes the DirectX 8 light type from D3DLIGHT_SPOT to D3DLIGHT_POINT. However, in my opinion this isn't really a good solution because point lights behave very differently from spot lights.

I really wanted to look for a better solution, and based on the clue regarding the redirection of Direct3D8 to the Direct3D9 drivers, I checked out the documentation for the lights from both DirectX 8 and DirectX 9. One thing that stood out to me right away was that the description for the members Falloff, Theta, and Phi simply said "Not supported." in the DirectX 8 documentation, whereas in the DirectX 9 documentation it seems these definitely are supported. In the light specifications from the Thorn scripts I noticed that the value theta was set despite it apparently not even being supported in DirectX 8.

I wanted to find out whether or not the sudden support of the three values in DirectX 9 is the cause of the lighting bug. Though I did not know what the given light values for Falloff or Phi are since they are not specified in the light properties. Hence I decided to use @elishacloud's DirectX-Wrappers project to intercept the IDirect3DDevice8::SetLight method so I could see what the values for each light are and what happens when they are modified.

After some trial and error, I came up with this:

HRESULT m_IDirect3DDevice8::SetLight(DWORD Index, CONST D3DLIGHT8 *pLight)
{
    D3DLIGHT8* pLightEdit = const_cast<D3DLIGHT8*>(pLight);

    // Only the light type D3DLIGHT_SPOT suffers from the lighting bug
    if (pLightEdit->Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        // Fix lighting bug
        pLightEdit->Falloff *= 0.5f;
        pLightEdit->Theta *= 0.5f;

        // Impose minimum Falloff to ensure shadows won't look too sharp
        if (pLightEdit->Falloff < 0.6f)
            pLightEdit->Falloff = 0.6f;
    }

    return ProxyInterface->SetLight(Index, pLightEdit);
}

Using the d3d8 wrapper with this code in Freelancer completely rectifies the lighting bug. Basically I reduced the Falloff and Theta values for spot lights to make them appear less harsh. Furthermore, I set a minimum Falloff value to ensure the light edges/shadows look smoother.

I should mention that I can't confirm whether my suspicion about Falloff, Theta, and Phi not working on DirectX 8 is true because I actually haven't done any tests on an older Windows version. Also, with these updated values the lighting doesn't look exactly the same as how it originally did on older Windows versions; in most scenes the light looks slightly dimmer. Maybe a better DirectX programmer can have a look at it one day. Here's the source code for the implemented lighting bug fix as a proof of concept: https://github.com/BC46/freelancer-lighting-bug-fix.

Before and after screenshots:

[mirh]

If you have a possible regression range, bisecting with vmware workstation should really be a cakewalk. You can even go sub-file or sub-version if you have a bit more time.

p.s. for all their bluntness, I think the DXVK people would appreciate you opening a ticket for whatever bug they have (though first I'd hope you tried all the d3d9 compatibility options in dxvk.conf).

[elishacloud]

@BC46, thanks for your good research on this! I don't have the game so it is hard for me to test it. But I would like to make a generic fix for this.

One thing that stood out to me right away was that the description for the members Falloff, Theta, and Phi simply said "Not supported." in the DirectX 8 documentation

The documentation that you showed is for Windows CE, it may not be applicable to Windows XP or even Windows 7. Windows CE may be using a cut-down version of DirectX. Also, one thing I have seen in the past is that the DirectX APIs were more forgiving back then. Now Microsoft keeps adding more checks and rejecting API calls that used to work.

If we look at the documentation here we can see that Phi needs to have a number between 0 and Pi (3.14159...) and that Theta cannot be greater than Phi.

What if you change the code to look more like this?:

HRESULT m_IDirect3DDevice8::SetLight(DWORD Index, CONST D3DLIGHT8* pLight)
{
    D3DLIGHT8* pLightEdit = const_cast<D3DLIGHT8*>(pLight);

    //  Fix 'Phi'
    if (pLightEdit->Phi < 0.0f)
    {
        pLightEdit->Phi = 0.0f;
    }
    if (pLightEdit->Phi > M_PI)
    {
        pLightEdit->Phi = M_PI;
    }
    //  Fix 'Theta'
    if (pLightEdit->Theta < 0.0f)
    {
        pLightEdit->Theta = 0.0f;
    }
    if (pLightEdit->Theta > pLightEdit->Phi)
    {
        pLightEdit->Theta = pLightEdit->Phi;
    }

    return ProxyInterface->SetLight(Index, pLightEdit);
}

[elishacloud]

One more thing you could try is since the documentation claims that Falloff, Theta, and Phi are not supported in DirectX8 you could just try setting them to 0.0f which is the default, according to this site here.

You could also try this one:

HRESULT m_IDirect3DDevice8::SetLight(DWORD Index, CONST D3DLIGHT8* pLight)
{
    D3DLIGHT8* pLightEdit = const_cast<D3DLIGHT8*>(pLight);

    pLightEdit->Falloff = 0.0f;
    pLightEdit->Phi = 0.0f;
    pLightEdit->Theta = 0.0f;

    return ProxyInterface->SetLight(Index, pLightEdit);
}

Edit: this might be the best idea since the code I see on the Internet for d3d8 don't seem to ever set any of these three values and they tend to zero the memory before using:

D3DLIGHT8 light;
ZeroMemory( &light, sizeof(D3DLIGHT8) );

[elishacloud]

Here is some good documentation on how this works in DirectX: http://www.directxtutorial.com/Lesson.aspx?lessonid=9-4-9

[elishacloud]

So they "fixed" it by going into every Thorn script to change each light with the type L_SPOT to L_POINT. Internally this changes the DirectX 8 light type from D3DLIGHT_SPOT to D3DLIGHT_POINT.

I think this worked because D3DLIGHT_POINT ignores the values of Falloff, Theta, and Phi. The key issue, I think is that the game is sending values for Falloff, Theta, and Phi that are not allowed by DirectX. Thus, the best fix, I think is to fix the values so they are accepted.

I believe this is generic code that should fix this in all cases:

#define _USE_MATH_DEFINES
#include <math.h>

HRESULT m_IDirect3DDevice8::SetLight(DWORD Index, CONST D3DLIGHT8* pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    // The maximum allowable value for 'range' is the square root of FLT_MAX.
    static float MaxRange = sqrtf(FLT_MAX);
    if (LightEdit.Range > MaxRange)
    {
        LightEdit.Range = MaxRange;
    }

    switch  (LightEdit.Type)
    {
    case D3DLIGHTTYPE::D3DLIGHT_SPOT:
        // 'Phi' must be between 0 and pi.
        if (LightEdit.Phi < 0.0f)
        {
            LightEdit.Phi = 0.0f;
        }
        if (LightEdit.Phi > M_PI)
        {
            LightEdit.Phi = M_PI;
        }
        //  'Theta' must be in the range from 0 through the value specified by 'Phi'.
        if (LightEdit.Theta < 0.0f)
        {
            LightEdit.Theta = 0.0f;
        }
        if (LightEdit.Theta > LightEdit.Phi)
        {
            LightEdit.Theta = LightEdit.Phi;
        }
        // 'Falloff' a falloff other than 1.0 takes time to process, developers usually use 1.0
        LightEdit.Falloff = 1.0f;
        break;
    case D3DLIGHTTYPE::D3DLIGHT_DIRECTIONAL:
        LightEdit.Range = 0.0f;     // This member does not affect directional lights.
        __fallthrough;
    case D3DLIGHTTYPE::D3DLIGHT_POINT:
        LightEdit.Falloff = 0.0f;
        LightEdit.Phi = 0.0f;
        LightEdit.Theta = 0.0f;
        break;
    default:
        return D3DERR_INVALIDCALL;
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

Please let me know if this works.

[mirh]

Putting aside that this should all be easily readable in the offline docs shipped with the sdk, you are right that the windows CE version was different. https://web.archive.org/web/20040103201823/http://msdn.microsoft.com/archive/en-us/dx81_c/directx_cpp/GRAPHICS/Reference/CPP/D3D/Structures/d3dlight8.asp https://web.archive.org/web/20040806062027/http://msdn.microsoft.com/archive/en-us/directx9_c/directx/graphics/reference/d3d/structures/d3dlight9.asp https://web.archive.org/web/20040907224729/http://msdn.microsoft.com/archive/en-us/directx9_c/directx/graphics/reference/d3d/enums/d3dlighttype.asp

[BC46]

Thank you for the links and the code snippets @elishacloud!

With the first and third snippet the lighting bug was still present in the game unfortunately; same result as this:

The second one made all the objects either black or very dark:

Would it be helpful if I sent a collection of D3DLIGHT8 instances that DirectX receives from the game during this particular scene? Maybe that will allow you to spot any peculiar values if there are any.

[elishacloud]

Would it be helpful if I sent a collection of D3DLIGHT8 instances that DirectX receives from the game during this particular scene?

Yes, this would be helpful. Thanks!

[BC46]

I hope the logging format is a bit clear: d3dlight8_log.txt.

[elishacloud]

Thanks for the logs! It seems like in almost every case (except index 4) it sets Theta and Phi to the same value. In this case it sounds like d3d9 is working correctly. It should show sharp contrast (like the undesirable graphic above) rather than a gradient change. See the "Phi and Theta" section from this page.

FallOff is always set to 1, which is the normal value, so we should not need to change that at all.

I think what might be happening is that older versions of DirectX used to treat it differently when Theta and Phi had the same value. In this case, to simulate how the older DirectX used to work we may be able to just set Theta to 0 when Theta and Phi have the same value.

We can try something like this:

HRESULT m_IDirect3DDevice8::SetLight(DWORD Index, CONST D3DLIGHT8* pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    if (LightEdit.Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        if (LightEdit.Phi == LightEdit.Theta)
        {
            LightEdit.Theta = 0.0f;
        }
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

Another idea is that after looking at the LUA:

{
    entity_name  =  "ambi_LtG03_Basement_Ohd_Ylw",
    type  =  LIGHT,
    template_name  =  "",
    lt_grp  =  3, srt_grp  =  0, usr_flg  =  0,
    spatialprops  = 
    {
        pos  =  { -5.010058, 0, 1.758013 },
        orient  =  { { -0.105263, -0.008535, -0.994408 },
                   { 0.977260, 0.184208, -0.105029 },
                   { 0.184075, -0.982850, -0.011049 } }
    },
    lightprops  = 
    {
        on  =  Y,
        color  =  { 234, 192, 115 },
        diffuse  =  { 0.623529, 0.576471, 0.376471 },
        specular  =  { 0, 0, 0 },
        ambient  =  { 0.086275, 0.086275, 0.07451 },
        direction  =  { 0, 0, 1 },
        range  =  100,
        cutoff  =  179,
        type  =  L_SPOT,
        theta  =  179,
        atten  =  { 1, 0, 0 }
    }
}

I noticed that "cutoff" and "theta" are the same value. I guess that one of them affects Theta and the other one affects Phi. Maybe if you changed one of these values to "0" it would also solve the issue. Presumably, you would want to set "cutoff" to 0.

[BC46]

Yes, that one did the trick. Thanks a lot!

Lighting fix:

Original (legacy d3d8.dll):

Slightly dimmer than what it looks like with the legacy d3d8.dll, but overall still a great result!

As for the Lua script, setting theta to 0 for lights with the L_SPOT type indeed worked as well. Though I still prefer the DirectX call intercept approach since this way I can programmatically check if the lighting bug would be present on the user's system. Based on that I can determine whether or not the fix should be applied. Also saves me the hassle of having to modify every single Lua script. 😃

[elishacloud]

Thanks for testing this!

To solve the issue where the lighting is darker than the legacy d3d8.dll, I suspect you could just double the FallOff. However, I want to try and isolate the issue with the legacy d3d8.dll as much as possible to ensure we can match the functionality.

If you would be willing to try some tests with the legacy d3d8.dll I would be grateful. I want to see if this only happens when both the Theta and Phi are set the exact same. In the LUA script if you change the theta to be only slightly smaller than the Phi does the picture still look good in the legacy d3d8.dll or does that mess up the lighting, like it does with the newer d3d8.dll files?

Edit: Also, when you use the Lua script to set theta to 0 for lights with the L_SPOT type, can you test this with the legacy d3d8.dll? I want to see if this makes the legacy d3d8.dll and the newer d3d8.dll appear the same in the game.

[BC46]

In the LUA script if you change the theta to be only slightly smaller than the Phi does the picture still look good in the legacy d3d8.dll or does that mess up the lighting, like it does with the newer d3d8.dll files?

For all L_SPOT lights I set theta to its cutoff value minus 2. E.g. one light has cutoff = 30 and theta = 28 (I'm assuming cutoff controls Phi). Loading this scene with the legacy d3d8.dll made pretty much no difference; the lighting still looks good.

Also, when you use the Lua script to set theta to 0 for lights with the L_SPOT type, can you test this with the legacy d3d8.dll? I want to see if this makes the legacy d3d8.dll and the newer d3d8.dll appear the same in the game.

Setting theta = 0 for all L_SPOT lights and loading the scene with the legacy d3d8.dll indeed makes the lights look the same as if they were rendered using the newer d3d8.dll.

[elishacloud]

one light has cutoff = 30 and theta = 28 (I'm assuming cutoff controls Phi). Loading this scene with the legacy d3d8.dll made pretty much no difference; the lighting still looks good.

Ok, this is what I was afraid of. I believe Theta works differently in the legacy then it does in the new driver.

Try this one and see if it makes the newer d3d8.dll look more like the legacy one:

HRESULT STDMETHODCALLTYPE Direct3DDevice8::SetLight(DWORD Index, const D3DLIGHT8 *pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    if (LightEdit.Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        if (LightEdit.Phi >= LightEdit.Theta)
        {
            LightEdit.Theta = 0.0f;
            LightEdit.Falloff = LightEdit.Falloff * (1.0f + (LightEdit.Theta / LightEdit.Phi));
        }
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

[BC46]

Thanks for the code snippet! The lighting looks about the same as with the last one.

[elishacloud]

Ok, thanks for all the tests. There is no way we can reproduce this exactly here. I hope we can get pretty close.

Can you try one more? I am hoping this will be close enough.

HRESULT STDMETHODCALLTYPE Direct3DDevice8::SetLight(DWORD Index, const D3DLIGHT8 *pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    if (LightEdit.Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        if (LightEdit.Phi >= LightEdit.Theta && LightEdit.Phi)
        {
            LightEdit.Theta /= 1.5f;
            LightEdit.Falloff *= 1.0f + (LightEdit.Theta / LightEdit.Phi);
        }
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

[BC46]

With these changes the lighting looks a lot more like the original! Some areas appear to a bit brighter, but still great nevertheless.

Here's a few comparisons:

Lighting fix:

Original (legacy d3d8.dll):

---

Lighting fix:

Original (legacy d3d8.dll):

[elishacloud]

Ok, great! Maybe play around with this line here:

LightEdit.Theta /= 1.5f;

Maybe divide it by 1.75f or 2.0f to reduce the brightness. Once it looks close enough let me know what number you chose.

[BC46]

I think a division by 1.75f is more accurate; makes the objects look a bit less bright, but still not too dim. Also, I tinkered a bit with the Falloff calculation and must say the following satisfied me as well:

if (LightEdit.Phi >= LightEdit.Theta && LightEdit.Phi)
{
    LightEdit.Theta /= 1.75f;
    LightEdit.Falloff *= (LightEdit.Theta / LightEdit.Phi);
}

Though in some cases the Falloff value might become very small with this code, causing the light edges to look quite harsh. If Falloff is increased directly in the calculation, the darker parts will become brighter, and the brighter parts darker. Therefore, it might be better to set some minimum for Falloff.

[elishacloud]

With that formula if both Theta and Phi are equal (which is the case you are testing) then you are just multiplying Falloff by 1, which isn't going to change anything. I suggest we just remove the line that changes the Falloff. If we do that we can also remove one of the conditions. Let's use this code instead. It should give you the same result, but simpler:

HRESULT STDMETHODCALLTYPE Direct3DDevice8::SetLight(DWORD Index, const D3DLIGHT8 *pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    if (LightEdit.Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        LightEdit.Theta /= 1.75f;
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

[BC46]

Thank you for the suggestion! I'm not exactly sure why but it seems to make a very small difference to the lighting. In this scene it's the most noticeable:

With Falloff adjustment:

Without Falloff adjustment and removed condition:

Still good results though!

[elishacloud]

Ok, great. I'm going to keep the if statement because we don't want to do anything if Theta is larger than Phi. I'm also not going to change the Falloff since that is best left at it current value.

Here is the final code I will use. I will make a pull request later in d3d8to9.

HRESULT STDMETHODCALLTYPE Direct3DDevice8::SetLight(DWORD Index, const D3DLIGHT8 *pLight)
{
    if (!pLight)
    {
        return D3DERR_INVALIDCALL;
    }

    D3DLIGHT8 LightEdit;
    memcpy_s(&LightEdit, sizeof(LightEdit), pLight, sizeof(LightEdit));

    // Make spot light work more like Direct3D8 worked
    if (LightEdit.Type == D3DLIGHTTYPE::D3DLIGHT_SPOT)
    {
        if (LightEdit.Theta <= LightEdit.Phi)   // Theta must be in the range from 0 through the value specified by Phi.
        {
            LightEdit.Theta /= 1.75f;
        }
    }

    return ProxyInterface->SetLight(Index, &LightEdit);
}

Here is an updated dxwrapper build with this fix in it: dxwrapper.zip

I also enabled the following:

[Compatibility]
D3d8to9                    = 1

[d3d9]
AnisotropicFiltering       = 16
AntiAliasing               = 1

[BC46]

Sounds good! When testing the updated DxWrapper build I noticed another issue in Freelancer which I can only reproduce when the D3d8to9 option is enabled. Should I make an issue on the d3d8to9 repo about it?

[elishacloud]

Does the issue happen when only d3d8to9 is enabled? Is the issue related to the lighting fix?

Can you try with just d3d8to9: d3d8.zip

What is the issue? Can you give some details here?

[mirh]

Some hopeful inspiration btw https://github.com/wine-mirror/wine/blob/wine-8.3/dlls/wined3d/device.c#L1580 https://github.com/Cxbx-Reloaded/Cxbx-Reloaded/blob/CI-6389cb6/src/core/hle/D3D8/Direct3D9/FixedFunctionVertexShader.hlsl#L137

[elishacloud]

Just from reviewing the code, it looks like Wine would have the same lighting issue with Freelancer that newer versions of Direct3D has.

[BC46]

Is the issue related to the lighting fix?

This one has something to do with the textures it seems.

What is the issue? Can you give some details here?

Turns out the issue is slightly more specific than I had anticipated.

Basically Freelancer uses so-called "detail textures" which are grayscale textures that are mapped on top of regular textures to make those appear more detailed. Here's an example of a detail texture that the game uses:

Initially I tested your DxWrapper build on my old laptop which has a GTX 1050. When I looked at one of the scenes, I noticed that these detail textures did not render at all. On the laptop this issue occurs when the AnisotropicFiltering option from DxWrapper is set to any value other than 0.

AnisotropicFiltering = 0:

AnisotropicFiltering = 1:

Notice how the floor in the second image is missing the hexagon pattern. The strange part is that I could not reproduce the issue at all on my main rig which uses a Radeon RX 5700. In this case the detail textures loaded just fine when setting AnisotropicFiltering to any of the supported values. There were no issues either when testing the d3d8to9 build on the GTX 1050 laptop.

Edit: Also notice how in the second image the ship's lights on the top right appear to be turned off. This is because the emission map for it failed to load.

[BC46]

Just from reviewing the code, it looks like Wine would have the same lighting issue with Freelancer that newer versions of Direct3D has.

I know a few players who play Freelancer on Linux using Wine. They mostly use a Windows XP prefix for Wine on which the lighting bug isn't present when playing the game.

[elishacloud]

Yeah that is an issue with AnisotropicFiltering. The issue is that the game was created without anisotropic filtering in mind and because of that some textures don't look good when it is enabled. DxWrapper tries to smartly enable it only where it works best. However, DxWrapper may not know all cases for when to enable it. For example, it causes artifacts in Silent Hill 2 and will likley never work for that game.

It might be possible to add an exclusion specific for Freelancer, but I don't have enough information about that specific texture nor do I want to add code for individual games. I am trying to create a generic tool.

For this specific texture, it is getting generated correctly but anisotropic filtering blurs it so the texture us lost. You could try reducing the filtering by setting it to AnisotropicFiltering = 2. But if that does not work then you probably won't be able to use that feature on this game.

[BC46]

Setting AnisotropicFiltering to 2 unfortunately made no difference; the textures still didn't load.

When you say that the game was created without anisotropic filtering in mind, do you mean in general or specifically when converting Direct3D 8 to Direct3D 9? Because recently as a test I force-enabled 16x AF and 8x AA in Freelancer using the d3d8 wrapper, and that worked like a charm without any repercussions.

[mirh]

They mostly use a Windows XP prefix for Wine on which the lighting bug isn't present when playing the game.

That's odd then (assuming I'm not too dumb to be able to link the right code)... What if you try it yourself?

[elishacloud]

That's odd then (assuming I'm not too dumb to be able to link the right code)...

Based on this line you can see that they subtract Theta from Phi. That would cause it to be "0" in this case and would make rho be based only on Theta, which is the issue we are seeing in Direct3D9 (and newer versions of Direct3D8). Technically, this is the correct formula. However, legacy Direct3D8 did not work this way.

You could argue that the legacy Direct3D8 used these values incorrectly, which is probably true. However, when games are made based on how the legacy software works I would prefer to keep it that way even if it is the wrong way. I am just trying to get older software to run on newer Windows, that's all.

Keep in mind that this issue only happens when Theta and Phi are very close to the same value, which is done by the specific LUA script seen above. I don't know how common this is in normal practice.

Besides all that, they even say that "spot lights are rather rarely used in games (if ever used at all). Furthermore if still used, probably nobody pays attention to such details". Plus, they are not even computing Range at all which would almost certainly make it look wrong with this game.

What if you try it yourself?

I don't have the Freelancer game so I cannot test it.

[elishacloud]

Setting AnisotropicFiltering to 2 unfortunately made no difference; the textures still didn't load.

I believe the texture is getting loaded. However, AnisotropicFiltering is blurring the texture so that you cannot see all the details.

When you say that the game was created without anisotropic filtering in mind, do you mean in general or specifically when converting Direct3D 8 to Direct3D 9?

I mean in general.

Because recently as a test I force-enabled 16x AF and 8x AA in Freelancer using the d3d8 wrapper, and that worked like a charm without any repercussions.

Ok, I can look at this and see what is happening. However, I would need an output of that frame to see all the DirectX calls in that frame and see what is happening. Can you send me a Microsoft PIX output file of that frame?

[mirh]

Of course I was talking with OP. Sounds like fun is ahead if somehow people are truly good with wined3d..

[BC46]

I tried running the game with various versions of WineD3D. With all of them the game just showed either a static black or white screen. The game did seem to be running in the background as it still played music and handled user inputs.