Is the generated DLL complete, or is more needed to call code (from Metatrader 4)?

[DennisGundersen]

I've made a test project, but my client (MetaTrader 4) just collapses on the first method call (DLL does load successfully), so there's no feedback on what's happening. I was wondering if maybe the .h files has to be included somehow, but importing those just adds more calls for additional inclusions of stddef.h and stdint.h. Maybe it's the JSON files? I have virtually no skills in C++, but it feels to me that this is some sort of C++ connecting issue. Anything pop out? Could be something really simple that "everyone knows, but me".

The csproj is:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup Label="Project">
    <TargetFramework>net7.0-windows</TargetFramework>
    <Nullable>enable</Nullable>
    <Platforms>x86</Platforms>
    <PlatformTarget>x86</PlatformTarget>
    <RuntimeIdentifier>win-x86</RuntimeIdentifier>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <OutputType>library</OutputType>
  </PropertyGroup>
  <PropertyGroup Label="DNNE">
    <DnneSelfContained_Experimental>true</DnneSelfContained_Experimental>
    <DnneAddGeneratedBinaryToProject>true</DnneAddGeneratedBinaryToProject>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>
  <PropertyGroup Label="Publish">
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <RollForward>LatestMinor</RollForward>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>
    <PublishTrimmed>false</PublishTrimmed>
    <DebugType>embedded</DebugType>
    <IncludeNativeLibrariesForSelfExtract>false</IncludeNativeLibrariesForSelfExtract>
    <ErrorOnDuplicatePublishOutputFiles>false</ErrorOnDuplicatePublishOutputFiles>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="DNNE" Version="2.0.1" />
  </ItemGroup>
</Project>

and the class is simply:

using System.Runtime.InteropServices;
namespace BridgeUsingDNNEForMT4
{
    public static class DNNEBridge
    {
        [UnmanagedCallersOnly(EntryPoint = "GimmeAnInt")]
        public static int GimmeAnInt(int a)
        {
            return a + 1;
        }
    }
}

MT4 uses a language called MQL, which is a subset of C++ (apparently), to run external code. This is my "expert adviser" (name for compiled programs in MT4):

//+------------------------------------------------------------------+
//|                                                     DNNETest.mq4 |
//|                                  Copyright 2023, MetaQuotes Ltd. |
//|                                             https://www.mql5.com |
//+------------------------------------------------------------------+
#property copyright "Copyright 2023, MetaQuotes Ltd."
#property link      "https://www.mql5.com"
#property version   "1.00"
#property strict

//#include <BridgeUsingDNNEForMT4NE.h>

#import "BridgeUsingDNNEForMT4NE.dll"
   int GimmeAnInt(int a);
#import

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   int value =  2;
   int result = GimmeAnInt(value);
   Print("GimmeAnInt(" + IntegerToString(value) + ") returned: " + IntegerToString(result));
   return(INIT_SUCCEEDED);
  }

TIA! Dennis

[AaronRobinsonMSFT]

TL;DR Update project to <DnneSelfContained_Experimental>false</DnneSelfContained_Experimental> or ideally remove that entirely. It is experimental and not worth fighting unless actually needed.

just collapses on the first method call (DLL does load successfully), so there's no feedback on what's happening.

@DennisGundersen Yep. DNNE calls C's abort if something it can't recover from happens. The FAQ has guidance on how to customize this.

Just looking at the project, I see two conflicting ideas.

    <DnneSelfContained_Experimental>true</DnneSelfContained_Experimental>
...
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>

Asking DNNE to try self-contained, but then telling the .NET SDK to not use self-contained is going to be very confusing. My first pass would avoid self-contained entirely and get the typical framework dependent working. After that you can experiment with more advanced features.

[DennisGundersen]

Asking DNNE to try self-contained, but then telling the .NET SDK to not use self-contained is going to be very confusing. My first pass would avoid self-contained entirely and get the typical framework dependent working. After that you can experiment with more advanced features.

Okay, I think I get that... The original props were:

   <DnneSelfContained_Experimental>true</DnneSelfContained_Experimental>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>true</SelfContained>
    <PublishTrimmed>true</PublishTrimmed>

That gives me projectNE.dll + plus framework files, but it crashes

Changing it to:

    <DnneSelfContained_Experimental>false</DnneSelfContained_Experimental>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>
    <PublishTrimmed>true</PublishTrimmed>

fails on publish due to "Optimizing assemblies for size is not supported for the selected publish configuration. Please ensure that you are publishing a self-contained app."

I therefore end up with:

  <PropertyGroup Label="DNNE">
    <DnneSelfContained_Experimental>false</DnneSelfContained_Experimental>
    <DnneAddGeneratedBinaryToProject>true</DnneAddGeneratedBinaryToProject>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>
  <PropertyGroup Label="Publish">
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <RollForward>LatestMinor</RollForward>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>
    <PublishTrimmed>false</PublishTrimmed>
    <DebugType>embedded</DebugType>
    <IncludeNativeLibrariesForSelfExtract>false</IncludeNativeLibrariesForSelfExtract>
    <ErrorOnDuplicatePublishOutputFiles>false</ErrorOnDuplicatePublishOutputFiles>
  </PropertyGroup>

which gives me my projectNE.dll, no framework files, but also crashes.

I've also tried to update the export decorations to: [UnmanagedCallersOnly(CallConvs = new[] { typeof(System.Runtime.CompilerServices.CallConvCdecl) })] in case MT4 doesn't know what an int is, but it doesn't seem to make a difference.

Am I still doing DNNE wrong, or is it MT4?

Re Dennis

[AaronRobinsonMSFT]

@DennisGundersen Could be a few things. I assume you are building the DNNE assembly in Debug mode?

Is it possible to debug the MQL? If so, I would "step into" on the line below and you can see where the code is failing. Keep stepping in until you get to prepare_runtime() where you can likely narrow down the underlying issue.

//+------------------------------------------------------------------+
//| Expert initialization function                                   |
//+------------------------------------------------------------------+
int OnInit()
  {
   int value =  2;
   int result = GimmeAnInt(value); <----- Step in here.
   Print("GimmeAnInt(" + IntegerToString(value) + ") returned: " + IntegerToString(result));
   return(INIT_SUCCEEDED);
  }

[DennisGundersen]

Is it possible to debug the MQL? If so, I would "step into" on the line below and you can see where the code is failing. Keep stepping in until you get to [prepare_runtime()]

Unfortunately, not for external code, only inside the MQL itself. And it only has one error code. Programming is fun, isn't it?

Re Dennis

[AaronRobinsonMSFT]

Unfortunately, not for external code, only inside the MQL itself. And it only has one error code. Programming is fun, isn't it?

Sad. Let's try something else then.

1) Use ProcMon to see what files are being examined. This plugs into the file system so you should be able to see all file system access for the process in question.

2) You can override the abort() call. This is described in the FAQs. In your overridden version of abort(), you can trigger a breakpoint using DebugBreak() or even create a simple loop that will spin until you can attach a debugger like Visual Studio or WinDBG. From there, you should be able to see the stack and perhaps the error code looking up the stack.

[ThaDaVos]

@DennisGundersen - if I remember correctly, you're using my template, right?

@AaronRobinsonMSFT - if he's using my template - it's functional with the following:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup Label="Project">
    <TargetFramework>net7.0-windows</TargetFramework>
    <Nullable>enable</Nullable>
    <Platforms>x86</Platforms>
    <PlatformTarget>x86</PlatformTarget>
    <RuntimeIdentifier>win-x86</RuntimeIdentifier>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <OutputType>library</OutputType>
  </PropertyGroup>
  <PropertyGroup Label="DNNE">
    <!-- EXPERIMENTAL: The native hosting should assume it is in a self-contained scenario. 
         When
    setting this flag to true, the only change in output will be the generated hosting 
         layer
    will call an API that will permit self-contained runtime activation. In order for this
        option
    to work as desired the user must have manually configured the assembly's
        runtimeconfig.json
    file and deps.json file. The user must also copy the
        appropriate runtime binaries to create a
    valid self-contained install
        environment. -->
    <DnneSelfContained_Experimental>true</DnneSelfContained_Experimental>
    <DnneAddGeneratedBinaryToProject>true</DnneAddGeneratedBinaryToProject>
    <DnneAdditionalGenerators>Clarion*</DnneAdditionalGenerators>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <InvariantGlobalization>true</InvariantGlobalization>
  </PropertyGroup>
  <PropertyGroup Label="Publish">
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <RollForward>LatestMinor</RollForward>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>
    <PublishTrimmed>false</PublishTrimmed>
    <DebugType>embedded</DebugType>
    <IncludeNativeLibrariesForSelfExtract>false</IncludeNativeLibrariesForSelfExtract>
    <ErrorOnDuplicatePublishOutputFiles>false</ErrorOnDuplicatePublishOutputFiles>
  </PropertyGroup>
  <ItemGroup>
    <PackageReference Include="DNNE" Version="2.0.1-patched-8502.28389" />
  </ItemGroup>
</Project>

As you can see, I set DnneSelfContained_Experimental to true and SelfContained to false - and the DLL works for me in clarion - but just to make sure, @DennisGundersen - how do you publish the project? If you're using Visual Studio - make sure your publish profile has SelfContained turned of too - so it looks like the following:

Ignore the Target Location as it's my Clarion Testing project

[ThaDaVos]

This is my test project: https://github.com/thadavos/experimental-calculator

[DennisGundersen]

@DennisGundersen - if I remember correctly, you're using my template, right?

Yes, I'm using the @ThaDaVos template, and I have the same publishing setup. So I guess that means it must be a Metatrader issue then? I've asked an actual programmer (who knows MT4) to help me, so hopefully there will be some clarifications, or at least better questions soon.

Re Dennis

[ThaDaVos]

Something cool I just figured out - you can remove the:

<Platforms>x86</Platforms>
<PlatformTarget>x86</PlatformTarget>
<RuntimeIdentifier>win-x86</RuntimeIdentifier>

And use only

<DnneRuntimeIdentifier>win-x86</DnneRuntimeIdentifier>

This will make your Library DLL be AnyCPU but the DNNE native library be targeted to win-x86 - it works for me in Clarion - also, it seems to be easier to deal with

[DennisGundersen]

<DnneRuntimeIdentifier>win-x86</DnneRuntimeIdentifier>

This will make your Library DLL be AnyCPU but the DNNE native library be targeted to win-x86 - it works for me in Clarion - also, it seems to be easier to deal with

MT4 doesn't work with AnyCPU, only x86, but thanks for the suggestion. I'm also thinking that Framework-dependent might be a mistake for MT4. With the .NET Framework 4.8 version of this project, I had to dump some .NET assemblies for Entity Framework, NewtonSoft etc. and of course my extra libraries, into the MT4 installation folder or they wouldn't be found, while it didn't need System.dll and that basic stuff. Maybe this has all changed due to new architecture with Core, due to no more GAC maybe? Aaarg!!

I'm still looking for someone to help me trace the call and figure out where it breaks. I'll fill in anything I find.

Re Dennis

[ThaDaVos]

MT4 will be using the x86 NE dll - the actual project DLL will be AnyCPU.

About the dependencies - this, as far as I know, has always been this way - as the default DotNet install does not contain Nuget Package dll's and stuff.

One thing you could do - which is also what I am doing - is use Fody and Fody.ILMerge to merge the dependencies into your DotNet dll - that way you only have two files, the Native DLL and the DotNet DLL

[ThaDaVos]

Did you include your runtime.config.json in the directory of MetaTrader? If yes, can you share the contents?

[DennisGundersen]

About the dependencies - this, as far as I know, has always been this way - as the default DotNet install does not contain Nuget Package dll's and stuff.

Yes, that sounds right, I was just musing that things might have changed with .NET Core and hence it could open up even more venues for possible errors.

I have six suggested error sources right now, but of course I'm just pulling them out of my butt for a hail Mary... Potential bugs:

1. MT4 doesn't find the C# method (naming?)
2. Marshalling (not blittable even for int only?)
3. MT4 can't read the installed .NET libraries?
4. Needs "Self-contained" instead of "Framework-dependent"?
5. Can't read files in installation folder either?
6. Some other file placement issues?

Re D

MT4Bridge.runtimeconfig.json.txt

PS! I'm getting a feeling that Clarion is significantly less finicky than MT4...

[ThaDaVos]

Gonna try to answer your list:

1. Can you share code of both sides? (Maybe put the project on GitHub so we can help better)
2. What are you trying to Marshall? DNNE is setup in a way that that's not necassary
3. When deploying the generated DLL based on my template - you need to have DotNet Runtime installed for the version you used - that's what Framework Dependent means - I haven't tried to create a single file library yet - may try eventually when I get a good example working
4. Why needs Self-Contained?
5. Depends on how they are placed and which file you copied - can you show a list of the files you copied?
6. Maybe

PS: Let's not start about Clarion - it used custom formats for almost everything 🤣

[DennisGundersen]

Gonna try to answer your list:

1. Can you share code of both sides? (Maybe put the project on GitHub so we can help better)
2. What are you trying to Marshall? DNNE is setup in a way that that's not necassary
3. When deploying the generated DLL based on my template - you need to have DotNet Runtime installed for the version you used - that's what Framework Dependent means - I haven't tried to create a single file library yet - may try eventually when I get a good example working
4. Why needs Self-Contained?
5. Depends on how they are placed and which file you copied - can you show a list of the files you copied?
6. Maybe

Sorry for the slow reply, I had to learn Git. I have now created a repository and tried to fill it out with the info needed about both programming in MT4 and including DNNE (as far as I've been able to gather it so far). Hadn't really thought about publishing anything in my own name, but with some info about Git, I see how this will enable easier collaboration without messing up too much for Aaron. The repository is at https://github.com/DennisGundersen/MT4_To_CSharp_Bridge.

Re Dennis

[AaronRobinsonMSFT]

@DennisGundersen Thanks for the repo and example. Is there any possibility we can get free access to MT4 so we can run this locally?

Were you able to try using any of the approaches I mentioned at https://github.com/AaronRobinsonMSFT/DNNE/issues/165#issuecomment-1499670382?

I assume you've proved out that the 32-bit DNNE scenarios work on your machine outside of the MT4 environment.

[DennisGundersen]

@DennisGundersen Thanks for the repo and example. Is there any possibility we can get free access to MT4 so we can run this locally?

There are massive amounts of Forex brokers that offer free demo accounts with MT4 (downloadable) for training and development, such as https://roboforex.com/. (MT4 is the locally installed client app, and the brokers run the servers which receive the market data and place orders on your behalf).

Were you able to try using any of the approaches I mentioned at #165 (comment)?

I assume you've proved out that the 32-bit DNNE scenarios work on your machine outside of the MT4 environment.

I'm still looking to hire an actual developer over at UpWork.com (https://www.upwork.com/jobs/~0161db045c3c0e71a3) to help me with this part as NrSlionHeart hasn't gotten back to me in weeks, but since ThaDaVos offered to help if I set up a repository, I thought I might as well get a jump start on it. I'm really not expecting you to spend your time on my obscure situation though. I'm sure I'll find someone who can implement your suggestions and then ask cogent questions afterwards if needed.

Re Dennis

[ThaDaVos]

@DennisGundersen - I see in your repo that SelfContained is set to true - as far as I know this should be false My current .csproj looks as follows:

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup Label="Project">
    <TargetFramework>net7.0</TargetFramework>
    <Nullable>enable</Nullable>
    <EnableDynamicLoading>true</EnableDynamicLoading>
    <OutputType>library</OutputType>
    <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>
    <InvariantGlobalization>true</InvariantGlobalization>
    <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
  </PropertyGroup>
  <Target Name="Change DNNE Settings" BeforeTargets="DnneGenerateNativeExports">
    <PropertyGroup Label="DNNE">
      <DnneRuntimeIdentifier>win-x86</DnneRuntimeIdentifier>
      <DnneSelfContained_Experimental>true</DnneSelfContained_Experimental>
      <DnneAddGeneratedBinaryToProject>true</DnneAddGeneratedBinaryToProject>
    </PropertyGroup>
    <Message Text="Configuring DNNE using a task" />
  </Target>
  <PropertyGroup Label="Publish">
    <GenerateRuntimeConfigurationFiles>true</GenerateRuntimeConfigurationFiles>
    <RollForward>LatestMinor</RollForward>
    <PublishSingleFile>false</PublishSingleFile>
    <SelfContained>false</SelfContained>
    <PublishTrimmed>false</PublishTrimmed>
    <DebugType>embedded</DebugType>
  </PropertyGroup>
   <!-- PackageReferences etc -->
</Project>

You can leave the Target out - and just use the PropertyGroup inside as you do now - I was using a Target as I also modified my local DNNE for some extra functionality's like auto-generation of wrapping code in Clarion - see: https://github.com/AaronRobinsonMSFT/DNNE/issues/162

[DennisGundersen]

@DennisGundersen - I see in your repo that SelfContained is set to true - as far as I know this should be false My current .csproj looks as follows:

I've tried both, but that's not where the problem is apparently. Now I'm trying to setup a C++ project in VS and load either the C# project or the dll, but I can't find any good info on that either. All examples are the other way. Still haven't found a C++ developer to take on the project. And the weeks go by...

Re Dennis

[AaronRobinsonMSFT]

@DennisGundersen You should be able to reasonably generate or load the following into Visual Studio - https://github.com/AaronRobinsonMSFT/DNNE/tree/master/test/ImportingProcess. From there you can likely edit what is needed for your scenario. Let me know if you need some help editing the existing main().

[DennisGundersen]

Yeah, so... The thing is... It turns out that if you include both your own code (ProjectName.dll), DNNE (ProjectNameNE.dll) AND the .NET configuration (ProjectName.runtimeconfig.json) in the /MQL/Library folder, it works a lot better. Sorry about that. Complete prototype for MT4 to C# is now available at https://github.com/DennisGundersen/MT4_To_CSharp_Bridge.

Re Dennis

[AaronRobinsonMSFT]

Complete prototype for MT4 to C# is now available at

@DennisGundersen That is great to see! Thanks. I assume you are unblocked any everything is working as expected? If not let us know and we can reopen the issue.