CppSharp.Types.Std.String.CSharpMarshalToNative fails with "Sequence contains no matching element" exception

[Ark-kun]

I'm trying to wrap the Microsoft/CNTK library under Windows using VS2017. CppSharp throws exception during code generation.

Version: v0.8.14 NuGet package

Code

            var options = driver.Options;            
            options.GeneratorKind = GeneratorKind.CSharp;
            var module = options.AddModule("CntkLibrary");
            module.IncludeDirs.Add(@"d:\CNTK\Source\Math\");
            module.IncludeDirs.Add(@"d:\CNTK\Source\Common\Include\");
            module.IncludeDirs.Add(@"d:\CNTK\Source\ComputationNetworkLib\");
            module.IncludeDirs.Add(@"d:\CNTK\Source\CNTKv2LibraryDll\");
            module.IncludeDirs.Add(@"d:\CNTK\Source\CNTKv2LibraryDll\API\");
            module.IncludeDirs.Add(@"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\include\");
            driver.ParserOptions.SetupMSVC(VisualStudioVersion.VS2017);

            module.IncludeDirs.Add(@"C:\Program Files (x86)\Windows Kits\10\Include\10.0.10586.0\shared\");
            module.IncludeDirs.Add(@"C:\Program Files (x86)\Windows Kits\10\Include\10.0.10586.0\um\");
            module.IncludeDirs.Add(@"C:\Program Files (x86)\Windows Kits\10\Include\10.0.10586.0\ucrt\");

            driver.ParserOptions.EnableRTTI = true;
            driver.ParserOptions.AddArguments("-fexceptions");
            driver.ParserOptions.AddArguments("-fcxx-exceptions");

            module.Headers.Add(@"CNTKLibrary.h");
            module.Headers.Add(@"PrimitiveOpType.h");

Stack trace

System.InvalidOperationException occurred
  HResult=0x80131509
  Message=Sequence contains no matching element
  Source=System.Core
  StackTrace:
   at System.Linq.Enumerable.First[TSource](IEnumerable`1 source, Func`2 predicate)
   at CppSharp.Types.Std.String.CSharpMarshalToNative(CSharpMarshalContext ctx)
   at CppSharp.Generators.CSharp.CSharpMarshalManagedToNativePrinter.VisitType(Type type, TypeQualifiers quals)
   at CppSharp.AST.AstVisitor.VisitTemplateSpecializationType(TemplateSpecializationType template, TypeQualifiers quals)
   at CppSharp.AST.TemplateSpecializationType.Visit[T](ITypeVisitor`1 visitor, TypeQualifiers quals)
   at CppSharp.Generators.CSharp.CSharpMarshalManagedToNativePrinter.VisitTypedefType(TypedefType typedef, TypeQualifiers quals)
   at CppSharp.AST.TypedefType.Visit[T](ITypeVisitor`1 visitor, TypeQualifiers quals)
   at CppSharp.AST.AstVisitor.VisitParameterDecl(Parameter parameter)
   at CppSharp.AST.Parameter.Visit[T](IDeclVisitor`1 visitor)
   at CppSharp.Generators.CSharp.CSharpSources.GenerateFieldSetter(Field field, Class class)
   at CppSharp.Generators.CSharp.CSharpSourcesExtensions.GenerateField(CSharpSources gen, Class class, Field field, Action`2 generate, Boolean isVoid)
   at CppSharp.Generators.CSharp.CSharpSources.GeneratePropertySetter[T](T decl, Class class, Boolean isAbstract, Property property)
   at CppSharp.Generators.CSharp.CSharpSources.GenerateProperties(Class class)
   at CppSharp.Generators.CSharp.CSharpSources.GenerateClassProperties(Class class)
   at CppSharp.Generators.CSharp.CSharpSources.VisitClassDecl(Class class)
   at CppSharp.AST.Class.Visit[T](IDeclVisitor`1 visitor)
   at CppSharp.Generators.CSharp.CSharpSources.VisitDeclContext(DeclarationContext context)
   at CppSharp.Generators.CodeGenerator.VisitNamespace(Namespace namespace)
   at CppSharp.Generators.CSharp.CSharpSources.VisitNamespace(Namespace namespace)
   at CppSharp.AST.Namespace.Visit[T](IDeclVisitor`1 visitor)
   at CppSharp.Generators.CSharp.CSharpSources.VisitDeclContext(DeclarationContext context)
   at CppSharp.Generators.CodeGenerator.VisitTranslationUnit(TranslationUnit unit)
   at CppSharp.AST.TranslationUnit.Visit[T](IDeclVisitor`1 visitor)
   at CppSharp.Generators.CSharp.CSharpSources.Process()
   at CppSharp.Generators.Generator.GenerateSingleTemplate(ICollection`1 outputs)
   at CppSharp.Generators.Generator.Generate()
   at CppSharp.Driver.GenerateCode()
   at CppSharp.ConsoleDriver.Run(ILibrary library)
   at Test.CppSharp.Program.Main(String[] args) in A:\_All\My.Personal.My Documents\Visual Studio 2017\Projects\Test.CppSharp\Test.CppSharp\Program.cs:line 55

[tritao]

Are you using master or the latest release?

[Ark-kun]

@tritao Sorry for not mentioning the version. I'm using the v0.8.14 NuGet package.

[ddobrev]

@Ark-kun you have a few errors in your configuration. Basically, do not add the system include dirs to those of your module, such as:

module.IncludeDirs.Add(@"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\include\");

Just specify MicrosoftMode = true and C++# will find them for you. You don't need to therefore call SetupMSVC yourself either. Having said that, this is still a bug - we shouldn't crash regardless of the user's settings. However, if you follow my advice, you'll most probably get past this error.

[Ark-kun]

@ddobrev I added them when I got errors about failing to find headers. With MicrosoftMode and no VS/Win includes the errors return:

CNTK\Source\CNTKv2LibraryDll\API\CNTKLibrary.h(10,10): fatal: 'memory' file not found

Adding back the VS dir I get the following:

Error parsing 'CNTKLibrary.h'
C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\VC\Tools\MSVC\14.11.25503\include\crtdefs.h(10,10): fatal: 'corecrt.h' file not found

I suspect that CNTK's C++ code is not the best, but it builds on Windows, Linux and MacOSX.

[tritao]

Looks like we might still have some bugs dealing with certain MSVC setups.

Can you paste the output of MSVCToolchain.DumpSdks()?

[Ark-kun]

@tritao

Here they are

Visual Studio SDKs: (11) C:\Program Files (x86)\Microsoft Visual Studio 11.0\Common7\IDE\ (12) C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\ (14) C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\ (15) C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE\ (15) C:\Program Files (x86)\Microsoft Visual Studio\Preview\Community\Common7\IDE (15) C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE

Windows SDKs: (7) C:\Program Files (x86)\Microsoft SDKs\Windows\v7.0A\ (7.1) C:\Program Files (x86)\Microsoft SDKs\Windows\v7.1A\ (8) C:\Program Files (x86)\Windows Kits\8.0\ (8) C:\Program Files (x86)\Microsoft SDKs\Windows\v8.0A\ (8.1) C:\Program Files (x86)\Windows Kits\8.1\ (8.1) (10) C:\Program Files (x86)\Windows Kits\10\

Windows Kits SDKs: (1) C:\Program Files (x86)\Windows Kits\8.1\ (8) C:\Program Files (x86)\Windows Kits\8.0\ (10) C:\Program Files (x86)\Windows Kits\10\

.NET Framework SDKs:

MSBuild SDKs: (2) C:\Windows\Microsoft.NET\Framework\v2.0.50727\ (3.5) C:\Windows\Microsoft.NET\Framework\v3.5\ (4) C:\Windows\Microsoft.NET\Framework\v4.0.30319\ (12) C:\Program Files (x86)\MSBuild\12.0\bin\ (14) C:\Program Files (x86)\MSBuild\14.0\bin\

[ddobrev]

@Ark-kun @tritao 's right. All system dirs should be detected by the automatic call to SetupMSVC. Could you please debug and see why you get the bug? It might have also been fixed in our latest code since we resolved a similar problem a few weeks ago.

[ispysoftware]

The nuget package urgently needs to be updated. It is completely broken on windows 10.

[ddobrev]

@ispysoftware I am sorry for your trouble. Usually we ask users to help with fixes but in this case the fix has been added in weeks and I just haven't released it which is my fault. I am going to update the NuGet tomorrow.

[Ark-kun]

@ddobrev I'd like to help fixing this, but I kind of do not know what I'm doing =) What I mean is that it's hard for me to discern the buggy values/behavior from correct ones.

Do you see any problems with my MSVCToolchain.DumpSdks() output?

Or is the problem somewhere between the SDK paths and include paths?

[ddobrev]

@Ark-kun I've just released the new code. The bug @ispysoftware 's mentioned should be fixed. About yours I cannot tell, please try the new version and if it fails for you, we can talk more about you helping.

[Ark-kun]

@ddobrev I've build CppSHarp from the latest source (51a109d2dba2b793521906faff8d2f32a59f9dfc) and StandardLib.Gen.exe fails to work:

......\CppSharp\build\vs2017\lib\Release_x86\StandardLib.Gen.exe

Generating bindings for StandardLib (CLI) Looking for tests in: A:......\CppSharp\tests\StandardLib Error parsing 'StandardLib.h' A:.......\CppSharp\tests\StandardLib\StandardLib.h(2,10): fatal: 'vector' file not found

CppSharp has encountered an error while parsing code.

I'm running it from under the VS 2017 dev command prompt.

Is this behavior expected? (Say, the env is somehow wrong.)

[ddobrev]

@Ark-kun definitely not expected. Please open the generated solution in Visual Studio and debug, SetupMSVC in particular. Your system headers are not found for some reason.

[Ark-kun]

@ddobrev I've found the cause of the problem. As you see in my MSVCToolchain.DumpSdks() output, 3 versions of VS15/VS2017 were found:

By GetVisualStudioSdks

(15) C:\Program Files (x86)\Microsoft Visual Studio 15.0\Common7\IDE

By GetVs2017Instances:

(15) C:\Program Files (x86)\Microsoft Visual Studio\Preview\Community\Common7\IDE (15) C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\Common7\IDE

The first one is not a real instance. It does not have any includes.

Yet, GetVSToolchain logic is to just get the first one that matches the version. https://github.com/mono/CppSharp/blob/2b9bf29a70b10281e0759b39e893ccb555fdeb0c/src/Core/Toolchains/MSVCToolchain.cs#L114

Then, in GetSystemIncludesVS2017, the function enumerates the instances and tries to get the one with the matching installation path and no instance matches. https://github.com/mono/CppSharp/blob/2b9bf29a70b10281e0759b39e893ccb555fdeb0c/src/Core/Toolchains/MSVCToolchain.cs#L625

My first fix would be to stop getting the VS2017 paths from the registry.

This won't fix the problem though as my VS2017 Preview might be higher in the list and it lacks VC installation. GetSystemIncludesVS2017 makes effort to look at the installed packages ("Microsoft.VisualStudio.Component.VC.Tools"), but that only happens after the VS instance is already chosen.

The VS package checks should probably happen during the VS instance selection.

I'll hack it for myself and think about a way to fix it generally.

[ddobrev]

Hmm, I guess we need to somehow validate the obtained VS SDK. If it's impossible, then we could store all SDK-s with the desired version instead of just the first one found, and then match them against the instances, something like a cross reference. It would be great if you could do it because it would be difficult to us as we would be forced to deliberately mess with our systems.

[tritao]

Hello @Ark-kun.

Do you know if this is still an issue with the latest version from master?