How does Embeddinator-4000 use CppSharp?

[jbielski]

Please help me understand how Embeddinator uses CppSharp. CppSharp allows you to use C/C++ code from .NET C# while Embeddinator allows you to use .NET C# from C. Why then, does Embeddinator rely on CppSharp's 'CheckTypeIgnore' to determine which C# code features it generates C bindings for? It seems there are so many features of C# that don't get bindings due to Ignoring that Embeddinator isn't very useful for real world code.

Steps to Reproduce

Expected Behavior

Actual Behavior

Environment

Build Logs

Example Project (If Possible)

[chamons]

So @jbielski the Embeddinator-4000 doesn't actually use CppSharp under the hood. You can look at https://github.com/mono/Embeddinator-4000/blob/master/objcgen/objcgenerator.cs for example to see the ObjC code generator.

It sounds like you would like attributes or some other mechanism to ignore some code during bindings (like https://github.com/mono/CppSharp/blob/master/docs/UsersManual.md#helper-defines)? I don't specifically see CheckTypeIgnore thought. If this is the case, please consider opening another issue or subscribing to https://github.com/mono/Embeddinator-4000/issues/481.

In general, since the managed->native transition is so expensive it is expected that except for the most non-trivial cases you will wrap your C# library in a a less-chatty API. See https://docs.microsoft.com/en-us/xamarin/tools/dotnet-embedding/objective-c/best-practices#exposing-a-chunkier-api for details.

Since this specific issue seemed to be a question, I'm going to close it for now. Please though feel free to post technical (how things work) questions our gitter https://gitter.im/managed-interop/Lobby or open specific issues that are actionable. In particular, examples of real-world code you are unable to bind would be very useful.

[jbielski]

Actually I DID post to the managed-interop/Lobby a few weeks ago (July 25 16:10) but no one responded to me; that's when I posted my question here. "By making code changes on the .NET side, I was able to get Embeddinator to generate C bindings and a *.dylib that compiled for my assemblies. However, when I run Embeddinator with the -v option, the 'CppSharp.Passes.CheckIgnoredDeclsPass' logs a whole bunch of Ignores, which then means I don't get bindings for a bunch of methods that I need access to. Is this normal? Is there a list of limitations for CppSharp? Is this a bug?"

[chamons]

I realized that my response was done with Apple colored glasses, I'm honestly not sure if the Android (Java/C) side uses CppSharp. @jonathanpeppers might know.

Reopening

[jonathanpeppers]

The usage of CppSharp, I believe, was mainly for reusing existing code. The framework/concepts of "code passes" came from there, as well as a few passes that could be reused.

@jbielski can you post an example of the warnings you are seeing? And and example of the C# code that is being skipped?

[jbielski]

@jonathanpeppers here are some examples and code: Function 'GetPlugin' was ignored due to ignored return decl Function 'GetSupportedPlugins' was ignored due to ignored return decl Property 'AvailablePlugins' was ignored due to ignored decl Function 'GetPlugin' was ignored due to ignored return decl Function 'GetSupportedPlugins' was ignored due to ignored return decl Property 'AvailablePlugins' was ignored due to ignored decl Function 'CreateInstance' was ignored due to ignored return decl ... and many, many, more. Below is the source file that contains the C# code. Note that this is from an GitHub repository called ADAPT/ADAPT which has a project of code that is widely used in the agriculture industry, which my firm is a part of, so it is pretty important for us. /***

• Copyright (C) 2015 AgGateway and ADAPT Contributors
• Copyright (C) 2015 Deere and Company
• All rights reserved. This program and the accompanying materials
• are made available under the terms of the Eclipse Public License v1.0
• which accompanies this distribution, and is available at
• http://www.eclipse.org/legal/epl-v10.html http://www.eclipse.org/legal/epl-v10.html
• Contributors:
• Tarak Reddy, Tim Shearouse - initial API and implementation ***/

using System; using System.Collections.Generic; using System.IO; using System.Linq; using AgGateway.ADAPT.ApplicationDataModel.ADM;

namespace AgGateway.ADAPT.PluginManager { public interface IPluginFactory { ///

/// Returns a list of all available plugin names. ///

List AvailablePlugins { get; }

  /// <summary>
  /// Given a plugin name, load and return an actual instance of that plugin. 
  /// </summary>
  /// <param name="pluginName">Name of the plugin to load. The plugin name should be
  /// contained in the AvailablePlugins list.</param>
  /// <returns>An instance of the given plugin's implementation of the IPlugin interface</returns>
  IPlugin GetPlugin(string pluginName);

  /// <summary>
  /// Find all plugins which are able to read a given data set. Relies on the IPlugin.IsDatacardSupported method.
  /// </summary>
  /// <param name="dataPath">The filesystem location of the data you would like to import.</param>
  /// <param name="properties">Optional key/value pair configuration object.</param>
  /// <returns>List of IPlugin implementations which are able to import the given data. 
  /// May return an empty list if no plugins are loaded, or if no plugin can read the given data location.</returns>
  List<IPlugin> GetSupportedPlugins(string dataPath, Properties properties = null);

}

public class PluginFactory : IPluginFactory { private readonly IFileSystem _fileSystem; private readonly string _pluginDirectory; private readonly IPluginLoader _pluginLoader; private readonly List _availablePlugins;

   public PluginFactory(string pluginDirectory)
       : this(new FileSystem(), pluginDirectory, new PluginLoader())
   {

   }

   public PluginFactory(IFileSystem fileSystem, string pluginDirectory, IPluginLoader pluginLoader)
   {
       _fileSystem = fileSystem;
       _pluginLoader = pluginLoader;
       _pluginDirectory = string.IsNullOrWhiteSpace(pluginDirectory)
           ? Directory.GetCurrentDirectory()
           : pluginDirectory;
       _pluginLoader.SetupDependencyResolver(_pluginDirectory);
       _availablePlugins = new List<PluginMetadata>();
   }

   public IPlugin GetPlugin(string name)
   {
       EnsurePluginsAreLoaded();
       var metaData = GetMetadata(name);
       return metaData.AssemblyInstance ?? (metaData.AssemblyInstance = _pluginLoader.CreateInstance(metaData));
   }

   public List<string> AvailablePlugins
   {
       get
       {
           EnsurePluginsAreLoaded();
           return _availablePlugins.Select(x => x.Name).ToList();
       }
   }

   public List<IPlugin> GetSupportedPlugins(string dataPath, Properties properties = null)
   {
       return AvailablePlugins.Select(GetPlugin)
           .Where(plugin => plugin.IsDataCardSupported(dataPath, properties)).ToList();
   }

   private PluginMetadata GetMetadata(string pluginName)
   {
       var metaData = _availablePlugins.FirstOrDefault(x => x.Name == pluginName);
       if (metaData == null)
       {
           throw new NotSupportedException("Plugin Not Found!");
       }
       return metaData;
   }

   private void EnsurePluginsAreLoaded()
  {
     if (_availablePlugins.Any())
        return;

     LoadPlugins();
  }

  private void LoadPlugins()
  {
     LoadPlugins(_pluginDirectory);
     foreach (var subDirectory in _fileSystem.GetSubDirectories(_pluginDirectory))
        LoadPlugins(subDirectory);
  }

  private void LoadPlugins(string directory)
  {
     var pluginDlls = _fileSystem.GetFiles(directory, "*.dll");
     foreach (var pluginDll in pluginDlls)
        LoadPlugin(pluginDll);
  }

  private void LoadPlugin(string assemblyLocation)
  {
     var pluginInfo = _pluginLoader.InspectAssembly(assemblyLocation);
     if (pluginInfo != null)
     {
        PopulateMetadata(pluginInfo);
     }
  }

  private void PopulateMetadata(PluginMetadata pluginInfo)
  {
     var existingPlugin = _availablePlugins
         .FirstOrDefault(p => p.Name == pluginInfo.Name);

     if (existingPlugin != null)
     {
        _availablePlugins.Remove(existingPlugin);
     }

     _availablePlugins.Add(pluginInfo);
  }

} }

[jonathanpeppers]

Hi @jbielski,

Let's just focus on one of them:

Function 'GetPlugin' was ignored due to ignored return decl

C# method:

public IPlugin GetPlugin(string name)

Looks like it is hitting this message: https://github.com/mono/CppSharp/blob/3dba1eb594a46f8e3ea6adb7f61d56db53a06ea0/src/Generator/Passes/CheckIgnoredDecls.cs#L165

Then if I keep following the code, the type is ignored? https://github.com/mono/CppSharp/blob/3dba1eb594a46f8e3ea6adb7f61d56db53a06ea0/src/Generator/Passes/CheckIgnoredDecls.cs#L503

Sot then I go here: https://github.com/mono/CppSharp/blob/3dba1eb594a46f8e3ea6adb7f61d56db53a06ea0/src/Generator/Types/TypeIgnoreChecker.cs

It looks to me that returning interfaces isn't supported. It's been a while for me, so I'm not sure I have even tried this on the Android side.

@chamons do you know if this is supported on the iOS/Obj-C side?

[jbielski]

I’m using C bindings, not Objective C, in case that matters.

[jonathanpeppers]

Yeah, @jbielski they are implemented differently.

I'm thinking we might have an item we need to post on the limitations docs.

[chamons]

I think so based on this unit test - https://github.com/mono/Embeddinator-4000/blob/master/tests/managed/interfaces.cs

[jbielski]

What is the limitation and is there a workaround?

[jonathanpeppers]

At first I thought the C code generator might not support interfaces, but there is a test for it: https://github.com/mono/Embeddinator-4000/blob/d697cdfca5060c07e95063c30f4864bd0a36b0ca/tests/common/Tests.C.cpp#L380-L391

I think you are likely hitting a bug.

The workaround I would say is to craft the API surface yourself, as @chamons suggested there are some pointers here: https://docs.microsoft.com/en-us/xamarin/tools/dotnet-embedding/objective-c/best-practices#exposing-a-chunkier-api

I would start by making a single static class of methods you need to call from C and expand from there.

[jbielski]

Ok, thank you @jonathanpeppers. I will try to do that. In the mean time, is someone going to continue root causing this and confirm that this is a bug (or not), what the bug is, and whether it will be fixed or become a permanent limitation? As this code is open source, I know that I can also try to identify the problem. Is there a way I can incorporate my code into Tests.C.cpp and see if I can find what's breaking? When I build the Tests target on my system, the Generate-Android task fails when 'Compiling binding code'. My system is Mac OS 10.13.6 so not sure why Android task is running to begin with:

Unhandled Exception: System.ArgumentNullException: Value cannot be null. Parameter name: path1 at System.IO.Path.Combine (System.String path1, System.String path2) [0x000a0] in /Users/builder/jenkins/workspace/build-package-osx-mono/2017-12/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.IO/Path.cs:126 at Embeddinator.Driver.CompileNDK (System.Collections.Generic.IEnumerable1[T] files) [0x00106] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:718 at Embeddinator.Driver.CompileNativeCode (System.Collections.Generic.IEnumerable1[T] files) [0x000ba] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:772 at Embeddinator.Driver.CompileCode () [0x000c6] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:54 at Embeddinator.Driver.Run () [0x000a5] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Driver.cs:253 at Embeddinator.CLI.Main (System.String[] args) [0x00092] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/CLI.cs:220 [ERROR] FATAL UNHANDLED EXCEPTION: System.ArgumentNullException: Value cannot be null. Parameter name: path1 at System.IO.Path.Combine (System.String path1, System.String path2) [0x000a0] in /Users/builder/jenkins/workspace/build-package-osx-mono/2017-12/external/bockbuild/builds/mono-x64/mcs/class/corlib/System.IO/Path.cs:126 at Embeddinator.Driver.CompileNDK (System.Collections.Generic.IEnumerable1[T] files) [0x00106] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:718 at Embeddinator.Driver.CompileNativeCode (System.Collections.Generic.IEnumerable1[T] files) [0x000ba] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:772 at Embeddinator.Driver.CompileCode () [0x000c6] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Compilation.cs:54 at Embeddinator.Driver.Run () [0x000a5] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/Driver.cs:253 at Embeddinator.CLI.Main (System.String[] args) [0x00092] in /Users/juliannebielski/Projects/Embeddinator-4000/binder/CLI.cs:220 An error occurred when executing task 'Generate-Android'. Error: One or more errors occurred. mono failed!

[jonathanpeppers]

@jbielski the above error likely means E4K can't find one of the following:

• Android NDK
• Android SDK
• Java

If you have all of these, we may need to update xamarin-android-tools, a git submodule we are using that finds these things.

[jbielski]

I didn't have the Android NDK installed. I installed the latest, version 17.x, from Visual Studio for Mac, but that resulted in issue 574 from this repository. I followed the workaround of back leveling to version r15c and then I was able to build the Tests. Should I try to add tests to TEST_CASE("Interfaces.C", "[C][Interfaces]") to exercise my Interface? Or should I start trying to figure out why CppSharp doesn't recognize that Interfaces are supported?

[tritao]

Hey @jbielski, when starting the project I decided to re-use some infrastructure that I already built for CppSharp, hence the usage of some things.

The main dependency is the AST model.

The other pass I think is only being used to make sure that if declarations have types that are ignored, then they themselves are flagged as ignored. This behavior is kinda generic, so I just re-used the code I had already built for CppSharp.

Some interface support for Java is there, as can be seen on the tests in https://github.com/mono/Embeddinator-4000/blob/master/tests/common/java/mono/embeddinator/Tests.java#L324.

However it seems you are hitting a bug. Some piece of code seems to be making this particular type as ignored.

If you want to help track down where, I suggest adding a conditional breakpoint in the setter of Declaration.GenerationKind, https://github.com/mono/CppSharp/blob/master/src/AST/Declaration.cs#L226.

Something along the lines of:

if (Name == "GetPlugin")
    System.Diagnostics.Debugger.Break();

This should break where the type is being set as ignored. Then we can figure out how to fix it.

[tritao]

Is this type on a different assembly by any chance?

I just realized this might be caused by that limitation of Embeddinator.

https://github.com/mono/Embeddinator-4000/blob/master/binder/Generators/AstGenerator.cs#L507

If that is the case, then this limitation was put on temporarily to bootstrap development of Embeddinator, but is still there. It needs to be removed. I guess some minor fixes need to be added to the codebase to be able to deal with it, especially on the C backend.

[jbielski]

I added the conditional breakpoint in Declaration.GenerationKind. The type is definitely being set to ignored, though I don't know why at this point.

[jbielski]

Admittedly, I don't understand this code very well, but it seems that line 504 is found to be 'true' in AstGenerator.cs, and that's when the interface method gets set to UnsupportedType.

    case TypeCode.Object:
        case TypeCode.DateTime:
            if (managedType.FullName == "System.Void")
            {
                type = new BuiltinType(PrimitiveType.Void);
                break;
            }
            var currentUnit = GetTranslationUnit(CurrentAssembly);
            if (managedType.Assembly != ManagedAssemblies[currentUnit]
                || managedType.IsGenericType)
            {
                type = new UnsupportedType { Description = managedType.FullName };
                break;
            }

At this point in time, currentUnit.value is this: {AgGateway.ADAPT.ApplicationDataModel, Version=1.0.0.1, Culture=neutral, PublicKeyToken=null}

and managedType.Assembly is this: {AgGateway.ADAPT.PluginManager, Version=1.0.0.1, Culture=neutral, PublicKeyToken=null}

This seems to point to your hunch above about the temporary limitation. I made this change: var currentUnit = GetTranslationUnit(CurrentAssembly); // if (managedType.Assembly != ManagedAssemblies[currentUnit] // || managedType.IsGenericType) if (managedType.IsGenericType) { type = new UnsupportedType { Description = managedType.FullName }; break; } Then re-ran E4K with the -v option and re-directed the output to the attached file. GetPlugin is no longer ignored, but there are still a ton of ignores and now there are compilation errors as well. verbose.txt

[jbielski]

Does E4K support binding C# collections in any way to C? I notice that any place there is a List<> or a Dictionary<>, even when the types are concrete, I'm getting "ignored" messages. So any method with collection parameters or return types and any property of a collection type is getting ignored.

[jbielski]

@jonathanpeppers another example of an ignored method: List GetSupportedPlugins(string dataPath, Properties properties = null); The message I received was: Function 'GetSupportedPlugins' was ignored due to ignored return decl And an example of an ignored property: List AvailablePlugins { get; } The message I received was: Property 'AvailablePlugins' was ignored due to ignored type

It's going to be hard to avoid collections, even in a reduced interface. I spent a few days trying to debug this myself without success. During the AST generation pass, somewhere between KVM reflection and assigning Cpp.AST.Type, the List type is being initialized or set to be ignored in the declaration context of the function and the property.

[jbielski]

I may have found the code causing Issue 688 and I wonder if you could verify. If you open up AstGenerator.cs and scroll down to line 143, there is an "if statement" that filters out generic types from being generated using the flag typeInfo.IsGenericType. If I remove that flag from the OR statement, then I get C bindings for my C# functions that include List<>s in the signature. The reason I questioned it is that above in line 130, IsGenericType is tested for, but is not filtered out, albeit it's an AND statement with another flag.