search

mono / CppSharp

Tools and libraries to glue C/C++ APIs to high-level languages
MIT License
3.12k stars 513 forks source link

[Proposal] Revamping the the nuget package #1147

Closed damageboy closed 4 years ago

damageboy commented 5 years ago

If I understand correctly the current nuget packaging solution seems to be outdated and cannot support multiple operating systems from a single user install-able package.

I might be completely out of my way in suggesting this, but I would appreciate some feedback re. the following idea: Would it not make sense to provide a nuget package that is based on newer nuget standards that would:

If I am not mistaken, this sort of an architecture would allow CppSharp to provide cross-platform nuget packages (like https://github.com/libgit2/libgit2sharp/ does for example) for users to be able to run code-generators in multiple OS/arch combinations using a single binary distribution.

Ultimately, if this sort of approach is feasible, this could also become the output of a new, cross-platform CI solution that can build cross-platform nuget packages in the future.

I don't this my proposal is anywhere near complete at this stage, and there are many more small caveats that need to be discussed like:

But before going into that, I would like to understand if this sort of proposal be a direction the project should go for? Is the end goal something that is desirable?

tritao commented 5 years ago

Hey @damageboy, thanks for your suggestions.

The NuGet packages definitely need improvements, and I think the ideas you suggested all make sense for easier adoption of the project.

To make this a reality, we need some way to gather the outputs of CI artifacts, and package them all at the same time. The easiest way to achieve this that comes to mind is extending the CI that we already have to package and publish the Nuget.

We can have a different CI path for branches that are tagged, that downloads the artifact from Travis/AppVeyor and builds and publishes the Nuget packages automatically.

For packaging we could use Cake, I used it in Embeddinator project to do this and it works pretty well: https://github.com/mono/Embeddinator-4000/blob/master/build/Packaging.cake#L17 Using Python or Lua scripts to do this should also work.

I think it would be ok for now to start with .NET Framework support, and we can look into proper .NET Core support once that works.

We should also allow CppSharp.CLI as a Nuget global tool.

What do you think?

damageboy commented 5 years ago

I've recently given azure pipelines a go, and it actually looks very good...

Here's an example of a pipelines file: https://github.com/damageboy/daemaged.compression.native/blob/master/azure-pipelines.yml

And a resulting build which build both 6 variations of linux + windows for 6 native libraries (which I'm trying to use CppSharp to generate C# wrappers for).

Here's a sample build on Azure pipelines (I'm still missing Mac OSX builds, but those are too supported on that platform): https://dev.azure.com/damageboy/daemaged.compression/_build/results?buildId=57

The added value of this is that the entire build can be built in parallel on all OS, and then a single fan-in configuration can pick up ALL of the artifacts and package them with a script into a single nuget.

The reason I'm tying this (in my mind) to .NET Core, is that fact that I'm not too familiar with making multi-arch nuget packages before the .NET Core era, where in .NET Core I know I can officially provide that sort of package using RIDs and Architecture specific folders.

Would you be OK with giving Azure Pipelines a go? Do you think its possible to make a single nuget package, still targeting .NET Framework while trying to support users both on Linux + Windws + macOS?

The last part about the nuget package is what is worrying me a bit, since I don't have a good example of how this is done in the old framework (multi arch support)...

damageboy commented 5 years ago

@tritao Any comment? Would you be OK with me trying to make it work in Azure pipelines? Do you have any idea if this can be done with older nuget?

tritao commented 5 years ago

Hey @damageboy, sorry for delay in getting back to you.

I am fine with us supporting Azure Pipelines CI as well as the architecture you suggest for the buillds.

I think its possible to do it via DLL maps on Mono (https://www.mono-project.com/docs/advanced/pinvoke/dllmap/).

We can probably package all the needed files in a single package and use the maps to force usage of the right DLLs on Linux and macOS.

So we'd have for the native libs:

And then for the managed parser bindings, we would need a DLL per platform as well.

And we would try map to the right DLL in macOS and Linux using the DLL maps.

damageboy commented 5 years ago

Hi, sorry for the delay on my side...

So dll-maps would allow for mono specifically to find the right native-dll. I get that, it sounds workable on that front.

As for the managed bindings... It's obviously possible to do CppSharp.[OS].dll (I think you mistakenly added a lib prefix in the second bullet list), but if I'm not mistaken this is not so workable from the perspective of the ultimate nuget-user's end, no?

From what I understand, code-generation projects, depending on CppSharp need to always reference a "CppSharp.Parser.dll" assembly/nuget package, which should somehow map to a different generated CppSharp.Parser.dll on every supported OS/Arch.

How would the user code resolve to the CppSharp.Parser.macOS.dll in your example?

I found a "good" lead, which is how libgit2sharp does this, but it does not exactly map to cppsharp cleanly.

In libgit2sharp, it seems they split the entire native binaries into its own separate nuget: https://www.nuget.org/packages/LibGit2Sharp.NativeBinaries/

In that nuget they do provide the macosx/linux dll-map config, which they use build time props files to copy to the output directory. That's all fine and good.

But when you go the the managed library "wrapping" the native one: https://www.nuget.org/packages/LibGit2Sharp

They only have one managed DLL (e.g. only one CppSharp.Parser.dll) that happens to call and map the same functions from the native binary.

In theory you could make CppSharp also do the same for the managed part from the nuget. e.g. copy at runtime, per OS, the managed dll to the output folder, overwriting the default one the user compiled against, but this seems very hacky.

To get things rolling, does CppSharp.Parser.dll have the same public API signatures for all OS/Arch combinations?

feliwir commented 4 years ago

Any news on this?

tritao commented 4 years ago

@feliwir Not on my side.

@damageboy Sorry I didn't answer your questions earlier, if you'd still like to contribute and help on this I'm happy to help and come up with the best way to move forward on this.

damageboy commented 4 years ago

Hi all, I've given it some more thought, and I don't think it's right for me to pursue this work.

I'd love for this to happen, but at the end of the day, I'm no more than an opinionated user, I can't directly commit/make decisions on my own regarding the direction of the project, build system and how/what gets packaged.

I'd end up making decisions the risk breaking people's workflow, without really being responsible for the outcome, or being able to fix it, as this is not my full time dedicated job.

It also seems that the core developers are swamped with work and don't necessarily have enough time to respond keep track of all the user requests/issues.

Please don't think for a second I'm thinking a single bad thought about the amazing work done here by @tritao and @ddobrev. I see them as nothing short of being heroes.

This is the sort of decision making and work that cannot proceed without the full attention of the core developers, and I don't see this realistically happening.