Design errors prevent other project system from using this Managed project system

[RobertvanderHulst]

I have attempted to create a new project system, as a sibling for the C#, VB and F# project system (the same thing the managed c++ or Python team need to do when they want to use CPS). My experience is that the current design of the project system is locked to the 3 mentioned systems and other systems cannot use the managed project system code for the following reason:

• Several key types are marked as "internal"
• The C#, VB and F# system can access these types because they are listed in "InternalsVisibleTo"
• Other project systems are not listed and can't access these types
• Furthermore on various places the C#, VB and F# languages are explicitly listed as capabilities. That means that other systems must either pretend to be C#, VB or F#. Otherwise it will not work

I tried an alternative approach: recompile the managed project system as well and include my own project system in the list of assemblies that can access the internals. That works, but then I will have an assembly with the same name and same (but modified) types as the assembly used by the C# project system and that will most certainly create problems when our project system is running side by side with C#. Renaming the managed project system assembly is not an option, because it uses other assemblies (for example a languageservice assembly) with the same design flaw: its internals are exposed to the Managed project system. So renaming the DLL will make the internals of that assembly unacessible.

The only solution that I can think of is to keep the name of the managed project system DLL and rename all the namespaces in it, so they don't conflict with the namespaces that the C# project system uses. But that would mean that we have to touch each and every source file to fix a design error.

I am really surprised that you have decided to use InternalsVisibleTo and that you not simply make the types that you needed in the C# or VB system public. I hope it is not too late to correct this.

Robert

[natidea]

@RobertvanderHulst thanks for your query. Our long term intent is to create a generalized project system, with reusable components and services that are public. Our current implementation came together fairly quickly under some time pressures to provide tooling in time for the release of .NET Core 1.0. Hence we scoped the project to C# and VB (and later F#) .NET Core projects only.

We have already started planning to generalize the package management and dependency tree implementations (see https://github.com/dotnet/project-system/issues/2491 and https://github.com/dotnet/project-system/issues/1984). A few partner teams need to use those features for NuGet package management in their project systems. I have a prototype implementation for NuGetizer Project System which is described in https://github.com/NuGet/Home/issues/5868#issuecomment-329283346.

I think the next step would be to identify specific components and services that you would like to reuse, and file specific work items to investigate what it would take to generalize them.

[RobertvanderHulst]

@natidea, I can list all the types that I need, but I am afraid that I will forget some. By far the easiest solution is this: Our customers expect that our system can do the same as the C#, VB and F# project systems. So if remove the InternalsVisibleTo for the Microsoft.VisualStudio.ProjectSystem.CSharp (and VB/FSharp) as well as Microsoft.VisualStudio.ProjectSystem.CSharp.VS (and VB.VS and FSharp.VS). and their test assemblies. Then try to build the C#, VB an F# systems, and change the code in Microsoft.VisualStudio.ProjectSystem.Managed and Microsoft.VisualStudio.ProjectSystem.Managed .VS so these project systems will compile. If that works, then we can do the same with our project system.

The other change is the capabilities, but that is less urgent. We can always "tell" our project system to announce the CSharp or VB capability and that should work. Am I correct ?

[davkean]

@RobertvanderHulst Not all of that is applicable to every managed project system, other languages don't use the Roslyn workspace - yet lots of code in this repo is spent manipulating that. We have no plans on opening those APIs to other project systems because the Roslyn workspace isn't open ended.

Specifics would really help - lots of code in this repo is also just helpers that aren't supposed/designed to be public API.

[RobertvanderHulst]

@davkean. If you remove the InternalsVisibleTo for the FSharp projects from the Managed.csproj and Managed.Vs.csproj and their test projects you will see that at least the following types need to be exposed:

• BuildOptions
• ProjectDesignerPageMetadata
• ProjectImageKey
• ProjectCapability
• IProjectImageProvider
• IParseBuildOptions
• ClassRegistrationAttribute
• IProjectCapabilitiesService
• ManagedProjectSystemPackage
• DplOptOutRegistrationAttribute
• Order
• IServiceProviderExtensions
• IProjectCapabilitiesServiceFactory

When you change the visibility of these types from internal to public then the F# projects will compile without relying on the internalsvisibleto attribute. Our project system will then compile as well.

[natidea]

/cc @DustinCampbell who thinks about architecture for developer experiences team, and @richaverma1 who is interested in customer use cases.

Another detail that I got from meeting with Robert is that XSharp uses a fork of Roslyn as its compiler, so it can interact with project system in much the same way C# and VB does.

[RobertvanderHulst]

@natidea That is right. Our language is built on top of Roslyn. Our (Antlr) parser generates a parse tree which is transformed in a Roslyn Parse tree. Then we let Roslyn do the 'dirty' work such as binding and code generation. We have made minimal changes to the Roslyn code to support some special features of our language, such as case insensitivity and different ways of handling method resolution where the compiler finds 2 or more possible candidates for a method. We have not used the Workspace model (yet) because it does not fit in the integration with the MPF based project system that we have used so far.

[RobertvanderHulst]

Is there any news on this ?

[natidea]

/cc @Pilchie

[RobertvanderHulst]

Is there any news on this ? I am getting a little impatient as you may understand. This is literally a few hours work. I have even offered to do the work for you . That offer still stands.

[davkean]

@RobertvanderHulst We have marked our MEF components with a new capability called .NET- https://github.com/dotnet/project-system/pull/3299. If you opt into this capability for your project type (along with associated capabilities you can see here: https://github.com/dotnet/project-system/blob/master/src/Microsoft.VisualStudio.ProjectSystem.Managed/ProjectSystem/ProjectCapability.cs#L8), you will inherit ".NET" behavior and associated components. You should not opt into 'CSharp' or 'VB' capabilities, unless you are a C# or VB project.

At it stands, we have no intention of making the above APIs public that you call out above. We consider these private implementation details of this code base and are free to, and will, change version to version. Visual Studio is a platform; we make very deliberate and explicit choices about APIs that we make public. We are still supporting APIs that we added in the first version of Visual Studio.NET back in 2002. This project is under the Apache, you are free to copy it into your own project if would like.

We're willing to discuss APIs on a case-by-case basis (ie we need an API to do X), and please file those as separate bugs, but we're unwilling to making every type public across the code base.

[RobertvanderHulst]

David, Thanks for the new capability. That helps. Our project system and language (X#) is more like F# than C# or VB. I will create a clone of the F# project system and will transform that into a X# project system. I am sure that I will find APIs then that are used by F# and are not visible to us.

I will post a new issue or new issues to open up these APIs. Do you want separate issues for each of the APIs or should I create one issue and include a list of the APIs that we need to be opened up?

Robert

[davkean]

I will create a clone of the F# project system and will transform that into a X# project system. I am sure that I will find APIs then that are used by F# and are not visible to us. I am sure that I will find APIs then that are used by F# and are not visible to us.

Please do not do that, we'll be back having the same conversations. Start a project system from scratch: https://github.com/Microsoft/VSProjectSystem, if you see pieces from our tree that you would like to reuse, copy the code into your tree. We share code between these dlls only due to our own efficiency reasons. We are not a public API for all project systems, the APIs that are supported are available to all project systems are called out in https://github.com/Microsoft/VSProjectSystem.

[RobertvanderHulst]

David, So what you are basically are saying is that we should not use dotnet/project-system code but build on top of CPS directly. And then copy bits and pieces of your code over into our own project system. That is what we tried to avoid. Our customers use not only X# but also C# and/or VB. And they compare our project system with the C# and VB project system. By inheriting from your common code we wanted to make sure that our code has the same features that the C# and VB project systems have.

If that is not possible then that is very unfortunate.

Robert

[davkean]

We provide components that opt into C#/VB/F# "behavior" that you can opt into via the .NET capability and the other capabilities. Please try that first, and then we can discuss the APIs. I believe you are missing context, and hence we're talking past each other.