Please Consider Improving Project Format and Structure (Serialized POCO/Format Agnostic)

[Mike-E-angelo]

Going to try this again, but hopefully with a better defined ask. The request is not to support a particular format, but to improve the MSBuild system overall so that it can support any format, while also making it more accessible/discoverable by tooling and (possible) designers.

Based on the current climate of .NET Core having its project.json merged back into MSBuild, this topic might get a little more interest conversation. Also, the Roslyn team is sending developers your way, so, there's that. :)

Use Case: Developers REALLY like their file formats!

Problem(s):

1. MSBuild format is rooted in an arbitrary, not well-defined/known schema (difficult to explore and discover).
2. MSBuild project structure can only be defined/edited in XML.
3. MSFT development ecosystem (for better or for worse) is now comprised of two camps: web application developers and native application developers each has their own way of doing things and neither really likes the approach of the other.

Suggestions:

1. Create a well-defined, well-known .NET object model that defines the Project Model (Project API). Visual Studio then loads (and saves) these as POCOs on disk. In this design, there are no arbitrary schemas or data files, but 100% serialized POCOs which are read and saved to disk. Another issue on Roslyn's board goes into more detail around this.
2. Adopt a "Bring Your Own Serializer" strategy. Adding a new format/type is as simple as installing a Visual Studio extension (or even auto-detected and installed for you upon detection/initial file load). To start with, JSON5 and XML should be supported out of the box, but developers should be able to bring on any format they wish, such as Xaml, Yaml, etc.
3. Allow (optional) naming extensions for projects to help identify stored format. Examples:
   • MyProject.csproj.json5 <-- C# Project serialized as JSON5
   • MyProject.vbproj.xml <-- VB.NET Project serialized as XML
   • MyProject.fsproj.xaml <-- F# Project serialized as XAML (OH YES I WENT THERE!!!)

Those are off the top of my head to get the conversation started. All of these are open to feedback of course and I would love to hear the thoughts of developers around this. Thank you for any consideration and/or support!

[Mike-E-angelo]

MSBuild should be an option for those needing it's power and extensibility.

Really liking this approach! At the very least we need to have more of an options-based solution where you pick your own builder. We have this for pretty much everything else in the CI/CD toolchain... but build is something that is assumed is a particular tech, precisely how you describe here.

Also digging @RichiCoder1's suggestion:

Make it trivial to shell out to something like .csx or other scripting tools for easy extension

Is that PowerShell? There definitely needs to be better integration with that as well. (and also, PowerShell should be more like C#, but that's another discussion. #shamelessplug :stuck_out_tongue: ).

[migueldeicaza]

"Ancient legacy tech" like msbuild is usually well tested, covers many cases you are blissfully unaware of (thorough battle tested scenarios), has good integration with many other components and tools.

On a world where .NET is truly a universal platform: all devices, all application types, you need msbuild.

Here are some "ancient and legacy tech" that is as old or older than msbuild that you might be familiar with: UNIX, Linux (powers the world), C (another one), C++ (another one), C#, Visual Studio, MacOS.

Like those, msbuild has not stopped evolving.

[Mike-E-angelo]

Like those, msbuild has not stopped evolving.

Easier statement to accept if it were possible to write MSBuild definitions in JSON. :smile:

No one doubts its power and capability. I am in agreement that MSBuild should tag along for the ride ( @shederman wants off the bus though LOL!). The challenge (or one of them, at least) that this issue is attempting to address is that how you go about interfacing with MSBuild is still very much 2002/2003. I mentioned this in another thread, but it's much like creating a VSIX in many ways: it feels arcane and unlike anything "new" that you see today.

Yeah yeah, I know it's a fool's errand to go chasing after the latest shiny toy, but the best aspects that make these toys appealing should at least influence a good product in some way. I think proper evolution accounts for this, C# probably being the best example.

[shederman]

Nobody is saying we should deprecate MSBuild. Nobody is saying MSBuild doesn't have a future.

All we're saying is that right now, there are two build systems, we like the "new" build system, and we want to keep it. We're saying that MSBuild doesn't work well for a lot of our simpler scenarios. So, just let us keep the "simple build" alongside MSBuild support. Why take the choice away? Why force a complex, counter-intuitive build system on people who just wants to compile the stuff in the directory? If we want to run a non-MSBuild complex build system, why force us to run it through MSBuild too? I mean, that's ridiculous.

I'm even offering to do the work. Put up or shut up, right? I will do what refactoring is require in order to keep the "new" build system working side by side with the old. As long as you don't make it impossible for me, of course 😉.

And it's easy to say MSBuild has been evolving, but I've seen more offered "evolution" of MSBuild in the past week than in the past 5 years! Why? Because there's a prospect of competition. But even significant evolution won't change the fact that it's really more ideal for dedicated build masters running big CI pipelines and complex builds. We don't all have that these days. Not even in many of the corporates. We have simple builds, and simple CI pipelines, and complex deployments. We tend to treat our builds like cattle, not pets - just like our servers.

I personally believe one of the selling points of .NET Core and ASP.NET vNext was to attract a new generation of developers to the platform, to help counter the slow slide in .NET adoption. Is removing one of the selling points of .NET Core, one of the core promises - and moving back to a system despised by a large fraction of your community (especially those working in the side of programming where the new generation cuts it's teeth) really going to help?

This was a bad decision IMO and made without considering all your users or their opinions, so please stop doubling down on it. Admit it, look at alternatives that allow a side by side solution, and move on.

Solutions have already been proffered for a smooth side by side experience.

[kzu]

Saying MSBuild will be shiny and new just because it (say) can now use JSON is so naïve. project.json might look simple for simple things, like defining dependencies, runtimes and stuff. If you have done any serious build work, you know that covers about 10% of what you can do with MSBuild. And, if you need to do those, I'm positive the JSON editing experience will be even worse than XML (as of today's "support" in VS for editing either).

Now, if you were proposing yaml.... ;).

I think there are ways to make MSBuild more approachable (better intellisense, better factoring/split of a .csproj for easier tweaking -.props/.targets alongisde .csproj by default), but its core concepts and features aren't all that complicated to grasp.

Its power and flexibility are only evident once you go beyond "it's a simple coordinator", which is what project.json might suggest is all there is to a complicated build.

[Mike-E-angelo]

Now, if you were proposing yaml.... ;).

Well yes... suggesting that and much more, actually. :smile:

I will say that I think we do a disservice to MSBuild and to the problem at hand by bringing in formats, or rather to describe the problem by using/comparing formats. It's unfortunate that the .NET Core build system only supported JSON, and that MSBuild only supports XML. At the end of the day these formats are (or should be) representing serialized models that are deserialized into memory at some point for the system to execute. But because these systems only support a certain format, that complicates the issue as developers turn it into a religious battle on who has the superior format (never mind the technology!). Additionally, the locked-in format tends to lend itself as the lightning rod for overall user/developer experience and product association -- for better or for worse.

Its power and flexibility are only evident once you go beyond "it's a simple coordinator", which is what project.json might suggest is all there is to a complicated build.

You sort of touch on this, but one thing that is being made apparent to me through this is that there is definitely a conceptual separation between project definition (what .NET Core build is "good" at) and process definition (what MSBuild is "good" at). Having better logistical separation of files (as you suggest!) in the format that developers prefer would go a long way, I feel.

(Btw, I appreciate the heavyweights dropping by to grace this topic with their thoughts. @kzu and @migueldeicaza your opinions are always welcomed! Thank you for taking the time to share. :+1: )

[shederman]

@kzu As @Mike-EEE said, I don't believe anyone is saying we want MSBuild to support JSON. What's kind of evolved out of some of the discussion is that the project definition should probably be json based, and that the build process definition should be based in whatever language the "big" build tool uses (i.e. XML for MSBuild).

But a lot of builds won't need a big build tool, their build is just to build the project definition, without any complexity. If so, why would MSBuild even be required in such a case?

Not arguing it shouldn't be available, we all occasionally need a powerful build system. But I don't believe it should be the default for all project types.

Especially given that this lightweight build exists, and is working. It's only the attempts to turn is into an MSBuild analogue that cause a problem. It shouldn't be that.

[Mike-E-angelo]

I don't believe anyone is saying we want MSBuild to support JSON.

Welllllll... technically I am. :smile: (And Yaml, and Toml, and Xaml and ...)

But this issue (to kindly remind is for a Format Agnostic MSBuild, as well as an improved model/representation via POCO) is an "eventual" or future ask/goal, not meant to be considered in the current raging debate. But if it can influence it, all the better. :+1:

[kzu]

Wellllllll, MSBuild already has an in-memory POCO representation of the files, so it shouldn't be too hard for anyone to write another serialization of that. It's all in Microsoft.Build.Construction ;)

And from that model you can see that the underlying concepts aren't complicated at all, IMO

[Mike-E-angelo]

Exactly what we're aiming for here, @kzu!

But to be fair, I think you meant the Construction namespace. :smile:

The goal being that any developer can "bring their own serializer" and MSBuild "just works" nicely with it.

[Mike-E-angelo]

Currently, if you look in the files (sample) you can see that they are all competely entrenched with XML-centric modeling/entities, which is a primary challenge at present.

It also looks like they do not have public parameterless constructors, which will not work out-of-the-box with most serializers.

[kzu]

That is true. Maybe it's "just" a matter of POCO-izing a bit more that set of classes?

On Mon, May 16, 2016 at 2:10 PM Mike-EEE notifications@github.com wrote:

Currently, if you look in the files (sample https://github.com/Microsoft/msbuild/blob/master/src/XMakeBuildEngine/Construction/ProjectItemElement.cs) you can see that they are all competely entrenched with XML-centric modeling/entities, which is a primary challenge at present.

It also looks like they do not have public parameterless constructors, which will not work out-of-the-box with most serializers.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub https://github.com/Microsoft/msbuild/issues/613#issuecomment-219483928

[Mike-E-angelo]

Maybe it's "just" a matter of POCO-izing a bit more that set of classes?

Well-and-wisely-said. It's a slippery (or challenging?) slope. If we POCO-ize them, then the logic that is currently within the objects will most likely move into other classes/services.

And also... while we are in there... are we now defining "elements" or "objects"? :) ProjectItemElement describes or alludes to xml document nomenclature, whereas ProjectItem seems much more like a POCO. Well, to me, at least. :sunglasses:

No small task, but this is exactly the path I am thinking of.

[radical]

@Mike-EEE I'm not sure what "format agnostic msbuild" will solve. You would still need to understand the concept of properties, items, tasks, targets, order of evaluation, dependencies etc. And how is it supposed to work? Everyone writes in whatever format they want and msbuild can read/write all of those - based on a file extensions or some header in the files?

Won't it make more difficult for "casual" users who look for a solution on stackoverflow etc, now they need to map it to whatever format the project, they are working on, happens to be using? It would make it more cumbersome to help people and to read+debug targets files (context switching between different formats, ensuring that escaping was correctly done in the files).

What is the real problem that you are trying to solve here?

Like @kzu said, moving the defaults to props files, should mostly reduce the csproj file to the most relevant bits - list of files, references etc, which should make it easier to make sense of and edit.

[kzu]

Also, I think it would be pretty straight-forward to create an msbuild task and accompanying targets that turns an arbitrary file into an MSBuild file that can be imported. With proper inputs/outputs, you wouldn't even notice ;)

For that to work flawlessly though, we'd need to allow Import inside a Target (the one after we generate the MSBuild from the YAML/JSON/Whatever).... ;)

[radical]

But a lot of builds won't need a big build tool, their build is just to build the project definition, without any complexity. If so, why would MSBuild even be required in such a case?

Even regular C# projects depend on the logic in the Microsoft.*targets files which do lot of things like - figure out the assembly references, build resources, satellite assemblies, doing all that for project references, making sure those are available for the compiler to use, bookkeeping for future cleans, incremental builds, incremental cleans, while providing hooks to allow the user to tweak the build process as little or as much they need to. There is a lot of stuff that happens "behind the scenes" in the msbuild targets files and tasks.

And if you are using products like Xamarin.Android/iOS, which have more custom requirements to work with other tools to build, package, deploy, have non-.net target frameworks etc, then all that logic goes in the msbuild targets and tasks files. The user still sees the regular looking csproj with the source/content files, references etc. All the complexity is still hidden in the targets/tasks files.

[Mike-E-angelo]

And how is it supposed to work? Everyone writes in whatever format they want and msbuild can read/write all of those - based on a file extensions or some header in the files?

Correct, it could be a combination of solutions, and I have given my thoughts in the OP. Totally open to discussion/consideration! This is just to get the ball rolling. And it appears at this point it is rolling up a very, very steep hill. :smile:

Won't it make more difficult for "casual" users who look for a solution on stackoverflow etc, now they need to map it to whatever format the project, they are working on, happens to be using?

Same could be said w/ C# vs. VB.NET. :smile: I am sure you have seen the differences of projects in different formats. As long as the model is the same, the format won't introduce variance in how the model behaves once deserialized, correct?

What is the real problem that you are trying to solve here?

There are three (that I can think of at the moment):

1. One of them as I (tried to :smile: ) state in the OP is that developers/organizations really love their formats, especially after the project.json movement (I invite you to read/participate this thread to see some of the sentiment towards JSON). Me, I personally am very much partial to Xaml. By restricting to a particular format you are precluding developers from using their format (and tooling) of choice. Making the way you can define your project files format agnostic allows developers/organizations to define their builds, and the tooling that might be developed around them to develop those definitions.
2. The second is that of tooling. Here I bring up Xaml again because I think it has the most mature tooling built around it. Here is an article that I wrote that sort of sums up why I prefer it when defining application entities. If we are free to use any format we like, then we can take advantage around any tooling that has already been created around that format to work as productively as possible.
3. Consistency. I don't know about you, but I prefer to work in C# for my code. If I can help it, I want to work in Xaml for all my data. Having one consistent format within solutions reduces complexity, context-switching, and overall confusion/hassle. Developers/organizations have their own recommended formats because of this.

Hopefully that helps clarify my perspective. Based on the feedback so far I am/have been doing a terrible job! :stuck_out_tongue:

[Mike-E-angelo]

For that to work flawlessly though, we'd need to allow Import inside a Target (the one after we generate the MSBuild from the YAML/JSON/Whatever).... ;)

Appreciating the ideas here, @kzu. Thank you for brainstorming and providing constructive ideas and feedback. :+1:

Just to be sure here, I'm old skool here and have over a decade's (on and off!) experience going through MSBuild "scripts." I am "ok" with the XML experience, but I would definitely like to see it improved if it can be helped. My goal/intention here is also to consider/onboard the JSON camp and help improve perception/adoption towards MSBuild and its perceived (poor) product experience.

[shederman]

@Mike-EEE help improve perception/adoption towards MSBuild and its perceived (poor) product experience.

For me, that's a simple solution: stop making it the core of all .NET build experiences. MSBuild is a very powerful system, but is complete overkill for 90% of use cases.

So here's a simple question. I get why

msbuild project.xproj

Would use MSBuild. I get it, I'm happy with it.

Why would

dotnet build project.json

Use MSBuild exactly? I mean you've got msbuild right? What does dotnet build add? Nothing other than confusion. If you're going to deprecate dotnet build with msbuild have the honesty to actually pull the trigger on it.

[wekempf]

@aolszowka No offense, but you know less about how IntelliSense works than I or @Mike-EEE does. XML schemas are used when editing XML, sure. I highly doubt they are used anywhere else though. Like @Mike-EEE says, it would make no sense to do so. You mention XML documentation, but code that doesn't compile XML documentation (the default) and doesn't even include triple-slash comments still produces IntelliSense. Yes, XAML has an XSD, but if you look into that it includes very little of what your XAML files actually contain or that IntelliSense displays. No, there may be some internal implementation detail that means XSD is produced from other sources internally, in memory, but you'll have to prove that before I believe it. In any event, MOST IntelliSense comes directly from sources other than XSD. The fact that you have to manually produce an XSD to get IntelliSense within an MSBuild file is a non-starter for most. That said, that's obviously a tooling issue that can be fixed, and not a complete argument against MSBuild. I'm not taking sides in that argument, but it's not helpful to have you ignoring other peoples input solely based on "IntelliSense uses XSD" when in fact, it doesn't (beyond some hypothetical internal requirement of the implementation that hasn't been shown to be true).

[Mike-E-angelo]

Haha... @wekempf thank you for that. I think it's safe to say that I have found the code/technical equivalent of my anger translator. :smile: You are welcomed to explain my position any time, haha. :grin: @aolszowka is clearly a super smart cat. We just got caught up in the weeds in trying to articulate my ideas/position, which can be absolutely exhausting I might add. :stuck_out_tongue: I am hoping that we can continue constructive dialogue towards improving MSBuild and its current marketplace perception while maintaining all the power/functionality that those who have invested into it already enjoy without reservation. :+1:

[colin-young]

Here's the thing: if you aren't a build/release engineer or otherwise intimately acquainted with msbuild, it's a miserable experience.

Without getting into specific file formats (I don't want to debate XML vs JSON vs YAML vs XAML) all I want (and I suspect many others do also) is the ability to quickly create a new standard project type (assembly, console app) that can immediately build and package on any supported .Net platform that has any .Net toolchain installed. i.e. if I have a full luxury edition Visual Studio install on Windows, the Core SDK on OS X or the Core SDK installed on my Windows build server, I can just do:

1. git clone {whatever}
2. {installation specific build command}, e.g. dotnet build or Build - Build Solution

There's no "if you're using the SDK you need to copy these targets from a machine that has Visual Studio installed" and I don't need to worry about magic csharp.targets or any of that nonsense.

It's the difference between saying:

1. here's a list of ingredients for a cake, go bake me a cake

and

1. go out to the garage and find this list of parts
2. take those parts and assemble them into a car
3. drive to the store, following this route
4. for each ingredient, find it in
5. take the following route home
6. in the following order, do with each ingredient
7. ...

That's all well and good if you're trying to do something weird, but when it's something simple like building an assembly and packaging it into NuGet package or creating a simple one-assembly program, msbuild is overkill. I don't need or want all that ceremony.

The mistake was trying to make project.json do everything instead of recognizing that it was really good at one set of tasks, msbuild is really good at another (much larger) set of tasks and figuring out how to make them work together. It's like progressive enhancement. You can start with a simple declarative project description, and when you need the full power of msbuild you can add it.

Some tools to aid in managing build files would be nice too...

[wekempf]

@Mike-EEE Yep. I don't intend to tell @aolszowka to stop arguing his position, on topic. But his technical argument about IntelliSense is wrong, not only in theory, but in practice as well. When half this thread is taken up with the back and forth between the two of you on that specific point, we're all wasting time here.

If MSBuild files were to acquire the same IntelliSense support that other file types have, even if it did so through the generation of XSD files (provided that generation was friction free), then another MSBuild pain point would be ticked off the list. There's a lot of them to tick off, though. Some more important than others (while I don't care for XML in this context, I'm not convinced any other format is a big enough "win" to matter, for instance). The best idea I've actually heard was to separate the build definition from the project definition, for instance. Do that and it becomes the project's choice to use MSBuild, Make, PSake or any other solution they care to. I actually mostly like MSBuild, to be honest, though comments about "posers" (never constructive) and "build masters" sure indicates that the tool is far too complex at least for the simple tasks, if not in general. A good tool makes it easy to accomplish the simple tasks, and possible to accomplish the complex ones. I'm not convinced MSBuild, today, does that.

[aolszowka]

@wekempf I have no problem being called out on anything I've said here; but please take the time to test and prove it to yourself before calling me out.

You could have easily proved this to yourself; as could have anyone in this thread that was unfamiliar with this; but instead I'll do it for you and others in the hopes of moving this conversation forward.

If you open up a new MSBuild File in Visual Studio and start with the default Project Tag you'll notice that there is no Intellisense being provided to you (See screenshot):

Now if we add the xmlns (basically importing the schema/xsd) you'll immediately start to get Intellisense:

The specification says that editors should be free to download from the specified location; but in Visual Studio's case (and I'm speculating on this point) it looks like they made the decision to cache the XSD locally to avoid the web call. You can find the XSD for Visual Studio 2012 located in C:\Program Files (x86)\Microsoft Visual Studio 12.0\Xml\Schemas\1033\MSBuild

The XSD standard allows for comments; if you look at that file you'll see there are several comments already available; so I could quickly show this in action I simply edited the documentation provided for ToolVersion

I hope that settles both your and @Mike-EEE 's concerns that The Visual Studio Editor does indeed use XSD to provide Intellisense as has been documented by Microsoft in several places as the MSBuild file is nothing but an XML Document.

[wekempf]

@colin-young

That's all well and good if you're trying to do something weird, but when it's something simple like building an assembly and packaging it into NuGet package or creating a simple one-assembly program, msbuild is overkill. I don't need or want all that ceremony.

I get, and agree with, the sentiment here. But, "if all you're doing is creating a simple blog, using C# is overkill" can be said as well. The power of MSBuild isn't really the issue. The complexity certainly is, but there may way be some things that can be done to reduce that complexity, at least in the "simple" cases. Maybe not. But dismissing the discussion about that is akin to the claim that Microsoft is dismissing the discussion of throwing MSBuild off the bus.

[Mike-E-angelo]

I don't want to debate XML vs JSON vs YAML vs XAML

while I don't care for XML in this context, I'm not convinced any other format is a big enough "win" to matter, for instance

I would like to kindly remind everyone that the purpose of this thread is to remove a dependency of format altogether, so that there is no longer a debate/battle at all :) If anyone can choose the format they wish to bring to the table (or "Bring your own serializer" as I saw in another issue) then they not only have the freedom to work with MSBuild in a manner that suits them best, but avoid having to argue about the most optimal format (as they see fit) to use in the first place.

The best idea I've actually heard was to separate the build definition from the project definition, for instance.

This is probably the biggest/beneficial yield from all of @shederman's ranting a raving :smile: , so I credit him: the notion that there is a project definition and then build (process) definition. I agree this is a major "aha" for me personally, and would love to see this adopted in some way going forward.

[wekempf]

@aolszowka You're simply wrong. I'll prove it to you. Create a C# project add a class and see you get IntelliSense.

See, you're arguing from a very narrow POV. You are 100% correct about how the XML editor provides IntelliSense. You're also 100% wrong on how Visual Studio provides IntelliSense.

[aolszowka]

@wekempf We're focused on how documentation could be provided to MSBuild in this conversation; not how Intellisense operates for other source types or editors. I think if you read up above I mention that XSD's are but one of many patterns Intellisense uses to gather input for its purposes.

[Mike-E-angelo]

Hahaha OMG @aolszowka at this point I am happier that at least someone else (@wekempf) understands what I was saying rather than trying to correct your understanding of what we are trying to convey. :smile:

[wekempf]

@aolszowka Yes, and you're POV is still too narrow. XAML is "just" XML. The XAML editor in Visual Studio, however, is NOT the XML editor, and it provides IntelliSense for everything without using XSD files. Create a WPF project and prove that to yourself, if you insist on such non-"proofs" on this topic. What @Mike-EEE is suggesting is that the MSBuild files can have a much better editing experience if we don't treat it as "just XML".

[wekempf]

@aolszowka BTW, I've never considered MSBuild's XSD files to be well thought out. You add properties to a property set with arbitrary element names that the editor flags as errors, for instance. That's not a good experience, especially for those new to/learning MSBuild.

[kzu]

@colin-young with everything becoming/being nugets, even down to MSBuild itself and even the compilers, you get your git clone && build for free and without changing a bit of anything ;)

[wekempf]

@kzu Maybe eventually, not today. And we've a long way to go before that's true in the future. There's still pain points in bundling .targets up in NuGet today, for instance (chicken and egg issues that require you to boot strap before you can really build). These things can certainly be fixed, and there's work arounds now, but "without changing a bit of anything" is overselling it.

[colin-young]

@wekempf I'm not trying to dismiss any of the discussion here. I was merely trying to point out that:

• there will likely be a lot of input from people in the "I never want to touch msbuild" camp if we do indeed end up with msbuild as the default project format, and they are going to have very different motivations and requirements than the typical msbuild expert/power user
• we could really use some better tools for editing msbuild files (I'm not touching the XML/XAML/Intellisense discussion myself as I have no dog in that hunt)
• just maybe we're (I'm?) trying to attack 2 different problems at once
  • a declarative description of the purpose and outputs of a project
  • instructions about how to achieve that

Any discussion about ways to improve msbuild are important, IMHO.

@kzu it's a bit difficult to comment without knowing what the final file formats are going to look like. I really just wanted to express my experience with what I liked about project.json and how it eliminated a class of issues I've historically encountered with trying to build stuff on headless build servers. I will continue to follow the latest changes in the .Net ecosystem, and will continue to express my opinion when I feel that some of my favorite features might be under threat.

[MarkPflug]

I think the problem is that right now an MSBuild file (.csproj, .vbproj, etc) is just treated as an XML file. This means that it only gets the XML intellisense provider, which is entirely based upon the XSD provided, as @aolszowka has repeatedly pointed out.

Rather than suggest that MSBuild be able to consume arbitrary file types (json, yaml, XAML, etc) which I think is an unrealistic idea (as well as a bad idea), I think it would make sense to suggest that there be a custom intellisense provider for MSBuild files. For example, when I type "$(", the intellisense provider could be smart enough to provide a completion list of all the properties that are currently in scope. The same for "@(" and items. F12 navigation for items and properties would also make MSBuild files a lot easier to author. I think this is what people are really asking for, they want it to be easier to author. XAML is easier to author because it has really nice intellisense support. That intellisense comes from the rich object model that drives XAML. There is a lot of "reflective metadata" that could be made available to an MSBuild file without completely re-imagining its fundamental nature (XML).

Why do I think that arbitrary formats is a bad idea? I'm glad you asked. Because it means that when I join a new project, I will likely have some new flavor of build file to learn. I disagree with the opinion that json is somehow magically easier to merge than XML. Both can be easy to merge, or a disaster to merge depending on the formatting and structure. XAML is just XML, so it clearly cannot be easier (or harder) to merge.

I think a lot can be done to simplify the default project templates, and make them easier to maintain, but that is a completely different discussion (that is currently happening in other github repos).

[Mike-E-angelo]

That intellisense comes from the rich object model that drives XAML

To be sure, that rich object model is actually the class definition(s) of the .NET POCO(s) you work with, which is what we're aiming for here. It is powerful (and dynamic) because if you change a property to your POCO, you've automatically changed the schema to the document in which you are authoring. This is a key dynamic that I have yet to find in any serialization tech and it's (one of the many reasons) why I prefer Xaml.

One of the (latent) problems we're encountering is that the "$(" and "@(" aren't really based on POCO's or anything .NET, but are ultmately magic strings (for lack of a better word) that are native to MSBuild's (XML) DSL, and not necessarily anything in .NET. If it's not tied to a .NET (POCO) model, then there is a large amount of (unnecessary) work that is required to build the model to make it tool friendly (XSD coming to mind here) and as a result should be a design approach that is ideally avoided.

Because it means that when I join a new project, I will likely have some new flavor of build file to learn.

Now you know how team JSON are feeling these days. :smile: Hopefully if you join a (functional) team, one of the discussions that they (should) have up front are things such as preferred formats and code formatting/styling rules. Meaning you should have a voice (vote) on the type of format you would prefer. Having options allows for more voices to be heard, and hopefully lending towards a better team experience.

[wekempf]

@MarkPflug Yeah, that's mostly what @Mike-EEE was arguing for. You have to go a bit deeper, because MSBuild is an "extensible" system, and currently the only way for IntelliSense to know how to work with those extensions is through XSD. Either that XSD needs to be generated from the source (for instance, the C# code that implements a new task) automatically, or better yet, we drop the need for XSD entirely, as the XAML editor did. No more squiggles for things like SccAuxPath. IntelliSense for $() references. IntelliSense for arbitrary arguments to new task elements without the need to hand generate an XSD.

Other things that would help:

1. Full support for NuGet. Rather than assume .target files are somewhere in the file system, specify NuGet references for them, solving the "boot strap" issue in the process. Also, adding a NuGet reference shouldn't require both the project file and the packages.config file to be modified. Far too often one gets modified correctly and the other doesn't, leaving you with a mess to clean up.
2. "Proper" support for wild cards (yes, this is mostly a Visual Studio issue, not an MSBuild issue).
3. Not requiring unload/edit/reload. Yes, that's entirely a Visual Studio issue, but this is the only reasonable place to bring up that discussion.
4. Some consideration in how to reduce merge conflicts. (2) may go a long ways towards that. We don't complain as much about merge conflicts in XAML files, likely do to them being edited less frequently and edited in a fashion that generally maintains the structure, while simply adding a class file today can result in multiple conflicts.

Those are the biggies, but there's lots of other pain points with MSBuild. Like I said, the need to have a "build master" is telling here.

[MarkPflug]

@wekempf

1. So then MSBuild has a dependency on NuGet. Now I need to boot strap NuGet. I'm trading one bootstrap problem for another. There needs to be "something" on the machine to enable the build. Right now it is MSBuild, and a well known set of targets files.
2. That is NOT an MSBuild issue at all. It is entirely a Visual Studio project system issue. There are discussions going on over in the AspNet/Home repo around this issue right now I believe.
3. Again a VS issue, not an MSBuild issue. If you use VSCode, this issue doesn't exist.
4. I agree here to some extent. The difficulty in merging the current .csproj flavor is due to the way that VS updates it. If it was more rigorous about the ordering of the files (alpha), or used globbing includes (which I think is the current proposal) **/*.cs, that would alleviate this problem as well.

Please list the "other pain points" with MSBuild so that we have something to work with. This argument that we need an expert to use MSBuild is true, in the same way that you need an "expert" in C# to write C# code, or an expert in Node to write node.js stuff. Yes, complex things require expertise to master, that is a fact.

[aolszowka]

@wekempf @Mike-EEE So to rephrase your ask in another way: You'd like an MSBuild-centric Editor Interface to Visual Studio? (Similar to the designer surfaces for other file types such as Resource Files, XSD Explorer, and the Xaml Editor).

That's a much different ask in my opinion than what was being posted here.

While it something that I think would be useful; if I were asked to pit that against other needs in the Visual Studio ecosystem I'd personally place it low. However I'd still throw a vote it's way.

For what its worth the undocumented debugger understood enough about the $() and @() syntax to give a somewhat usable view of these properties and ItemGroups; could that be leveraged?

[Mike-E-angelo]

You'd like an MSBuild-centric Editor Interface to Visual Studio?

Well, close. You mention Xaml editor, so I will use that because that is precisely the paradigm that I am striving for here. However, why make a whole new editor/interface when the Xaml editor you mention that is in VS works perfectly well for defining/describing any POCO you can think of?

Today. No installs or modifications or magic of the sort.

Accordingly, what is being asked here is is a POCO-based model to describe MSBuild documents. Once that is done, it can take advantage of the tooling that already exists in Visual Studio to describe/define those documents. Additionally, opening up the format to any format/serializer that is available only opens the options even more (to other IDEs/tools other than VS, or even editors within VS).

So I guess I should say that... the tooling/editor/interface is already available in Visual Studio, it's just that MSBuild needs to evolve/update itself so that it can be compatible with it and take advantage of it, if that makes sense.

[aolszowka]

@Mike-EEE That makes much more sense; I'm not sure I agree with a change to the format. However it is much clearer that your end goal is a better designer; using the Xaml Designer as an example.

[MarkPflug]

@Mike-EEE I'm trying to imagine what a XAML based build solution would even look like.

Imagine I want to customize my build process. What is involved? I need to define a POCO? Okay, I write a new C# file. Presumably, we'd want to support any .NET language, VB, F#? Once I've got my task type defined, that lives in a new project somewhere, right? I need to build it into an assembly? Well, we've got a recursive build issue to think about now, but we can surely solve that. That assembly needs to be referenced from the XAML. Now, I have intelliense in my XAML file, so I can have an easier time writing my original build script.

That doesn't actually sound like it is making the "overall" experience easier to me. It is making one small part of the build authoring process easier.

If I am completely misinterpreting your suggestion, please clarify.

[Mike-E-angelo]

Hey @aolszowka as long as it supports Xaml that is the only format I care about. :wink: Again for me supporting additional formats is a value/adoption proposition. I think it would be a cool way to extend an olive branch (for lack of a better analogy) to the community to say "hey look, we support your favorite format, get over here!" This sort of emulates the strategy that MSFT is implementing as a whole with Azure and Visual Studio, too. :)

[MarkPflug]

@Mike-EEE I get that you are in love with XAML. But honestly, XAML is going to do nothing to attract non-MS developers to the ecosystem. MSBuild is obviously not going to be rewritten to ONLY support XAML. I think we can agree that that is never going to happen. So, you are asking for XAML to be supported in addition to the existing XML format that we have today? Why do you want XAML? My understanding is that you want the XAML intellisense provider when editing your build files. The XAML intellisense provider isn't going to give the same experience that project.json provided anyway. The autocompletion on nuget packages, requires querying the nuget repositories, that isn't something that XAML is going to magically provide.

People have espoused the simplicity of project.json, but I think it is only true when compared to the current complexity of the default .csproj file format. There is no reason that the default template needs to contain everything that it does today. If you have a simple build, your .csproj file could be as simple as importing the "Microsoft.CSharpDefaults.targets" (this doesn't exist today, I'm imagining what things could look like). Additionally, the .csproj file could be split up into separate pieces. Perhaps all your NuGet dependencies live in a separate msbuild file (similar to packages.config) that gets included for you, and that particular file would get intellisense (and UI) support from VS; it also wouldn't require unload/reload.

I actually, really like XAML too. It is a great tool for what it was designed for. I just think you need to really consider what, and more importantly why, it is that you are asking for.

[kzu]

@wekempf this can be done today. Here's a trivial batch file that downloads nuget.exe, restores build-script packages from a packages.config (which could include MSBuild or MS's version once it ships, plus the MS compilers, plus the NuGet Tasks and targets -someone needs to package those as another nuget ;)).

My point is that we're way closer with existing tools to achieve that seamless experience than many think. @MarkPflug bootstrapping MSBuild, NuGet and the .NET compilers is getting easier by the day now.

[Mike-E-angelo]

That doesn't actually sound like it is making the "overall" experience easier to me. It is making one small part of the build authoring process easier.

Definitely. If you know anything about TFS Build process, then you know this was/is a nightmare. Getting all the assemblies resolving correctly, and then setting up the environment. Total hassle and a terrible experience/(and I would say name) for Xaml.

So, after bragging that there isn't any installs or magic (naturally :smile:), the point you make is quite salient, in that currently the Xaml designer works perfectly in the current csproj scheme. Its references/assemblies/projects are resolved from such. This of course would have to change with the new format. The csproj file (keep in mind is a serialized POCO) would get all of its references/data from ... itself? Another file? This a tooling consideration/concern that would certainly need to be ironed out.

But honestly, XAML is going to do nothing to attract non-MS developers to the ecosystem

Sooooooo that's a "yes" to supporting other formats, then? :smile:

Why do you want XAML? My understanding is that you want the XAML intellisense provider when editing your build files.

Well that's a part of it. It's the overall tooling experience, and also design. I explain here why Xaml is better than the app/web.config and project.json model, and I also explain here why it offers a better IDE experience. The key with the latter is that it lends itself very well for potential visual design. Which is great for supplementing the text design approach.

The autocompletion on nuget packages, required querying the nuget repositories, that isn't something that XAML is going to magically provide.

This is true, but in my opinion we shouldn't be using a text editor to define such things. :) Rather we should be making use of the wonderful UI that has been built around the NuGet Package Manager and have better integration with files with which they interact with and/or create (IMO of course -- I'm sure there's at least one person out there who disagrees with me :wink:). Which leads to your next point:

(this doesn't exist today, I'm imagining what things could look like). Additionally, the .csproj file could be split up into separate pieces.

This is great! No I don't have all the answers and am just getting the conversation started. And I agree with the separate files. This is something the TFS Build system could have benefited from tremendously. I am definitely pro-Xaml but I am not locked in as it being the one and only format, but I think it should be one of the formats supported.

Basically, in my world "all roads lead to MSBuild." Whether that road is built on XML, JSON, XAML, YAML, TOML, is up to the adventurer who takes the journey. How is that for some potential cheesy marketing mumbo jumbo? Ha ha. :smile:

[shederman]

Basically, in my world "all roads lead to MSBuild."

Just to be very, very clear: for a great many people that statement does not hold. I certainly see no reason why MSBuild must be executed by dotnet build unless I explicitly tell something (in project.json, or by presence of a xproj file or whatever) to use msbuild.

Should you choose to type msbuild I see every reason why that should run MSBuild 😉

Discussion of file formats should be fit for purpose. XML has a place for complex, powerful build systems with great customisability. So does XAML. I agree that in such a system JSON may not be ideal, although my preference would be for Roslyn provided build definitions ala sbt or gulp, but that's a different argument.

XML, however, is overkill for a project definition with a static, never changing schema. It's unnecessarily verbose, and adds a level of self-documentation which is just 100% unnecessary for a static schema.

So, why does my simple build that just builds the project definition require MSBuild?

I have heard not one valid reason for that. Just hand wavy rubbish from pro-MSBuild people talking about how powerful it is. Yes, it is, but that's a knife that cuts both directions.

I understand why Build Masters like @aolszowka have a vested interest in companies like mine using Build Masters for even small, simple builds; but it's just a waste of time and money, and makes zero economic sense to me.

Is .NET Core a jobs program for MSBuild Build Masters? Or is it a framework for effective software development?

[Mike-E-angelo]

Discussion of file formats should be fit for purpose.

Ah but again in my world there is no discussion of formats. You pick the format you prefer and enjoy the most, with the best of tooling to support it. Look how ideal my world is, so perfect, so pristine, so fantastic and wonderful... so completely non-existent and devoid of reality. :laughing:

XML, however, is overkill for a project definition with a static, never changing schema. It's unnecessarily verbose, and adds a level of self-documentation which is just 100% unnecessary for a static schema

Correct, according to your preference and way of approaching problems, which might be more ideal, proficient, and efficient. But might not be the case for others, where that verbosity is familiar, and as such (ironically) makes them more efficient and proficient. :) Why would we want to deny them their preference? Again, I go back to the original spirit of .NET: build your application in the language you prefer. Those who prefer C# can look at all the negative qualities of VB.NET and say why it is an inferior language. And those who (increasingly, I might add -- I am turning into one of them!) prefer F# look at all the negative qualities of C# (OO) and espouse why it is more expensive to build applications using its featureset.

But even still there are still experts in C# that can actually build solutions cheaper in that language than F#. So which is "better?" This foolish argument/discussion is the same I see whenever I see data formats. The "problem" isn't the format, but the technology that dictates/enforces one and only one. This is what I was driving at with the road analogy (seems like that one, didn't stick, either, LOL!).

So to (hopefully) sum it up: options -- like greed -- are good. :smile:

[shederman]

options -- like greed -- are good. The "problem" isn't the format, but the technology that dictates/enforces one and only one build your application in the language you prefer

👍

Dude, as long as I don't have to put my project definition in XML, and I don't have MSBuild running when I dotnet build, I really don't care what other formats are supported.

[kzu]

@shederman that's like saying that as long as whenever you write C#, it's compiled with such and such compiler and it's AOT'ed rather than JIT'ed and at runtime the GC uses that algorithm instead of that other one and what-not, "you really don't care".

why would you even care what happens behind the scenes as long as you write C# or "not-XML" as in this case?