Pipeline dotnet tool

[jnoyola]

I don't believe there's been much discussion of the design of the Pipeline dotnet global tool. I'm interested in helping with that next, as I think that's the last piece that need to be nugetized.

Open Questions

1. Future of Pipeline tool. [EDIT] Resolved: Pipeline is here to stay, while cra0zy is working on his own alternative.

   @Jjagg I am in process of writing something custom for the content builder, among the other stuff, so I'm leaving this to you. I really don't wanna touch MGCB/Pipeline Tool anymore.

   Originally posted by @cra0zy in https://github.com/MonoGame/MonoGame/pull/6905#issuecomment-546690990

   Should we be expecting something to replace the current tooling, or is that just personal preference or maybe an alternative option? (I assume the latter)

2. Name. [EDIT] Resolved: MGCB Editor is currently the top contender, with commands like dotnet tool run mgcb-editor

   As a public package, "Pipeline" is too generic. What name should be used for the package and the command? (Note: our other tools are dotnet-mgcb and dotnet-2mgfx.) For example:

   • Tool: dotnet-mgpipeline
   • Global command: mgpipeline
   • Local command: dotnet tool run mgpipeline

3. Structure. [EDIT] Resolved: The platform packages can't be unified, because we want native UI controls on Windows. We can, however, package them together and have a cross-platform launcher so the experience is seamless across all platforms.

   Should we package platform-specific Pipeline tools, or unify them? Is UI the only thing preventing these from being unified? Presumably shaders can just continue to not compile on non-Windows for now.

   • Platform-specific would be easiest since it would keep the same structure and Eto dependencies. We can publish dotnet-mgpipeline-windows, dotnet-mgpipeline-mac, dotnet-mgpipeline-linux, but this is clearly not ideal.
   • We can switch to a new cross-platform .NET Core UI framework and publish a single cross-platform tool. The most popular appears to be Avalonia.
   • We can use MonoGame.DesktopGL itself for the UI -- either a 3rd party UI framework, or maintain an extremely slim set of controls just for the tools (not a part of the MG framework).

4. MGCB dependency. [EDIT] Resolved: MGCB will be packaged directly into MGCB Editor. The duplicated files aren't too large and shouldn't be much concern for devs.

   How should the MGCB dependency be handled?

   • Just package MGCB binaries into the Pipeline tool as well? Then users who use the Pipeline tool and build with the MonoGame.Content.Builder package (#6905) would have a handful of files duplicated, but it's not a big issue. Is there a concern about the two versions needing to be manually kept in sync?
   • Package MGCB binaries into the Pipeline tool, but don't additionally required the MGCB tool when using the Pipeline tool. Instead use MGCB out of the Pipeline tool. See the Usage option below for more details.
   • Have the Pipeline tool depend on the MGCB dotnet tool. This would be difficult, because if the Pipeline tool is installed globally, then it would have to also install the MGCB tool globally, which is an option we decided to avoid with MonoGame.Content.Builder. This could also be difficult to run locally out of the MonoGame repo, because the Pipeline tool would need to be able to either depend on the MGCB dotnet tool or use the local MGCB project.

5. Usage. [EDIT] Resolved: /register and /unregister CLI commands will be available to associate the app with the .mgcb extension by running pipeline /register (global) or dotnet tool run pipeline /register (local).

   I expect a common scenario to be installing as a global tool, but can we enable more options?

   • As an additional option, I was thinking about being able to specify an option in the MonoGame.Content.Builder package (#6905) to restore the Pipeline tool instead of MGCB. This would make it easier for content designers to just clone a repo and automatically get the recommended tooling. And obviously this wouldn't have to compromise any of the core MonoGame.Content.Builder functionality.
   • Will we need to give up associating the .mgcb file extension with the tool? This doesn't seem like a good option anymore since dotnet tools can be installed globally or locally. Following the potential usage option above, in my own repos I plan to change my Pipeline extensions project into a netcoreapp that can simply call dotnet tool run pipeline on the correct .mgcb file.

[harry-cpp]

1. Should we be expecting something to replace the current tooling, or is that just personal preference or maybe an alternative option? (I assume the latter)

Alternative thing, basically you get a code project for compiling content. Right now the basic design goal is to be able to do something like:

// Add content stuff
contentBuilder.Add("Textures/*", new TextureImporter(), new TextureProcessor());

// This one will override the file if it was selected above
contentBuilder.Add(("Textures/SpecificTexture", new TextureImporter() { SomeProperty = 0.3f; }, new TextureProcessor());

// Build everything added
contentBuilder.Build();

[jnoyola]

Right now the basic design goal is to be able to do something like

Cool. Reminds me of some of those Cake alternatives like Nuke or Bullseye.

[mrhelmut]

1. Pipeline will be around for quite some time, I believe, so this discussion is very welcome. It's still something very useful to new users.

2. I'm personally very fine with mgpipeline, it continues the naming convention of the other tools and the templates.

3. Shaders will need more work that are safer to be addressed in a separate topic. For the time being, I think that the best is to stick with platform specific builds. Switching to a net core UI library sounds interesting to unify! But that sure sounds like quite some work.

4. Having the Pipeline to rely on the Content Builder would be the best, but if it's too much of mess regarding what's global and what's local, I think it's better to go with everything packed into a single package (duplicate files are a minor issue IMO compared to having everything up and running).

5. This actually sounds like a plan! It would make everything much smoother and easier to document.

Now that I think about it, I don't know about you all, but "Pipeline" always sounded confusing to me. Do you think that changing the name would prove useful? E.g. "MGCB GUI" (We can discuss this in another thread if you believe that's too much for now.)

[jnoyola]

"Pipeline" always sounded confusing to me

+1. "Pipeline" doesn't really describe what it does, and it's also easy to confuse with the Pipeline library. Fine with me to discuss other options here. I agree, something involving "Content", "MGCB", or "GUI" could be better.

2. I'm personally very fine with mgpipeline

   I suppose this should be deferred until we decide whether to change the name.

3. Shaders will need more work that are safer to be addressed in a separate topic.

   I agree a discussion about shaders won't be productive here. I just wanted to check: if we can merge the platforms under a single net core UI library, will this work without touching the shader code at all?

   It might be a lot of work. I can look into that, unless anyone else is familiar with Avalonia.

[Jjagg]

The shader code is a part of MonoGame.Framework.Content.Pipeline. It does not concern the Pipeline Tool. Currently the Pipeline Tool uses MGCB by executing it in a separate process, so it does not have a hard dependency on MGCB. In fact, something that cra0zy and me briefly talked about is that the pipeline tool could have a UI to pick what version of MGCB to use. Because the installations are not tied together, the Pipeline Tool could install MGCB as a local tool in the working directory, just like MonoGame.Content.Builder.

I do not think it is a good idea to rewrite the UI of the Pipeline Tool. A lot of issues users have with it are because of underlying design issues with the Content Pipeline. I'd rather see us focus on improving in that area than writing a new UI that only fixes the issue of portability. In fact I believe the best way to improve the CP is designing a new system from scratch, but let's not get into that in this thread.

Regarding cross-platformness, can we not make things work with a single nuget package? Or maybe we can have refactor things so we have runtime abstractions instead of build time (I'm not sure you can do that with Eto.Forms).

[Jjagg]

By the way, regarding the naming:

• (MonoGame) Content Pipeline = the library underneath MGCB that contains the framework for building content writers and the built-in writers.
• MGCB = the CLI for Content Pipeline
• (MonoGame) Pipeline Tool = the GUI to build response files (text files with command line options) for MGCB. In the end the Pipelie Tool simply runs MGCB with a bunch of arguments.

[mrhelmut]

I find it quite complex to have "MonoGame Content Pipeline" "MGCB" "Pipeline Tool" and "MonoGame Content Builder" all being different aspects of the same thing. Pretty sure we can come up with a more straightforward naming for all of those.

Regarding the Pipeline discussion, I guess that the short term goal would be to package it as-is for Windows, macOS, and Linux, and therefore have 3 nugets.

I don't know if it would be useful to choose which MGCB version to use, the typical use case is that you'd use the same version as the main framework.

My personal take would be to package both MGCB and the Pipeline Tool as one unique package to load with the builder, but this is assuming that we overcome the cross-platform issue. It'd be less complex and easier to explain, and users would have everything at hand without anything extra to do (e.g. install the templates and call it a day).

[Jjagg]

I find it quite complex to have "MonoGame Content Pipeline" "MGCB" "Pipeline Tool" and "MonoGame Content Builder" all being different aspects of the same thing.

This is especially true for MGCB and the content builder nuget because they're basically named the same except we use the acronym for the CLI tool.

Regarding the Pipeline discussion, I guess that the short term goal would be to package it as-is for Windows, macOS, and Linux, and therefore have 3 nugets.

I think that's fine for moving forward with this. It'll probably take longer to get this right with a single nuget, we shouldn't let that block us.

I don't know if it would be useful to choose which MGCB version to use, the typical use case is that you'd use the same version as the main framework.

Yeah, you're right. The way to make it the most useful is to tie the versions together so users don't have to configure the version or anything. I don't think there's a way for us to have the pipeline tool detect the version in any way.

How do we want users to use the pipeline tool? Launch it from the command line? What steps should they take? It would be nice if we could still register it in the registry to make the file association with .mgcb files. I think I also discussed that with cra0zy. The dotnet tool for the pipeline tool could have an install command that can do things like making the mgcb association or creating a start menu entry on windows or a .desktop file on Linux.

Edit: I should mention I'm not 100% sure this is possible.

[mrhelmut]

It would be nice if we could still register it in the registry to make the file association with .mgcb files. I think I also discussed that with cra0zy. The dotnet tool for the pipeline tool could have an install command that can do things like making the mgcb association or creating a start menu entry on windows or a .desktop file on Linux.

That would sound like a plan. I've been thinking about distributing it as an "extra" on the MonoGame's website with its own installer (which wouldn't be quite smooth of a user experience), but if we can perform the file association and not necessarily having to launch it from the CLI with just a nuget, that would be way better.

[tomspilman]

It would be nice if we could still register it in the registry to make the file association with .mgcb files.

So what happens when i have two different projects i'm working on my PC and both use different versions of MG? Both of them cannot be associated the .mgcb files at the same time.

Maybe something to consider there.

[jnoyola]

I find it quite complex to have "MonoGame Content Pipeline" "MGCB" "Pipeline Tool" and "MonoGame Content Builder" all being different aspects of the same thing.

What about leaving the Pipeline library, but then reflecting the relation between the others by naming them MGCB, MGCB.GUI, and MGCB.BuildTools, or something like that.

Both of them cannot be associated the .mgcb files at the same time.

Yeah, that’s what I was trying to point out in the OP. Having associations and icons just doesn’t seem feasible if we want it to be a dotnet tool.

[Jjagg]

So what happens when i have two different projects i'm working on my PC and both use different versions of MG?

Then you open it from the command line. This is not an issue unique to the new setup. If you have the tool installed on the system, you have a certain version installed. My proposal basically makes the version of the Pipeline tool that you run it with the system installed version. You can change the version by running the install command with a different version. If we want the benefits of a local installation (like file association) we need do support this somehow.

If we want to solve versioning in combination with the local installation features, our only option is to decouple Pipeline Tool and MGCB version like I said before. But the pipeline tool needs to figure out what version of mgcb to use somehow. I don't know how we can solve that. We could store MGCB version in the MGCB file, but that does not really make sense because it's (mostly) a response file.

[tomspilman]

This is not an issue unique to the new setup. If you have the tool installed on the system, you have a certain version installed.

I guess dot net tools allow you to install multiple copies at once and not just have one running.

I wonder if we could have some sort of pipeline.exe /register or something that you can do to have it self register as the default handler for MG files.

[jnoyola]

If we want to solve versioning in combination with the local installation features, our only option is to decouple Pipeline Tool and MGCB version like I said before.

I have two concerns with that still.

1. The Pipeline tool itself has it's own changes between versions. Your two projects could want different versions of the Pipeline tool itself, aside from the different versions of MGCB. In that case, which is associated with files -- the one installed later? That would be similar to how it would work with the current Pipeline tool I suppose.
2. Global tools are installed to a different location from local tools. If you install the Pipeline tool globally, but a project also installs it locally, which gets the association or desktop icon?

have it self register as the default handler for MG files

Do you mean you manually run this to choose which instance of the tool to associate with mgcb files? That seems reasonable to me.

[Jjagg]

I guess I wasn't clear about my proposal.

Users can install any version they want using the dotnet tool system, global or local. If they want to, they can execute a command with an installed pipeline tool (as a dotnet tool, not sure if we can do this for local tools though) to make that the one that handles mgcb files, similar to what happens when you run the SDK installer now. That's what I mean for the install command to do, and I think that's also what Tom means with /register. This is what I meant from the beginning.

In that case, which is associated with files -- the one installed later?

The last one the user ran the register command for. You can't have context sensitive file associations, there's no way around this. The version of the Pipeline Tool does not affect content in any way, so unless the format of MGCB changes any Pipeline Tool works with any MGCB version. We can track this like we do with shaders, by having an MGCB format version in MGCB files that gets incremented if the format changes. But generally you only need to have a recent version installed. Because the Pipeline Tool does not directly affect content you don't really need a specific version unless you have bugs or missing features in the UI itself (or a mgcb format mismatch).

[jnoyola]

Ah, my mistake. I didn't realize you were also referring to a manual "register" command. Yes, I like that too.

[Jjagg]

Regarding the naming, we should probably rename MonoGame.Content.Builder at least. I'd go with MGCB.Build or MGCB.Task. If we're gonna change the Pipeline Tool's name too, I like MGCB.Gui.

[tomspilman]

I'd go with MGCB.Build

That seems reasonable to me.... although it is in there twice... MonoGame Content Builder Build.

If we're gonna change the Pipeline Tool's name too, I like MGCB.Gui.

Always disliked the GUI idea... seemed like it didn't describe enough of what it does.

Some random thoughts...

• MGCB.Editor
• MGCB.Edit
• MGCB.Designer
• MGCB.Organizer
• MGCB.Inspector
• MGCB.Maker
• MGCB.Planner
• MGCB.Author
• MGCB.Creator
• MGCB.Writer
• MGCB.Scribe
• MGCB.Composer
• MGCB.Architect

[harry-cpp]

I vote for MGCB.Editor.

[Jjagg]

I like Editor too!

[jnoyola]

So the project can be MGCB.Editor, and the tool dotnet-mgcb-editor?

And it occurred to me that the library nugets are MonoGame.*, so should we use some expanded form of MGCB.Build for consistency there? (e.g. MonoGame.Content.Builder.Build) The current pattern is acronym for the tools but full name for the libraries.

[Jjagg]

I think it makes sense to write out the names. So then we'll have

• MonoGame.Content.Builder with dotnet tool command mgcb, referred to as MGCB.
• MonoGame.Content.Builder.Editor with dotnet tool command mgcb-editor, referred to as MGCB editor (pipeline tool might stick around, but we should probably try and avoid that).
• MonoGame.Content.Builder.Task (?) referred to as MGCB build task.

[jnoyola]

I like that, but I still think something other than Task might be more appropriate, since it’s technically not a task like the current project. Although “Builder.Build” could also be confusing. Where’s Tom with his thesaurus :P

[Jjagg]

It is a task, except it's not explicitly run by users. Maybe Builder.Target or Builder.Run? I still like Build because 'Builder' implies the program can build something, but 'Build' implies it actually does the thing. An alternative interpretation is that it runs MonoGame.Content.Builder at Build time. That's how I justify it anyway.

[mrhelmut]

I feel like Builder.Task or Builder.Runner are as close as it can get. Or maybe AutoRun?

[tomspilman]

So not sure what all the rules are here. But why can it not be MonoGame.Content.Builder and MonoGame.Content.Editor ?

[jnoyola]

I think the general idea is that MonoGame.Content.Builder (mgcb) is the CLI that builds content. And then we need names for the mgcb editor, and the package that runs mgcb as part of a build. These packages aren’t directly building content or editing content.

It is a task, except it's not explicitly run by users.

I was just pointing out the technicality that the old Tasks project contains actual MSBuild tasks that require a UsingTask to run, but the new package is really just a collection of Targets that are run as part of the build. I’m still fine with MonoGame.Content.Builder.Task if you guys think that’s best.

[Jjagg]

But why can it not be MonoGame.Content.Builder and MonoGame.Content.Editor

Yeah, MonoGame Content Builder is what mgcb stands for. That's exactly the confusion we want to avoid. Unless we change mgcb's name too. Then we can have MonoGame.Content.Cli, MonoGame.Content.Build and MonoGame.Content.Editor. But mgcb is nice as a short command to run for the dotnet tool and I'm not sure what we should replace it with then. Also the other two use mgcb, so it makes sense to have their name be hierarchically below mgcb's name.

[jnoyola]

Regarding the UI framework, I realized Eto can’t build all platforms on .NET Core, even as separate projects. So if we want it to be a .NET tool, we’d need a different framework anyway, right?

[Jjagg]

@jnoyola I think only Mac might be troublesome. Windows should be easy with .NET Core 3 and Linux should work with @cra0zy's GTK stuff. Relevant issue in Eto: https://github.com/picoe/Eto/issues/457

[jnoyola]

Yeah I was looking at that. I wonder how troublesome for Mac. I’m not super familiar with Eto.

[harry-cpp]

Technically speaking you can use Eto.Gtk for Mac if you want to build it for .NET Core. Gtk works on all desktop platforms.

[jnoyola]

Doesn't GTK need to be installed manually on Mac though? And even if we could somehow include it in the package, that would probably be too much extra bloat for Linux and Windows, so we might still want to keep those separate.

[mrhelmut]

Native dependencies for each platform aren't an Issue I believe, the tool already comes with a bunch of dependencies that kind of bloat it and we don't really a choice there. If we can make a GTK version to run as a core app, that would be cool actually (providing GTK runtimes can be included).

[jnoyola]

providing GTK runtimes can be included

That's the big 'if' for which I still haven't found anything promising.

[harry-cpp]

Bundling Gtk for Mac should add around an extra 50 MB once zipped.

[jnoyola]

I found some prebuilt Windows GTK binaries, but Eto is hardcoded to not use GTK for Windows.

Do we really want to use .NET Core 3.0 WPF for Windows? That would restrict users to Windows 10, right?

[persn]

Isn't .NET Core 3.0 supported down to Windows 7 SP1? https://github.com/dotnet/core/blob/master/release-notes/3.0/3.0-supported-os.md

[jnoyola]

Isn't .NET Core 3.0 supported down to Windows 7 SP1?

Oops, my mistake! I was just looking at that page but missed 7 SP1.

EDIT: Ok, so the plan is to use the following?

• .NET Core 2.1 (for broader OS support) with GDK for Linux/Mac
• .NET Core 3.0 (for WPF support) WPF for Windows

WPF requires Sdk="Microsoft.NET.Sdk.WindowsDesktop" -- can these be packaged together?

[jnoyola]

WPF requires Sdk="Microsoft.NET.Sdk.WindowsDesktop" -- can these be packaged together?

It looks like this isn’t possible. (And neither is multitargeting a dotnet tool in a single package either, but that’s not necessary if we can’t package GTK and WPF together.)

So I guess the next plan would be to try GTK for all platforms. @cra0zy do you know how to use Eto.Gtk on Windows? I see lots of threads about it, but it appears to be impossible based on the link in my previous comment, which I believe was written by you. Any insight?

EDIT: Never mind. It looks like you can just force the platform instead of using auto-detect. Now to see if the Windows GTK runtime will work.

[harry-cpp]

You answered all your questions XD

Anyway as long as you update both GtkSharp and Eto, and override the starting platform, the pipeline tool works without problems on Windows with Gtk backend (use Linux solution to run it):

[jnoyola]

Were you running that with the installed version of GTK? I tried placing the prebuilt vcpkg binaries in the same folder but still got a not found error for the GTK DLL.

[harry-cpp]

Were you running that with the installed version of GTK? I tried placing the prebuilt vcpkg binaries in the same folder but still got a not found error for the GTK DLL.

No, just use nuget to get the latest version of GtkSharp and Eto.Platform.Gtk packages instead of local refs, you don't need to manually install Gtk.

[jnoyola]

use nuget to get the latest version of GtkSharp and Eto.Platform.Gtk

That’s what I did. But don’t those just contain libraries for making calls to GTK, not the GTK runtime itself?

cc @mrhelmut since he seemed to have the same impression

[harry-cpp]

not the GTK runtime itself?

I made the newest version include an automated downloader for Gtk under Windows platform.

[harry-cpp]

PS. Do the following if you wanna see something cool 😈 :

dotnet new -i MonoGame.Template.Gtk.CSharp
dotnet new mggtk
dotnet run

[jnoyola]

I made the newest version include an automated downloader for Gtk under Windows platform.

That's cool. Maybe include a line in the readme. I saw you removed the part about installing GTK on Windows but I wasn't sure why.

However, this only automatically installs the runtime for devs when you build the project, right? Packing/Publishing the project doesn't include the runtime for distribution. If I copy the project to another computer, it still gets the DLL not found error. It would be cool if there were some option to copy the runtime to the output when building.

[harry-cpp]

However, this only automatically installs the runtime for devs when you build the project, right? Packing/Publishing the project doesn't include the runtime for distribution. If I copy the project to another computer, it still gets the DLL not found error. It would be cool if there were some option to copy the runtime to the output when building.

Packaging for Windows with Gtk is something that you should manually do as you can optimize the file size and install time by a lot if you know what you need.

If you are lazy you can just extract the following lib inside the publish folder and it will work just fine: https://github.com/GtkSharp/Dependencies/blob/master/gtk-3.24.zip

[jnoyola]

Ah makes sense. For the proof of concept I'm going to be lazy 😛

For final packaging, I might need help knowing what we need. I imagine it should be pretty simple to validate the UI, icons, file selection. MGCB.Editor doesn't really have uncommon edge cases.

[harry-cpp]

Now I started thinking and I could totally make an easy to use publish targets. I just need to have users specify which libraries they need (I can easily check the lib dependencies) and which icons they need.