Enable integration with 3rd party assets management solutions

[mkArtakMSFT]

Consider productionizing Microsoft.AspNetCore.ClientAssets package

[javiercn]

Here is a draft of the design:

Productionizing the client assets package

There are several challenges integrating third-party tools that produce or transform web assets as part of the build process:

• Developers need to write MSBuild targets to invoke their tools.
• Developers do not generally run their custom targets at the right point during the build.
• Developers generate the output into the wwwroot folder of the app, which then causes it to be considered as an input in the next build.
• Developers don't account for inputs and outputs for the tools, which means the tools always run and add time to the build even if the outputs are up to date.

Goals

• Simplify the integration of third-party tooling with static web asssets.

Out of scope

• Solutions for complex scenarios like dealing with NPM workspaces and other mono-repo setups.
• Orchestrating invocation in sequence for third-party dependency tools beyond installing tools and running them.
• Watch support (if the tool has support for watch that is something we might consider in the future)

Technical design

We want to offer a declarative model where customers can indicate what tool to run, the inputs and the outputs and we take care of the rest.

We will likely need a couple of "phases" or extensibility points in the pipeline.

One initial phase to ensure that any third-party dependency toolchain (like npm or yarn) had a chance to run.

A second phase where we invoke whatever tool or script we are given.

Ideally, we want this to be a collaborative process where multiple tools can be added provided they don't depend on each other.

That allows packages to include a command for their toolchain as long as it doesn't depend on the outputs of any other tools.

One way we can make this work is by defining the invocations inside an item group as follows:

<ItemGroup>
  <ClientAssetTool Include="InstallNpmDependencies">
    <Stage>Dependencies</Stage>
    <Command>npm install</Command>
    <Directory>assets</Directory>
    <Inputs>assets\package.json;assets\package-lock.json</Inputs>
    <ExcludeInputs></ExcludeInputs>
    <Outputs>assets\node_modules\package-lock.json</Outputs>
    <ExcludeOutputs></ExcludeOutputs>
  </ClientAssetTool>

  <ClientAssetTool Include="RunNpmBuildScript">
    <Stage>Generation</Stage>
    <Command>npm run build:$(Configuration)</Command>
    <Directory>assets</Directory>
    <Inputs>assets\**</Inputs>
    <ExcludeInputs></ExcludeInputs>
    <Outputs>*.js</Outputs>
    <ExcludeOutputs>*.map.js</ExcludeOutputs>
  </ClientAssetTool>
</ItemGroup>

The metadata included in the item group defines all the aspects of how and when the tool needs to run:

• Stage indicates when to run the tool:
  • Dependencies: This stage is intented to restore/install/retrieve required packages.
  • I want to avoid calling it Restore to avoid confusion as this will only run as part of build.
  • Generation: This stage is intended to run whatever commands need to run to generate the asset outputs.
  • If a set of tools need to be run sequentially the correct approach is to use a script, we won't handle ordering the different tools meant to run at a given stage.
• Command indicates the command to run.
• Directory indicates the directory from where to run the command from.
• Inputs, ExcludeInputs: Allows defining patterns to compute the inputs for a tool.
• Outputs, ExcludeOutputs: Allows defining patterns to compute the outputs for a tool.

With this metadata we will do as follows:

• Add a Target to static web assets:
  • ResolveGeneratedStaticWebAssetsInputs that will run before resolving assets for the current project.
• Create a new task RunClientAssetsTool that handles executing the client assets commands:
  • Resolves the inputs and outputs for a given entry.
  • Determines if it needs to run the tool.
  • Runs the tool.
  • Captures the outputs.
  • Returns the outputs.
• The Dependencies stage doesn't add any output to the build (only captures them for incrementalism).
• The Generation stage adds the outputs as ContentItems with the Link to the wwwroot folder.
• During each stage, we log in the inputs, outputs, and the decissions we make along the way to make troubleshooting inside MSBuild easy.

Notes on testing

• We don't have to use any third-party tool for testing this, we can use a cmd/powershell/bash script as long as we make it do whatever is needed to ensure it covers the scenarios we care about.

The biggest part in this aspect is the incrementalism. Some scenarios that I can enumerate:

• Things that should trigger the tool to run:
  • An additional input was added.
  • An existing input was removed.
  • An existing input is newer than the last outputs.
  • Any information about the tool changed (command, input pattern, output pattern, exclusions, directory, etc.)
  • No outputs were found.
• Things that should not trigger the tool to run again:
  • Inputs are the same and older than outputs.
  • Outputs didn't change after the tool ran (even if they are older).

Questions

• Do we need a separate stage for preparing production level assets?
  • SPAs currently generate the production bundle only during publish.
  • Static Web Assets already supports the concept of Build and Publish assets, so we might want to levarage it here.
• How do we pass the output folder information to a third-party tool?
  • We want to put the outputs in the intermediate output folder obj\clientassets\toolname to avoid them confliciting with the build.
  • We also want to support tools that put the output within the "source" tree and prevent those outputs from becoming inputs in the next build.
  • We can pass the output path as an environment variable, but some tools might only take command-line arguments.
  • We could have a token that we replace in the command before running it, like [TOOL_OUTPUTDIR] in addition to setting an environment variable when running the command.
• Do we want to define the generated assets as static web assets directly or do we want to mark them as content.
  • I believe it is better to mark them as Content and use the Link functionality to define their relative paths in the project. Static web assets already takes that into account when discovering new assets and that automatically gets them to participate in other parts of the pipeline, like scoped CSS or JS modules. Alternatively, features like scoped CSS and JS modules would need to look at an additional item group.
• Do we want to provide conventions for popular tools like NPM?
  • These can be on the SDK or in packages.

[MarkStega]

This is close to what I would like to see but I believe that there needs to be a way to have multiple sets of input & output specified. For example, I build both js & css; If a js source file changes I don't want to build css. As long as I can have multiple sections with generate and different commands/inputs/outputs I'd be covered

[javiercn]

@MarkStega this design currently contemplates that.

You will create two separate item group items each one specifying a different command and a different set of inputs/outputs, one for your CSS and one for your JS

[MarkStega]

@TanayParikh I am very interested in using this feature so I will volunteer to test off of nightly builds when you get to that point.

[mkArtakMSFT]

@TanayParikh please make sure this is also covered as part of the work: https://github.com/dotnet/aspnetcore/issues/42472

[MariovanZeist]

@TanayParikh I have been using this package for a while now. (Well a slightly modified version of this library to be fair) The only problem I had with it was that when the path would include spaces, the output would not be generated I created a PR to fix the issue https://github.com/aspnet/AspLabs/pull/405, (And I have been using that modified version myself) but I must admit I am not a node nor an MSBuild specialist, so my fix could be wrong.

Do you have any insights into this?

[javiercn]

@MariovanZeist we do not plan to move forward with the experimental package (as we are planning to integrate the functionality directly into the SDK). Being an experimental package, it was expected to have bugs. We plan to address things like this as part of moving the functionality to the SDK repo, as we have there the infrastructure to test it properly.

That said, the new functionality will give you the ability to define the command yourself, so you'll be responsible for things being quoted appropriately.

We might define some out of the box conventions either on the SDK or inside a new package to further streamline the scenario when following the conventions (so that basically you put assets in a specific folder, and everything magically works after that). In that case we would be taking care of the quoting.

[MariovanZeist]

@javiercn Thanks for clearing it up, I was indeed under the mistaken assumption that this particular package would be used. I prefer it being part of the SDK anyway, so thanks for the info.

[javiercn]

@MariovanZeist yeah, the idea is that we make this process declarative, you get to say what commands to run, what inputs and outputs and we take care of making sure that they end up in the right place inside the wwwroot folder and that things are done incrementally (provided you correctly specify inputs and outputs).

It is much easier for us to test those things in the SDK repo because we have infrastructure that can create a project and run through different scenarios to make sure that stuff works, and it does not regress over time when changes are made.

[jeffputz]

I want to reiterate the desire from #40572 about how there is a production need as well as a development need. Even the nomenclature of "asset" implies that Typescript is not code. If I were to simplify the need it looks like this:

• Typescript gets compiled and triggers a browser refresh when I save a Typescript file in a web project, and it also builds in the output locally or in pipelines or whatever.
• If I move it to an RCL, honestly the building parts are fine, but I get no browser refresh, meaning I need to trigger an entire build which is terrible for development. Accommodating this one scenario would be a game changer for folks who want to package all of their "assets" along with dotnet in an RCL, especially if you publish as a Nuget package.

[ghost]

Thanks for contacting us.

We're moving this issue to the .NET 9 Planning milestone for future evaluation / consideration. We would like to keep this around to collect more feedback, which can help us with prioritizing this work. We will re-evaluate this issue, during our next planning meeting(s). If we later determine, that the issue has no community involvement, or it's very rare and low-impact issue, we will close it - so that the team can focus on more important and high impact issues. To learn more about what to expect next and how this issue will be handled you can read more about our triage process here.

[ghost]

We've moved this issue to the Backlog milestone. This means that it is not going to be worked on for the coming release. We will reassess the backlog following the current release and consider this item at that time. To learn more about our issue management process and to have better expectation regarding different types of issues you can read our Triage Process.