Post-build YAML stages should push NPM and Maven artifacts to Azure Artifacts

[analogrelay]

Currently, the post-build YAML stages push NPM and Maven artifacts to blob storage. However, since this storage doesn't provide the proper registry layout for NPM and Maven, these packages can't be accessed via those protocols.

We should push the NPM and Maven artifacts to Azure Artifacts, like we do with the NuGet packages. Azure Artifacts has full support for NPM and Maven, so this should be possible.

There seem to be a few issues in the way of this support. First is that the InferCategory method used to identify the type of an artifact based on file extension isn't configured correctly for NPM and Maven. NPM packages have a .tgz extension, but the method expects .npm as a file extension. Similarly, Maven packages are made up of three separate .jar files and a .pom file but only the .pom file is categorized as a Maven artifact. The .jar extension appears to be categorized as an "Installer" artifact.

[analogrelay]

FYI, this issue is blocking full migration of ASP.NET Core artifacts from MyGet to Azure Artifacts.

[riarenas]

It's not just about the categories. We simply only manage NuGet packages right now and that's the only "package type" that is supported. Publishing npm and maven packages to azure artifact feeds need its own set of commands and configuration.

The ask makes total sense though, we just haven't implemented anything yet that would let us publish anything but NuGet packages to artifact feeds.

[analogrelay]

Yes, I noticed that. My description of issues I found was not meant to be a complete set :).

[riarenas]

(Just adding some context for when we look at this)

for NPM: https://docs.microsoft.com/en-us/azure/devops/artifacts/get-started-npm?view=azure-devops&tabs=windows

for Maven: https://docs.microsoft.com/en-us/azure/devops/artifacts/get-started-maven?view=azure-devops#publish-an-artifact

[analogrelay]

for Maven: docs.microsoft.com/en-us/azure/devops/artifacts/get-started-maven?view=azure-devops#publish-an-artifact

One note about this. The doc appears to describe using mvn deploy which you generally run from the source directory, but in our current process, we just have the binaries at this point and are publishing those. I'm not sure exactly what the process for those is. Will likely need a bit more investigation :).

We'll also want to be able to (optionally) publish to Maven Central's staging repo (unlike NuGet, Maven has a "staging" repo you publish to that is not yet public and then you can select a version to publish to the "public" repo). The process will likely be similar (we'll investigate that later, in a separate work item, just raising that note here).

[dougbu]

Slight side note:

We simply only manage NuGet packages right now and that's the only "package type" that is supported.

This isn't true when publishInstallersAndChecksums: true is passed to post-build.yml. That at least gets any type of blob into a feed. The needs beyond what exists in Arcade may be limited primarily by the capabilities of AzDO public feeds. Where those feeds are insufficient, may need work arounds in Arcade.

In addition, we should be careful here to avoid touching on pushes to final locations such as Maven and npmjs.org when discussing this issue. Need to completely avoid such pushes in many cases e.g. servicing. Will still need separate pipelines to manage final deployments.

[riarenas]

This isn't true when publishInstallersAndChecksums: true is passed to post-build.yml. That at least gets any type of blob into a feed

We don't actually push any of those as packages in a feed, we publish them to blob storage as non-restorable blobs. This issue asks about actually publishing these types of artifacts as packages that can be consumed by the different package restoring mechanisms for these ecosystems.

In addition, we should be careful here to avoid touching on pushes to final locations such as Maven and npmjs.org when discussing this issue. Need to completely avoid such pushes in many cases e.g. servicing. Will still need separate pipelines to manage final deployments

Definitely. This should only track publishing these packages to the azure devops feeds that we already publish NuGet packages to. These feeds also support npm and maven packages, so it makes sense to publish them that way.

[analogrelay]

Need to completely avoid such pushes in many cases e.g. servicing. Will still need separate pipelines to manage final deployments.

Yep, that's totally fine, they can be separate pipelines.

[markwilkie]

@elbatk - What's the roadmap (if any) for Azure Artifacts support for NPN/Maven?

[markwilkie]

@jcagme - another loose end to follow up on.

[jcagme]

Would this be more of a build thing? As we push nupkgs to feeds and AzDo feeds as the build is promoted? Happy to own the investigation.

[markwilkie]

This is about publishing - namely npm and Maven. Maybe that work is going more into the dependency flow stuff @mmitche ?

[JohnTortugo]

This is about publishing - namely npm and Maven. Maybe that work is going more into the dependency flow stuff @mmitche ?

At least it'll require some work from the publishing part. We don't even upload that kind of asset to AzDO artifacts.

[jcagme]

@JohnTortugo is there an epic around publishing where this issue is better located?

[riarenas]

We got another ask for this by @brettfo. They are building their own custom steps to publish npm packages to our feeds, so maybe it's a good time to think about setting up something in the shared publishing infrastructure.

The only current epic working in the publishing area is https://github.com/dotnet/arcade/issues/5789 to unify and improve publishing, but it's kind of a stretch. Not sure what the proper place for this is.

[riarenas]

Moving this back to dotnet/arcade-services#2570 to be taken care of when we have funding for that work.

[riarenas]

Another ask about this came up today by @BrennanConroy