Use NuGet Release notes

sommmen commented 2 years ago

What would you like Renovate to be able to do?

I've got several packages where the changes are not in a changelog.md file, and the project's github url is not linked or github releases are not used. In those cases no changelogs are put in the PR's opened by Renovate.

Sample package with release notes:

Take these packages for example:

https://github.com/BingAds/BingAds-dotNet-SDK/releases/tag/v13.0.13
- No github linked but the nuget changelog is being updated.
https://www.nuget.org/packages/EFCore.BulkExtensions/
- This author does not use changelog files nor github releases, but does change the package release notes manually each release, see for ex: https://github.com/borisdj/EFCore.BulkExtensions/commit/2a9a6c8b75607e7c74e75c109ce143340b6ffda2

I believe (and im not 100% sure) the nuget package release notes field is not adapted broadly and preference is generally to changelog files etc. Changelog / github releases should therefore take precedence.

I have more like these and if you take EfCore.BulkExtensions for example it has 8.3M downloads and gets updated daily - so there's some impact to be had here. For me the release notes is a big feature because it saves me a lot of time tracking down what changed before i can complete my pr.

EDIT: I'd also love for the PR to contain a link to the package on nuget so i could also view the release note(s) there - but perhaps thats a setting or something and ill investigate.

If you have any ideas on how this should be implemented, please tell us here.

info can be fetched directly from nuget.

If someone could point me in the right direction i can maybe also take a look at working on this myself.

Is this a feature you are interested in implementing yourself?

Maybe

github-actions[bot] commented 2 years ago

Hi there,

Help us by making a minimal reproduction repository.

Before we can start work on your issue we first need to know exactly what's causing the current behavior. A minimal reproduction helps us with this.

To get started, please read our guide on creating a minimal reproduction to understand what is needed.

We may close the issue if you (or someone else) have not provided a minimal reproduction within two weeks. If you need more time, or are stuck, please ask for help or more time in a comment.

Good luck,

The Renovate team

rarkins commented 2 years ago

Good to start with a minimal reproduction repo to test against.

Next thing to check is: does the NuGet API for those packages tell us about the source URL? e.g. https://www.nuget.org/packages/Microsoft.BingAds.SDK/ doesn't appear to link to its source website on github.com

Here is where we attempt to get sourceUrl from the API: https://github.com/renovatebot/renovate/blob/a859c7f279863028414377f5660e7bff15b94bcd/lib/datasource/nuget/v3.ts#L177

If that sourceUrl is found then we attempt to fetch release notes from the repo.

viceice commented 2 years ago

Workaround for the bingads sdk it to add a manual source url here:

https://github.com/renovatebot/renovate/blob/bc3a5281ca2543e5749b2bb0970586b1a17c0f8b/lib/datasource/metadata.ts#L70

viceice commented 2 years ago

Please vote for:

BingAds/BingAds-dotNet-SDK#103

sommmen commented 2 years ago

Hello - Thanks for taking a look at this!

@rarkins

Good to start with a minimal reproduction repo to test against.

A public github repo with a .csproj with those libraries would suffice? Or do i need to add test somewhere? I'm using azure devops personally.

If that sourceUrl is found then we attempt to fetch release notes from the repo.

To clarify i want to handle 2 cases:

There is no link to Github (like BingAds)
There is a link to Github but there is no changelog file in the Github repo (like BulkExtensions)

Both publish release notes in the package metadata under the 'releaseNotes' metadata tag:

I'm grateful for the workaround you've provided for these packages - but that is not the point - i want to fetch the releasenotes from the package metadata and show that in the PR. These package authors dont use a changelog or github releases. They do use the releaseNotes metadata embedded in their package definition.

I'm unfamiliar with the nuget api - so i'm not sure if there is an endpoint to fetch meta data. I took a quick look but couldn't find something - ill investigate more later.

Nuget.info can also display the release notes metadata and it does so by fetching the nuspec from the package (afaik by inspeciting with the browser) https://nuget.info/packages/Microsoft.BingAds.SDK/13.0.13

@viceice

Please vote for:

Sure voted!

rarkins commented 2 years ago

In this case, there are two possibilities:

If fetching the release notes at the same time as the list of releases itself is lightweight (no extra API requests) then we could enhance our datasource concept to allow optional release notes per release
Otherwise, we need to enhance our release notes fetching code (which currently supports GitHub and GitLab hosts) to fetch from NuGet when applicable

viceice commented 2 years ago

related discussion:

https://github.com/renovatebot/renovate/discussions/10324

viceice commented 2 years ago

related issues:

9061

sommmen commented 2 years ago

@rarkins

In this case, there are two possibilities:

If fetching the release notes at the same time as the list of releases itself is lightweight (no extra API requests) then we could enhance our datasource concept to allow optional release notes per release

Otherwise, we need to enhance our release notes fetching code (which currently supports GitHub and GitLab hosts) to fetch from NuGet when applicable

Well there'd be (optional) extra api requests to nuget - but not to github or anything. We'd have to take private feeds in mind.

Here's an link that works:

https://api.nuget.org/v3-flatcontainer/EFCore.BulkExtensions/6.4.0/EFCore.BulkExtensions.nuspec

<?xml version="1.0" encoding="utf-8"?>
<package xmlns="http://schemas.microsoft.com/packaging/2013/05/nuspec.xsd">
  <metadata>
    <id>EFCore.BulkExtensions</id>
    <version>6.4.0</version>
    <title>EFCore.BulkExtensions</title>
    <authors>borisdj</authors>
    <license type="expression">MIT</license>
    <licenseUrl>https://licenses.nuget.org/MIT</licenseUrl>
    <icon>EFCoreBulk.png</icon>
    <projectUrl>https://github.com/borisdj/EFCore.BulkExtensions</projectUrl>
    <description>EntityFramework EF Core Bulk Batch Extensions for Insert Update Delete Read (CRUD) operations on SQL Server, PostgreSQL, SQLite</description>
    <releaseNotes>PS SaveChanges fix</releaseNotes>
    <tags>EntityFrameworkCore Entity Framework Core EFCore EF Core SQL Server PostgreSQL SQLite Bulk Batch Extensions Insert Update Delete Read Truncate SaveChanges</tags>
    <repository type="Git" url="https://github.com/borisdj/EFCore.BulkExtensions" />
    <dependencies>
      <group targetFramework="net6.0">
        <dependency id="EntityFrameworkCore.SqlServer.HierarchyId" version="3.0.0" exclude="Build,Analyzers" />
        <dependency id="Microsoft.Data.SqlClient" version="2.1.4" exclude="Build,Analyzers" />
        <dependency id="Microsoft.EntityFrameworkCore.Relational" version="6.0.0" exclude="Build,Analyzers" />
        <dependency id="Microsoft.EntityFrameworkCore.Sqlite.Core" version="6.0.0" exclude="Build,Analyzers" />
        <dependency id="NetTopologySuite.IO.SqlServerBytes" version="2.0.0" exclude="Build,Analyzers" />
        <dependency id="Npgsql.EntityFrameworkCore.PostgreSQL" version="6.0.1" exclude="Build,Analyzers" />
        <dependency id="System.Diagnostics.DiagnosticSource" version="6.0.0" exclude="Build,Analyzers" />
      </group>
    </dependencies>
  </metadata>
</package>

And there's the releaseNotes field! I think there's also a way to extract the just that field by using the registration api but the docs are a bit confusing.

https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource

@viceice

related issues: #9061

I also noticed this for this package: https://www.nuget.org/packages/System.Linq.Async/

Releasenotes are fetched from github releases - but they're releasing 2 packages seperately in the same github release: https://github.com/dotnet/reactive/releases

But in that case it seems impossible to determine which release belongs to which.

viceice commented 2 years ago

releaseNotes are not included in CatalogPage and we don't like to fetch every nuspec` file to get all entries. But this can probably changed when #14138 is done and we can cache those info's for a very long time.

https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource#catalog-entry

sommmen commented 2 years ago

releaseNotes are not included in CatalogPage and we don't like to fetch every nuspec` file to get all entries. But this can probably changed when #14138 is done and we can cache those info's for a very long time.

https://docs.microsoft.com/en-us/nuget/api/registration-base-url-resource#catalog-entry

I get where you're coming from. I'll watch #14138 .

SierraNL commented 1 year ago

I'm a bit confused reading through this thread and looking at the code, it seems that the nuspec is already being read to fetch the sourceUrl it would be trivial to also fetch the releasenotes from the nuspec if they exist.

The only issue I see is that the returning object only supports a changeLogUrl, so either this needs to be extended, or if we see that this element exists, we need to try and build a url to the webpage of this package, but that will differ depending on the source (nuget.org / private feeds like myget, azure devops artifact feed, etc).

Or am I missing something?

We use the releasenotes element extensively for a lot internal packages used by quite a few different teams, since they are displayed properly in Azure DevOps Artifact Feeds. But having them in our renovate PR's would help teams safe time.

SeanKilleen commented 4 months ago

Hi @rarkins I wanted to chime in here to see if I might be able to move this forward.

Our particular use case is that we use IronPDF (https://www.nuget.org/packages/IronPdf). Their project is not OSS, but they do publish release notes in their nuget feed (Example: https://www.nuget.org/packages/IronPdf/2024.8.3#releasenotes-body-tab)

Based on @SierraNL's comment above, it seems like the path to getting this in would be to do the following (supported by tests):

A field on the Release type called specifiedReleaseNotes or directlyProvidedReleaseNotes or datasourceProvidedReleaseNotes or similar
The updating of the nuget datasource to look at the metadata.releaseNotes field of the nuget XML and populate the above field with contents if found
A fallback during the place where release notes are specified (I haven't hunted that down yet 😅 ) to fall back to this directly specified text as a last resort if other higher-fidelity release note options aren't found.

Does that make sense as an overall approach, or am I missing something? If folks feel good about that, I could probably start on a PR since you've all been kind enough to do the nice work of getting such a robust test suite and devcontainer in place already.

Let me know!

rarkins commented 4 months ago

Hi @SeanKilleen, thanks for proposing to advance this!

Can you clarify one simple/quick question to get started: In cases where release notes are included on the registry, are they contained in the data (i.e. API responses) which Renovate already retrieves using the nuget datasource, or would this be new endpoints on the nuget registry which Renovate isn't currently querying?

SeanKilleen commented 3 months ago

@rarkins no additional API call AFAIK. There's a place in the code where RenovateBot already has the nuspec XML back from the API and is using it to look for an official repo URL. My plan is to add a way to extract the information at that point, and then figure out how to move it along the pipeline and fallback to it if needed.

In the associated PR I've gotten started but since it's the first time I'm working out loud on a draft and fumbling around. That's fine by me but I'll look for more guidance as it firms up. I've already got the test cases in place and passing to extract the nuget data to a new field (I think in the right place? We'll see?) -- next step is figure out how to pass it along in the process and include that data as a fallback when the release note text is determined. (Again...I think 🤞)

Advice and blunt feedback welcome along the journey.

rarkins commented 3 months ago

FYI I'm on vacation until mid august so won't be particularly active myself

viceice commented 3 months ago

@rarkins no additional API call AFAIK. There's a place in the code where RenovateBot already has the nuspec XML back from the API and is using it to look for an official repo URL. My plan is to add a way to extract the information at that point, and then figure out how to move it along the pipeline and fallback to it if needed.

In the associated PR I've gotten started but since it's the first time I'm working out loud on a draft and fumbling around. That's fine by me but I'll look for more guidance as it firms up. I've already got the test cases in place and passing to extract the nuget data to a new field (I think in the right place? We'll see?) -- next step is figure out how to pass it along in the process and include that data as a fallback when the release note text is determined. (Again...I think 🤞)

Advice and blunt feedback welcome along the journey.

We only do that for the latest version. We don't want to fetch that file for every version to get all release notes. Too many API calls.

SeanKilleen commented 3 months ago

@viceice that makes sense and helps me with some of my confusion on my PR. Getting release notes for the latest version still seems like an improvement I'm thinking. Do you agree?

viceice commented 3 months ago

Sure, but don't @ maintainers, We're subscribed to all topics.

renovatebot / renovate