Add option to always include changesets

RsZoli commented 1 year ago

Azure DevOps Extensions

Generate Release Notes (Node Cross Platform)

Is your feature request related to a problem? Please describe.

I would like to include the changesets associated to a build, even there is nothing new, also, since i'm running it from the release pipeline, i would like to have the changesets every time i run the release of the same build!

The old version did this just fine without any setting needed! :(

Describe the solution you'd like

A checkbox would be fine, to always include changesets, no matter what!

Describe alternatives you've considered

No response

Additional context

No response

rfennell commented 1 year ago

This task has only ever returned the list of changes between two builds, so I am unclear what you mean by

'The old version did this just fine without any setting needed! :('.

Which version of this task were you using?

RsZoli commented 1 year ago

@rfennell thank you for your attention to this! sorry my english is not very good, so i will try to explain it via an example!

Let's say i make a build, with 2 changesets, it will be build 1, and in the release pipeline i want to make a release notes to send out in email, with that 2 changesets. But after that, i want to make a new build without any new changesets, it will be build 2! So there is no difference between the two builds, so no changesets would be gathered by your tool in the release pipeline!

Also, if i want to re-release the build 1, the same thing happens!

I used the V2 beforehand, but i ran into some encoding issue which i could not resolve, and it just doesnt work anymore!

rfennell commented 1 year ago

What you describe is a common problem with use of this task in Classic Releases

I think you need to use the overrideBuildReleaseId parameter to fix the version you wish to compare against.

As detailed in the WIKI/Readme

Possible sets of parameters depending on your usage are summarized below

Option	Multi Stage YAML	Classic Build/Release
Generate notes for just the current build	Requires `checkstages=false` parameter	Run inside the build
Generate notes since the last successful release. Option 1. Place the task in a stage that is only run when you wish to generate release notes. Usually this will be guarded by branch based filters or manual approvals.	Requires `checkstages=true` parameter	Run inside the release. Supported and you can override the stage name used for comparison using the `overrideStageName` parameter
Generate notes since the last successful release. Option 2. Set the task to look back for the last successful build that has a given tag	Requires `checkstages=true` and the `tags` parameters	Not supported
Generate notes since the last successful release. Option 3. Override the build that the task uses for comparison with a fixed value	Requires `checkstages=true` and the `overrideBuildReleaseId` parameters	Run inside the release. Requires the `overrideBuildReleaseId` parameter

You might find this video helpful too

RsZoli commented 1 year ago

@rfennell Thank you, but i need this to be automated! :(

i'm using the 3.9.10 version, could you help me resolve this one issue, i would be happy to continue to use that!

"2023-04-05T07:09:15.2017119Z ##[error]A parameter cannot be found that matches parameter name 'Encoding'."

it was working perfectly like 2 weeks ago, maybe our devops environment has changed, but i have no control over that :(

thank you!

rfennell commented 1 year ago

As to automating this look back, I have no plans to alter the current functionality.

As to your error can you provide the full log so I can see where the error is occurring

RsZoli commented 1 year ago

@rfennell of course, here is the full output:

2023-04-05T07:09:11.2630554Z ##[section]Starting: Generate release notes (PowerShell) 2023-04-05T07:09:11.2735599Z ============================================================================== 2023-04-05T07:09:11.2735906Z Task : (deprecated) Generate Release Notes for Pipeline Builds or Releases (PowerShell) 2023-04-05T07:09:11.2736298Z Description : (deprecated) Task is now deprecated. Please consider my Cross Platform Version https://marketplace.visualstudio.com/items?itemName=richardfennellBM.BM-VSTS-XplatGenerateReleaseNotes 2023-04-05T07:09:11.2736627Z Version : 3.9.10 2023-04-05T07:09:11.2736755Z Author : Black Marble 2023-04-05T07:09:11.2737730Z Help : Version: 3.9.10. More Information 2023-04-05T07:09:11.2738014Z ============================================================================== 2023-04-05T07:09:15.2017119Z ##[error]A parameter cannot be found that matches parameter name 'Encoding'. 2023-04-05T07:09:15.2270138Z ##[section]Finishing: Generate release notes (PowerShell)

rfennell commented 1 year ago

Ah, I see you are using the 'Generate release notes (PowerShell)' based release notes, and not the 'Generate Release Notes (Node Cross Platform)' task as this issue was logged under. So, you can ignore all my previous answers as I was discussing a completely different task.

I ceased support for the 'Generate release notes (PowerShell)' in Jan 2022, and focused all my efforts on the far for capable 'Generate Release Notes (Node Cross Platform)' task. This deprecation is noted in the task log.

Task : (deprecated) Generate Release Notes for Pipeline Builds or Releases (PowerShell) Description : (deprecated) Task is now deprecated. Please consider my Cross Platform Version https://marketplace.visualstudio.com/items?itemName=richardfennellBM.BM-VSTS-XplatGenerateReleaseNotes

You can of course pull down the source and fork your own version of this old task, but will not be publishing any updates.

If you do that, I suspect you will find the the issue is that your Azure DevOps agent runner itself has been updated to from 2.x to 3.x. as the V3 agents are now generally available. This could have changed the underlying installed version of PowerShell and my guess is that a function's parameters has changed.

RsZoli commented 1 year ago

@rfennell Yes, i'm using the old, depreceted one! I really would like to use your latest, but it does not work for me! :(

I'm not able to create and run the plugin for myself, however i can run powershell scripts in our pipeline, so if you would be so kind and give me the old, deprecated script which i could use inline, i would be most thankful!

rfennell commented 1 year ago

The code for all my tasks is freely available in this repo. They are all in the extensions folder.

The one you want is generatereleasenotes one. You should be able to extract that task as a script.

RsZoli commented 1 year ago

@rfennell Thank you again, for your help!

I tried the scripts, but i have the following error:

2023-04-12T09:31:56.5530397Z D:\Agent0_work\r243\a\Demo\drop\GenerateReleaseNotes.ps1 : The term 'Get-VstsInput' is not recognized as the nam 2023-04-12T09:31:56.5531034Z e of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, v 2023-04-12T09:31:56.5531370Z erify that the path is correct and try again. 2023-04-12T09:31:56.5531730Z At D:\Agent0_work_temp\abcbf565-2a18-4732-b966-e53b0ac02780.ps1:3 char:1 2023-04-12T09:31:56.5532086Z + . 'D:\Agent0_work\r243\a\Demo\drop\GenerateReleaseNotes.ps1' 2023-04-12T09:31:56.5532390Z + ~~~~~~~~~~~~~~~ 2023-04-12T09:31:56.5532983Z + CategoryInfo : ObjectNotFound: (Get-VstsInput:String) [GenerateReleaseNotes.ps1], CommandNotFoundExcept 2023-04-12T09:31:56.5533557Z ion 2023-04-12T09:31:56.5533796Z + FullyQualifiedErrorId : CommandNotFoundException,GenerateReleaseNotes.ps1 2023-04-12T09:31:56.5534038Z
2023-04-12T09:31:56.6082902Z ##[error]PowerShell exited with code '1'.

The Get-VstsInput is not a built in command?

rfennell commented 1 year ago

All Azure DevOps PowerShell based tasks are reliant on the Azure DevOps PowerShell SDK which is shipped in this folder with the task. I would suspect you will have to load this module, or replace it's functions.

For example Get-VstsInput gets an Azure DevOps task parameter value, but if you are just running the code as a script this could be replaced by a simple PowerShell Param block

RsZoli commented 1 year ago

@rfennell Thank you, i'm really starting to get the jist of it!

But as always, new problems arose...

2023-04-12T13:00:02.3109382Z VERBOSE: Using SystemVssConnection personal access token 2023-04-12T13:00:02.3998910Z Invoke-GetCommand : The term 'Get-VstsEndpoint' is not recognized as the name of a cmdlet, function, script file

I guess i could replace this with a prameter as well, but how can i get the needed information for this?

Thank you!

rfennell commented 1 year ago

Your best bet will be to replace the function

function Invoke-GetCommand {
    [CmdletBinding()]
    param
    (
        $uri,
        $usedefaultcreds
    )

    # When debugging locally, this variable can be set to use personal access token.
    $debugpat = $env:PAT

    [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
    $webclient = new-object System.Net.WebClient
    $webclient.Encoding = [System.Text.Encoding]::UTF8

    if ([System.Convert]::ToBoolean($usedefaultcreds) -eq $true) {
        Write-Verbose "Using default credentials"
        $webclient.UseDefaultCredentials = $true
    }
    elseif ([string]::IsNullOrEmpty($debugpat) -eq $false) {
        $encodedPat = [System.Convert]::ToBase64String([System.Text.Encoding]::UTF8.GetBytes(":$debugpat"))
        $webclient.Headers.Add("Authorization", "Basic $encodedPat")
    }
    else {
        # Write-Verbose "Using SystemVssConnection personal access token"
        Write-Verbose "Using SystemVssConnection personal access token"
        $vstsEndpoint = Get-VstsEndpoint -Name SystemVssConnection -Require
        $webclient.Headers.Add("Authorization" ,"Bearer $($vstsEndpoint.auth.parameters.AccessToken)")
    }

    #write-verbose "REST Call [$uri]"
    $webclient.DownloadString($uri)
}

With one that does not rely on the Azure DevOps Pipeline SDK

function Get-WebClient {
    param
    (
        [string] $uri,
        [string]$pat,
        [string]$ContentType = "application/json"
    )

    $wc = New-Object System.Net.WebClient
    $wc.Headers["Content-Type"] = $ContentType
    $pair = ":${pat}"
    $bytes = [System.Text.Encoding]::ASCII.GetBytes($pair)
    $base64 = [System.Convert]::ToBase64String($bytes)
    $wc.Headers.Add("Authorization", "Basic $base64");

    #write-verbose "REST Call [$uri]"
    $wc.DownloadString($uri)

  }

You might find this repo of Azure DevOps PowerShell scripts are of use, particularly the REST folder.

RsZoli commented 1 year ago

@rfennell Well, thank you very much again, i have managed to run the script finally!

The current situation is:

2023-04-12T15:17:47.5953370Z VERBOSE: Discovered [1] releases for processing after checking a total of [1] releases 2023-04-12T15:17:47.6015744Z VERBOSE: Found a potential list of 0 artifact 2023-04-12T15:17:47.6020639Z VERBOSE: Filtering on primary artifact only [false] 2023-04-12T15:17:47.6025760Z VERBOSE: Filtering on trigger artifact only [false] 2023-04-12T15:17:47.6033349Z VERBOSE: Found 0 build artifacts to check for builds that fall in range

It was a new build with a new changeset, jut to make sure, but the script did not find any artifacts! However, in the json file there is a artifactSourceDefinitionUrl:

"artifacts": [ { "sourceId": "5d164357-b01b-4975-9f3b-b694a1b0b9b4:905", "type": "Build", "alias": "Demo", "definitionReference": { "artifactSourceDefinitionUrl": { "id": "https://demo.hu/_permalink/_build/index?collectionId=82ac4b58-af43-45df-bed0-cfb4ec1f3aed&projectId=5d164357-b01b-4975-9f3b-b694a1b0b9b4&definitionId=905", "name": "" }

And when i check the url, it lists 3 builds that included in the release with the 3 changesets i'm looking for!

Maybe there is some settings that i have left out which is automatically set when i use the GUI to run the task?

Currently, i'm using these:

$outputfile = $env:SYSTEM_DEFAULTWORKINGDIRECTORY + "\" + $env:SOURCEALIAS + "\drop\releasenotes.md" $outputvariablename = "" $templatefile = $env:SYSTEM_DEFAULTWORKINGDIRECTORY + "\" + $env:SOURCEALIAS + "\drop\GenerateReleaseNotes.template" $inlinetemplate = "" $templateLocation = "File" $usedefaultcreds = "true" $generateForOnlyPrimary = "false" $generateForOnlyTriggerArtifact = "false" $generateForCurrentRelease = "true" $overrideStageName = "" $emptySetText = "" $maxChanges= "" $maxWi = "" $wiFilter = "" $wiStateFilter= "" $showParents = "false" $appendToFile = "false" $unifiedList = "" $buildTags = ""

RsZoli commented 1 year ago

@rfennell Hey, i'm sorry to bother you again, since i could not fix this, i decided to use the new version of your plugin, but it still doesnt give me anything!

Two seperate builds, with two seperate releases after each other, the latest run clearly indicates that there are workitems changed:

2023-04-27T07:17:42.3342839Z Found 1 artifacts for most recent successful release 2023-04-27T07:17:42.3343096Z Looking at artifact [Demo] 2023-04-27T07:17:42.3343282Z Artifact type [Build] 2023-04-27T07:17:42.3343472Z Build Definition ID [905] 2023-04-27T07:17:42.3343669Z Build Number: [1.0.0-12] 2023-04-27T07:17:42.3343851Z Is Primary: [true] 2023-04-27T07:17:42.3344091Z Looking for the [Demo] in the most recent successful release [Release-28] 2023-04-27T07:17:42.3344405Z Found artifact [Demo] with build number [1.0.0-11] in release [Release-28] 2023-04-27T07:17:42.3344756Z Checking what commits and workitems have changed from [1.0.0-11][ID 47374] => [1.0.0-12] [ID 47442] 2023-04-27T07:17:42.3345137Z Defaulting on the workaround for build API limitation (see issue #349 set 'ReleaseNotes.Fix349=false' to disable) 2023-04-27T07:17:42.3345479Z Using workaround for build API limitation (see issue #349) 2023-04-27T07:17:42.4365825Z Getting test associated with the latest build [47442] 2023-04-27T07:17:42.4722684Z Adding the build [47442] and its associations to the unified results object 2023-04-27T07:17:42.4723158Z Detected 0 workitems between the current build and the last successful one

What am i doing wrong?

rfennell commented 1 year ago

Can you provide the full log so we can see what build/releases numbers it has selected.

I doubt you are doing anything wrong, it is just that if using Classic Releases you only see associates on the first deployment of a given build

RsZoli commented 1 year ago

@rfennell Here you go, it was a first deployment:

2023-04-27T07:17:27.6556169Z ##[section]Starting: Generate Release Notes based on Release Comparison API 2023-04-27T07:17:27.6649839Z ============================================================================== 2023-04-27T07:17:27.6650138Z Task : Generate Release Notes (Crossplatform) 2023-04-27T07:17:27.6650398Z Description : Generates a release notes file in a format of your choice from the build or release history 2023-04-27T07:17:27.6650609Z Version : 4.6.2 2023-04-27T07:17:27.6650759Z Author : Black Marble 2023-04-27T07:17:27.6651822Z Help : Version: 4.6.2. More Information 2023-04-27T07:17:27.6652166Z ============================================================================== 2023-04-27T07:17:41.6087963Z Creating Azure DevOps API connections for https://demo.hu/DEMOCollection/ with 'allowRetries' set to 'true' and 'maxRetries' count to '20' 2023-04-27T07:17:41.6092618Z Creating the credential handler from the OAUTH token 2023-04-27T07:17:42.2375088Z Getting the current release details 2023-04-27T07:17:42.2762074Z Show associated items for primary artifact only is set to false 2023-04-27T07:17:42.2762454Z Getting the Environment Id 2023-04-27T07:17:42.2762684Z Identified [dev] as having id [690] 2023-04-27T07:17:42.2765202Z Finding successful deployments 2023-04-27T07:17:42.2955784Z Found 6 releases to consider 2023-04-27T07:17:42.2958479Z Finding the last successful release Release-28 2023-04-27T07:17:42.2958727Z Getting all artifacts in the current release... 2023-04-27T07:17:42.2960432Z Found 1 artifacts for current release 2023-04-27T07:17:42.3342142Z Getting all artifacts in the most recent successful release [Release-28]... 2023-04-27T07:17:42.3342839Z Found 1 artifacts for most recent successful release 2023-04-27T07:17:42.3343096Z Looking at artifact [Demo] 2023-04-27T07:17:42.3343282Z Artifact type [Build] 2023-04-27T07:17:42.3343472Z Build Definition ID [905] 2023-04-27T07:17:42.3343669Z Build Number: [1.0.0-12] 2023-04-27T07:17:42.3343851Z Is Primary: [true] 2023-04-27T07:17:42.3344091Z Looking for the [Demo] in the most recent successful release [Release-28] 2023-04-27T07:17:42.3344405Z Found artifact [Demo] with build number [1.0.0-11] in release [Release-28] 2023-04-27T07:17:42.3344756Z Checking what commits and workitems have changed from [1.0.0-11][ID 47374] => [1.0.0-12] [ID 47442] 2023-04-27T07:17:42.3345137Z Defaulting on the workaround for build API limitation (see issue #349 set 'ReleaseNotes.Fix349=false' to disable) 2023-04-27T07:17:42.3345479Z Using workaround for build API limitation (see issue #349) 2023-04-27T07:17:42.4365825Z Getting test associated with the latest build [47442] 2023-04-27T07:17:42.4722684Z Adding the build [47442] and its associations to the unified results object 2023-04-27T07:17:42.4723158Z Detected 0 workitems between the current build and the last successful one 2023-04-27T07:17:42.4723576Z Found 0 tests associated with the build [47442] adding any not already in the global test list to the list 2023-04-27T07:17:42.4866177Z Found 0 manual tests associated with the build [47442] adding any not already in the global test list to the list 2023-04-27T07:17:42.4867652Z 2023-04-27T07:17:42.5392507Z Removing duplicate Commits from master list 2023-04-27T07:17:42.5395658Z Expanding the truncated commit messages... 2023-04-27T07:17:42.5395906Z Expanded truncated commit messages 0 2023-04-27T07:17:42.5396132Z Leaving CS in default order as returned by API 2023-04-27T07:17:42.5396389Z Find any WorkItems linked from GitHub using the AB#123 format 2023-04-27T07:17:42.5398160Z Adding 0 found using AB#123 links in GitHub comments 2023-04-27T07:17:42.5398475Z Removing duplicate WorkItems from master list 2023-04-27T07:17:42.5398752Z Getting tests linked to WorkItems 2023-04-27T07:17:42.5399769Z Checking for user defined WIQL query 2023-04-27T07:17:42.5401334Z No WIQL query parameters defined 2023-04-27T07:17:42.5402481Z Leaving WI in default order as returned by API 2023-04-27T07:17:42.5525669Z The default artifact for the release was not linked to an Azure DevOps Git Repo Pull Request 2023-04-27T07:17:42.5526169Z Getting all completed Azure DevOps Git Repo PRs in the Team Project IFP 2023-04-27T07:17:42.5526406Z Getting associated PRs 2023-04-27T07:17:44.7616471Z Found 3164 Azure DevOps PRs in the repo 2023-04-27T07:17:44.7617140Z Enriching known Pull Requests 2023-04-27T07:17:44.7619105Z Total Builds: [1] 2023-04-27T07:17:44.7619684Z Total Commits: [0] 2023-04-27T07:17:44.7620056Z Total Workitems: [0] 2023-04-27T07:17:44.7620378Z Total Related Workitems (Parent/Children): [0] 2023-04-27T07:17:44.7620745Z Total Release Tests: [0] 2023-04-27T07:17:44.7620996Z Total Tests: [0] 2023-04-27T07:17:44.7621566Z Total Manual Test Runs: [0] 2023-04-27T07:17:44.7621846Z Total Manual Test Configurations: [0] 2023-04-27T07:17:44.7622157Z Total Pull Requests: [0] 2023-04-27T07:17:44.7622447Z Total Indirect Pull Requests: [0] 2023-04-27T07:17:44.7622718Z Total Associated Test WI: [0] 2023-04-27T07:17:44.7622970Z Total Published Artifacts: [0] 2023-04-27T07:17:44.7623202Z Total Consumed Artifacts: [0] 2023-04-27T07:17:44.7623423Z Total WIQL Workitems: [0] 2023-04-27T07:17:44.7848779Z Generating the release notes, the are 1 template(s) to process 2023-04-27T07:17:44.7849712Z Using in-line template 2023-04-27T07:17:45.8862399Z Loading handlebars-helpers extension 2023-04-27T07:17:55.8041927Z Completed processing template 2023-04-27T07:17:55.8042573Z Writing output file D:\Agent0_work\r243\a\Demo\drop\releasenotes.md 2023-04-27T07:17:55.8045899Z Finished writing output file D:\Agent0_work\r243\a\Demo\drop\releasenotes.md 2023-04-27T07:17:55.8046299Z Output variable 'undefined' set to value of first generated release notes 2023-04-27T07:17:55.8858046Z ##[section]Finishing: Generate Release Notes based on Release Comparison API

rfennell commented 1 year ago

I can see you are running in a Classic Release (not a Classic Build or multi-stage YAML), this reduces what the task can do, due to the APIs available, and makes testing harder

So your flow is

The task found 6 past releases to consider, the last successful one was Release-28, so this is the base for comparison
It looks for artifacts on the current release, and found 1 [Demo]
It then compares the version of [Demo] released in Release-28 and the the current release to see any new WI or CS added, Checking what commits and workitems have changed from [1.0.0-11][ID 47374] => [1.0.0-12] [ID 47442]
It find none, hence nothing in the release notes.

As I said testing Classic releases it awkward as you really have to trigger new builds (with new Commits & WI) for every test.

RsZoli commented 1 year ago

@rfennell I have changesets

159529 and 159731 in build 1.0.0-11 for Release-28, 159821 and 159823 in build 1.0.0-12 for Release-29

So it must be some mistake on my end if the generator finds nothing?

rfennell commented 1 year ago

In the end you will find that basically all the task does is make calls to the APIs

You can test these calls in a browser or Postman to see what is returned.

After that, all I can suggest is you experiment with the command line tester and step through the code being run.

RsZoli commented 1 year ago

@rfennell Well, the Build 11 has 47374 and the Build 12 has 47442 id.

This one: _apis/build/workitems?fromBuildId=47374&toBuildId=47442 returns abolutely nothing:

{"count":0,"value":[]}

However this one: _apis/build/changes?fromBuildId=47374&toBuildId=47442 has what i need:

{ "count": 2, "value": [ { "id": "C159821", "message": "!Android belső frissítés", "type": "TfsVersionControl", "author": { "displayName": "Aros Zoltán", "id": "63d966a0-82c8-49d5-a068-85af1366bc9d", "descriptor": "win.Uy0xLTUtMjEtMTQ4MjQ3NjUwMS0xMjc1MjEwMDcxLTcyNTM0NTU0My0xMjQ5NDU" }, "timestamp": "2023-04-26T13:54:52.433Z", "location": "/_apis/tfvc/changesets/159821" }, { "id": "C159733", "message": "!Automatic application versioning check-back from pipeline", "type": "TfsVersionControl", "author": { "displayName": "Project Collection Build Service (DEMOCollection)", "id": "5be81851-307c-4a4d-b06f-f0dbfea8d4d5", "descriptor": "svc.ZTE2ZjIzOGQtMDJmOS00MGNhLTliOWUtYzJhNjI4NDRjOTdhOkJ1aWxkOjgyYWM0YjU4LWFmNDMtNDVkZi1iZWQwLWNmYjRlYzFmM2FlZA" }, "timestamp": "2023-04-23T12:06:45.537Z", "location": "/_apis/tfvc/changesets/159733" } ] }

rfennell commented 1 year ago

The first query is getting associated WI between the builds, the second is the changeset difference between the builds

RsZoli commented 1 year ago

@rfennell Thanks! so if the API endpoints give back the data, why does your generator give me back nothing?

rfennell commented 1 year ago

What does the log show if you run a pipeline with the variable system.debug=true or via the test console

With debug on it shown where all the data is gather from

rfennell commented 1 year ago

A new release has been i 2.7.15 that now provides the option that is a retry is needed (after the commit is made locally when the push fails) that you can (default as before) try a get pull or a pull pull --rebase=true

To enable this option you need to set the retrymode parameter and set your branch name

  - task: richardfennellBM.BM-VSTS-WIKIUpdater-Tasks.WikiUpdaterTask.WikiUpdaterTask@2
    displayName: 'Git based WIKI Updater - GitHub folder'
    inputs:
      repo: github.com/rfennell/demorepo.wiki
      filename: TestFolder/${{parameters.platform}}-PageFromWikiUpdater.md
      message: 'Update from Release - $(Release.ReleaseName)'
      contents: |
        Header
        ---
        $(OutputText)
        ---
        Footer
        message: 'Update from Release - $(Release.ReleaseName)'
      gitname: builduser
      gitemail: 'build@demo'
      RetryMode: Rebase
      branch: main
      user: rfennell
      password: '$(githubPAT)'
      localpath: '$(System.DefaultWorkingDirectory)\githubrepo'

I am far from sure this change will actually help, and I have struggled to test it, so let me know how it goes

rfennell / AzurePipelines