DownloadLatestCUs.ps1 no longer works for me

[GeriOnline]

I just started a new project with the re-branded MSLab (v21.02.5). I tried to download the latest CUs with DownloadLatestCUs.ps1 however I cannot get the .MSU files. While debugging the script, I noticed that in line 49

> $latestCU=Get-LatestCumulativeUpdate -OperatingSystem Windows10 -Version $Version | Where-Object -Property Note -Like "Windows 10x64*" | Select-Object -First 1 #1903 has 2 objects

a NULL value is returned to $latestCU regardless of which Windows version I have selected in the outgrid. And of course then no CUs can be downloaded.

As far as I can remember there have been no problems with previous projects to run the script. Do you have any idea what goes wrong?

[jaromirk]

Hi, looks like Get-LatestCumulativeUpdate simply stopped working. Project is archived. I guess we will need to come up with different way to grab CU/SSU. @machv ideas? I was briefly looking into https://github.com/ryan-jan/MSCatalog

[jaromirk]

Looks like we can use API to search for updates https://api.msrc.microsoft.com/cvrf/v2.0/swagger/index ... here is list of SSUs https://msrc.microsoft.com/update-guide/en-us/vulnerability/ADV990001

[jaromirk]

hmm, we can parse latest KB from here

https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3 and https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd

this is the code of the web page where we can search for string (Azshci)

Azure Stack HCI, version 20H2

• Azure Stack HCI, version 20H2
And win10
Windows 10, version 20H2 and Windows Server, version 20H2 update history
• Windows 10, version 20H2 and Windows Server, version 20H2 update history
• March 29, 2021—KB5000842 (OS Builds 19041.906 and 19042.906) Preview
This is list of versions:, so we can simply search for "KB xxxx" after following text "Windows 10, version 20H2 and Windows Server, version 20H2" "Windows 10, version 2004 and Windows Server, version 2004" "Windows 10, version 1909 and Windows Server, version 1909" "Windows 10, version 1903 and Windows Server, version 1903" "Windows 10, version 1809, Windows Server, version 1809, and Windows Server 2019" "Windows 10, version 1803" "Windows 10, version 1709" "Windows 10, version 1703" "Windows 10, version 1607 and Windows Server 2016" "Windows 10, version 1511" "Azure Stack HCI, version 20H2" Using regex from these URLs: $win10Uri=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3 -UseBasicParsing $azurestackhciUri=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd -UseBasicParsing @machv - we should update this scenario I guess :) https://github.com/microsoft/MSLab/tree/master/Scenarios/Windows%20Update#check-if-servers-are-up-to-date
jaromirk commented 3 years ago

time to sleep...

jaromirk commented 3 years ago

Just woke up from limbo. The next step would be to download KB (using MSCatalog PS Module) ... Get-MSCatalogUpdate -Search "kb4595086" | Save-MSCatalogUpdate

machv commented 3 years ago

For SSUs I already created Azure Function that updates aka.ms links for latest available ones, those links should be:

• 2019 = https://aka.ms/ssu/2019
• 2016 = https://aka.ms/ssu/2016

It seems to be working, so these can be used in update scripts.

But for standard LCUs, I need to do more research about what APIs might be used as the ones in PowerShell module are not stable enough to return results always :-(

And Azure Stack HCI is a completely different story as it is using UUP for updates, instead of WinUpdate catalog?

jaromirk commented 3 years ago

@machv can you parse this for latest KB? :)

$win10=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3
$win10.ParsedHtml.getElementsByClassName("supLeftNavArticle") | select outertext

$azurestackhci=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd
$azurestackhci.ParsedHtml.getElementsByClassName("supLeftNavArticle") |select innerText

jaromirk commented 3 years ago

I made this POC to find KB

$win10=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3
$UpdatesList=($win10.ParsedHtml.getElementsByClassName("supLeftNavArticle") | Select-Object OuterText).OuterText
$titleswin10=$win10.ParsedHtml.getElementsByClassName("supLeftNavCategoryTitle") |select textcontent

$SelectedTitlesWin10 = ($titleswin10 | Out-GridView -OutputMode Multiple).textcontent
$CleanedList=@()
Foreach ($item in $UpdatesList){
    $CleanedList+=$item.Trim()
}
$preview=$true
foreach ($SelectedTitle in $SelectedTitlesWin10){
     $index=[array]::indexof($CleanedList,$SelectedTitle.TrimEnd())
     if ($preview){
        $string=$cleanedlist[$index+1]       
        "$SelectedTitle; $string"    
     }else{
        do {
        $index++
        $string=$cleanedlist[$index]       
        } until ($string -like "*)")
        "$SelectedTitle; $string"
    }
}



jaromirk commented 3 years ago

Ok, more progress...

$URLs=@()
$URLs+=@{Product="Windows10";URL="https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd"}
$URLs+=@{Product="AzureStackHCI";URL="https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3"}

$UpdatesList=@()
$Titles=@()

foreach ($url in $urls){
    $content=Invoke-WebRequest -Uri $url.url
    $UpdatesList+=($content.ParsedHtml.getElementsByClassName("supLeftNavArticle") | Select-Object OuterText).OuterText
    $Titles+=$content.ParsedHtml.getElementsByClassName("supLeftNavCategoryTitle") | Select-Object textcontent
}

#clean white space
$CleanedList=@()
Foreach ($item in $UpdatesList){
    $CleanedList+=$item.Trim()
}

#remove initial win10
$Titles=$Titles | Where-Object {$_.textcontent -notlike "*initial version*"}

#create output property
$Output=@()
foreach ($Title in $Titles){
    $Output += [PSCustomObject]@{
    "Product"="$(([Regex]::Match($title.textContent,'^.+?(?=,)')).Value)$(([Regex]::Match($title.textContent,'\ and\ .+?(?=,)')).value)"
    "Version"=([Regex]::Match($title.textContent,"(?<=version\ )\w{4}")).Value
    "NavCategoryTitle"=$title.textContent
    }
}

$SelectedTitles=$output | Out-GridView -OutputMode Multiple
$preview=$true

foreach ($SelectedTitle in $SelectedTitles){
     $index=[array]::indexof($CleanedList,$SelectedTitle.NavCategoryTitle.TrimEnd())
     if ($preview){
        #if preview is allowed, then just select last update
        $string=$cleanedlist[$index+1] 
     }else{
        #if AzSHCI
        if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
            do {
                $index++
                $string=$cleanedlist[$index]       
            } until ($string -like "*security update*")
        }
        #if preview is not allowed, then skip whatever does not end with ")" in case of windows server
        do {
            $index++
            $string=$cleanedlist[$index]       
        } until ($string -like "*)")
    }
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name Date -Value ([Regex]::Match($String,'\w*\ \d*,\ \d*')).Value
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name KB -Value ([Regex]::Match($String,'KB\d*')).Value
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name Build -Value ([Regex]::Match($String,'\d*\.\d*')).Value

    if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
        $PreviewString=([Regex]::Match($String,'preview')).Value
    }else{
        $PreviewString=([Regex]::Match($String,'(?<=\)\ ).*')).Value
    }
    If ($PreviewString){
        $SelectedTitle | Add-Member -MemberType NoteProperty -Name Preview -Value $PreviewString
    }else{
        $SelectedTitle | Add-Member -MemberType NoteProperty -Name Preview -Value $False
    }
    #Add SSU for Azure Stack HCI
    if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
        do {
        $index++
        $string=$cleanedlist[$index]       
        } until ($string -like "*servicing stack update*")
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name SSU -Value ([Regex]::Match($String,'KB\d*')).Value
    }
}



jaromirk commented 3 years ago

Here you go https://github.com/microsoft/MSLab/blob/dev/Tools/DownloadLatestCUs.ps1

@machv it's bit wild and it lacks SSUs for Windows Server - can you pls take a look and maybe use API for SSU?

GeriOnline commented 3 years ago

@jaromirk Downloaded and tested your new implementation from your dev branch. At least for CUs it seems to work. Found some smaller issues in the code:

lines 13/14 should be (typo): if($preview -eq "y"){ $preview = $true

line 67: Add culture info "en-US" instead of $null as 3rd parameter. Otherwise on non-English hosts an error is thrown (I'm working on a German Win10 System): $Date=([datetime]::ParseExact($DateString,$format,[Globalization.CultureInfo]::CreateSpecificCulture('en-US')))

line 127: Don't use the hard coded "C:\Temp" path. Instead use $folder from lines 7/8: $Path="$folder\$($item.Product)\$DateFolderName"

line 139: Add parameters for Save-MSCatalogUpdate at end of line: $update | Where-Object Title -Like "$($item.Version)" | Save-MSCatalogUpdate -UseBits -Destination $Path

So let's look further for the SSU challenge.

jaromirk commented 3 years ago

Cool! Thx! SSU should not be a big issue. Will take a look tonight. I need to think of a structure a bit... and remove unsupported systems (to clean it up a bit :)

jaromirk commented 3 years ago

Fixed all above, thx! Will now start cleaning it up + will add SSU

jaromirk commented 3 years ago

OK, I completely redesigned it. Much easier now. Give it a try :) https://github.com/microsoft/MSLab/blob/dev/Tools/DownloadLatestCUs.ps1

GeriOnline commented 3 years ago

I downloaded and tested your redesigned version. Seems to work at first glance. However I detected some strange things: When I pecify Preview = Yes and select an OS from the Out-Grid for which no preview CU exists (e.g. Windows Server 2016) a warning is displayed (I guess from the MSCatalog modul: WARNUNG: We did not find any results for Cumulative Update Preview for Windows Server 2016 for x64-based Systems) and the script jumps immediately to the end. This was not a problem with the last implementation from March 31st.

In addition I think this redesigned version couldn't be the final one. In my opinion it doesn't make much sense to hard code the search strings because each time new OS versions will appear you have to add / change the search string section in the script.

jaromirk commented 3 years ago

Hmm, Windows Server 2016 does not have previews. I'll add some logic to skip windows server 2016 preview and use regular

The problem I see with the first variation (parsing web) is that it is published there with delay. Adding one line once product is released is not a big deal.

Jaromir

jaromirk commented 3 years ago

How about this? https://github.com/microsoft/MSLab/commit/dac76443d231c48c0a8f8e45330f3ecdf07aa3eb - if preview=$true, it will download latest file (not exact). So it will include preview if there is preview newer than regular CU. Otherwise it will download regular CU.

GeriOnline commented 3 years ago

I tested your "simplified a bit" version with both Preview = True and Preview = False. Both cases apparently worked. It seems with that the issue for downloading latest CUs could be solvable

One more remark about the hard-coded search strings. I agree that it's not a big deal to add one line for a new product. But will we remember to do this when a new version is released let's say in 6 months? In your experience, how big is the delay before publication on the web?

jaromirk commented 3 years ago

No worries, I'll remember ;).

I think there is one day delay or so. So not big deal.

GeriOnline commented 3 years ago

Thanks for the support.

jaromirk commented 3 years ago

You are welcome!

• © Githubissues.
• Githubissues is a development platform for aggregating issues.

[jaromirk]

time to sleep...

[jaromirk]

Just woke up from limbo. The next step would be to download KB (using MSCatalog PS Module) ... Get-MSCatalogUpdate -Search "kb4595086" | Save-MSCatalogUpdate

[machv]

For SSUs I already created Azure Function that updates aka.ms links for latest available ones, those links should be:

• 2019 = https://aka.ms/ssu/2019
• 2016 = https://aka.ms/ssu/2016

It seems to be working, so these can be used in update scripts.

But for standard LCUs, I need to do more research about what APIs might be used as the ones in PowerShell module are not stable enough to return results always :-(

And Azure Stack HCI is a completely different story as it is using UUP for updates, instead of WinUpdate catalog?

[jaromirk]

@machv can you parse this for latest KB? :)

$win10=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3
$win10.ParsedHtml.getElementsByClassName("supLeftNavArticle") | select outertext

$azurestackhci=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd
$azurestackhci.ParsedHtml.getElementsByClassName("supLeftNavArticle") |select innerText

[jaromirk]

I made this POC to find KB

$win10=Invoke-WebRequest -Uri https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3
$UpdatesList=($win10.ParsedHtml.getElementsByClassName("supLeftNavArticle") | Select-Object OuterText).OuterText
$titleswin10=$win10.ParsedHtml.getElementsByClassName("supLeftNavCategoryTitle") |select textcontent

$SelectedTitlesWin10 = ($titleswin10 | Out-GridView -OutputMode Multiple).textcontent
$CleanedList=@()
Foreach ($item in $UpdatesList){
    $CleanedList+=$item.Trim()
}
$preview=$true
foreach ($SelectedTitle in $SelectedTitlesWin10){
     $index=[array]::indexof($CleanedList,$SelectedTitle.TrimEnd())
     if ($preview){
        $string=$cleanedlist[$index+1]       
        "$SelectedTitle; $string"    
     }else{
        do {
        $index++
        $string=$cleanedlist[$index]       
        } until ($string -like "*)")
        "$SelectedTitle; $string"
    }
}

[jaromirk]

Ok, more progress...

$URLs=@()
$URLs+=@{Product="Windows10";URL="https://support.microsoft.com/en-us/topic/release-notes-for-azure-stack-hci-64c79b7f-d536-015d-b8dd-575f01090efd"}
$URLs+=@{Product="AzureStackHCI";URL="https://support.microsoft.com/en-us/topic/windows-10-update-history-7dd3071a-3906-fa2c-c342-f7f86728a6e3"}

$UpdatesList=@()
$Titles=@()

foreach ($url in $urls){
    $content=Invoke-WebRequest -Uri $url.url
    $UpdatesList+=($content.ParsedHtml.getElementsByClassName("supLeftNavArticle") | Select-Object OuterText).OuterText
    $Titles+=$content.ParsedHtml.getElementsByClassName("supLeftNavCategoryTitle") | Select-Object textcontent
}

#clean white space
$CleanedList=@()
Foreach ($item in $UpdatesList){
    $CleanedList+=$item.Trim()
}

#remove initial win10
$Titles=$Titles | Where-Object {$_.textcontent -notlike "*initial version*"}

#create output property
$Output=@()
foreach ($Title in $Titles){
    $Output += [PSCustomObject]@{
    "Product"="$(([Regex]::Match($title.textContent,'^.+?(?=,)')).Value)$(([Regex]::Match($title.textContent,'\ and\ .+?(?=,)')).value)"
    "Version"=([Regex]::Match($title.textContent,"(?<=version\ )\w{4}")).Value
    "NavCategoryTitle"=$title.textContent
    }
}

$SelectedTitles=$output | Out-GridView -OutputMode Multiple
$preview=$true

foreach ($SelectedTitle in $SelectedTitles){
     $index=[array]::indexof($CleanedList,$SelectedTitle.NavCategoryTitle.TrimEnd())
     if ($preview){
        #if preview is allowed, then just select last update
        $string=$cleanedlist[$index+1] 
     }else{
        #if AzSHCI
        if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
            do {
                $index++
                $string=$cleanedlist[$index]       
            } until ($string -like "*security update*")
        }
        #if preview is not allowed, then skip whatever does not end with ")" in case of windows server
        do {
            $index++
            $string=$cleanedlist[$index]       
        } until ($string -like "*)")
    }
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name Date -Value ([Regex]::Match($String,'\w*\ \d*,\ \d*')).Value
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name KB -Value ([Regex]::Match($String,'KB\d*')).Value
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name Build -Value ([Regex]::Match($String,'\d*\.\d*')).Value

    if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
        $PreviewString=([Regex]::Match($String,'preview')).Value
    }else{
        $PreviewString=([Regex]::Match($String,'(?<=\)\ ).*')).Value
    }
    If ($PreviewString){
        $SelectedTitle | Add-Member -MemberType NoteProperty -Name Preview -Value $PreviewString
    }else{
        $SelectedTitle | Add-Member -MemberType NoteProperty -Name Preview -Value $False
    }
    #Add SSU for Azure Stack HCI
    if ($SelectedTitle.Product -like "*Azure Stack HCI*"){
        do {
        $index++
        $string=$cleanedlist[$index]       
        } until ($string -like "*servicing stack update*")
    $SelectedTitle | Add-Member -MemberType NoteProperty -Name SSU -Value ([Regex]::Match($String,'KB\d*')).Value
    }
}

[jaromirk]

Here you go https://github.com/microsoft/MSLab/blob/dev/Tools/DownloadLatestCUs.ps1

@machv it's bit wild and it lacks SSUs for Windows Server - can you pls take a look and maybe use API for SSU?

[GeriOnline]

@jaromirk Downloaded and tested your new implementation from your dev branch. At least for CUs it seems to work. Found some smaller issues in the code:

lines 13/14 should be (typo): if($preview -eq "y"){ $preview = $true

line 67: Add culture info "en-US" instead of $null as 3rd parameter. Otherwise on non-English hosts an error is thrown (I'm working on a German Win10 System): $Date=([datetime]::ParseExact($DateString,$format,[Globalization.CultureInfo]::CreateSpecificCulture('en-US')))

line 127: Don't use the hard coded "C:\Temp" path. Instead use $folder from lines 7/8: $Path="$folder\$($item.Product)\$DateFolderName"

line 139: Add parameters for Save-MSCatalogUpdate at end of line: $update | Where-Object Title -Like "$($item.Version)" | Save-MSCatalogUpdate -UseBits -Destination $Path

So let's look further for the SSU challenge.

[jaromirk]

Cool! Thx! SSU should not be a big issue. Will take a look tonight. I need to think of a structure a bit... and remove unsupported systems (to clean it up a bit :)

[jaromirk]

Fixed all above, thx! Will now start cleaning it up + will add SSU

[jaromirk]

OK, I completely redesigned it. Much easier now. Give it a try :) https://github.com/microsoft/MSLab/blob/dev/Tools/DownloadLatestCUs.ps1

[GeriOnline]

I downloaded and tested your redesigned version. Seems to work at first glance. However I detected some strange things: When I pecify Preview = Yes and select an OS from the Out-Grid for which no preview CU exists (e.g. Windows Server 2016) a warning is displayed (I guess from the MSCatalog modul: WARNUNG: We did not find any results for Cumulative Update Preview for Windows Server 2016 for x64-based Systems) and the script jumps immediately to the end. This was not a problem with the last implementation from March 31st.

In addition I think this redesigned version couldn't be the final one. In my opinion it doesn't make much sense to hard code the search strings because each time new OS versions will appear you have to add / change the search string section in the script.

[jaromirk]

Hmm, Windows Server 2016 does not have previews. I'll add some logic to skip windows server 2016 preview and use regular

The problem I see with the first variation (parsing web) is that it is published there with delay. Adding one line once product is released is not a big deal.

Jaromir

[jaromirk]

How about this? https://github.com/microsoft/MSLab/commit/dac76443d231c48c0a8f8e45330f3ecdf07aa3eb - if preview=$true, it will download latest file (not exact). So it will include preview if there is preview newer than regular CU. Otherwise it will download regular CU.

[GeriOnline]

I tested your "simplified a bit" version with both Preview = True and Preview = False. Both cases apparently worked. It seems with that the issue for downloading latest CUs could be solvable

One more remark about the hard-coded search strings. I agree that it's not a big deal to add one line for a new product. But will we remember to do this when a new version is released let's say in 6 months? In your experience, how big is the delay before publication on the web?

[jaromirk]

No worries, I'll remember ;).

I think there is one day delay or so. So not big deal.

[GeriOnline]

Thanks for the support.

[jaromirk]

You are welcome!