search

aaronparker / evergreen

Create evergreen Windows image build pipelines with the latest version numbers and download URLs for common applications
http://stealthpuppy.com/evergreen/
MIT License
375 stars 64 forks source link

[Bug]: Recent change to Resolve-SystemNetWebRequest appears to have broken URL's returned for Sourceforge based Apps #669

Closed chezzer64 closed 4 months ago

chezzer64 commented 5 months ago

What happened?

When running the latest version of Evergreen module (2405.959) URI's returned for Sourceforge based apps seem to be incorrect. I think this may be as a result of recent changes to Resolve-SystemNetWebRequest to add UserAgent? The Sourceforge mirror is not getting resolved. Please see verbose output below for comparison.

Version

2405.959

What PowerShell edition/s are you running Evergreen on?

Windows PowerShell

Which operating system/s are you running Evergreen on?

Windows 10+

Have you reviewed the documentation?

Verbose output

Get-EvergreenApp -Name WinSCP -Verbose
VERBOSE: Function path: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2405.959\Apps\Get-WinSCP.ps1
VERBOSE: Function exists: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2405.959\Apps\Get-WinSCP.ps1.
VERBOSE: Dot sourcing: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2405.959\Apps\Get-WinSCP.ps1.
VERBOSE: Get-FunctionResource: read application resource strings from [C:\Program Files\WindowsPowerShell\Modules\Evergreen\2405.959\Manifests\WinSCP.json]
VERBOSE: Calling: Get-WinSCP.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: MaximumRedirection: 2.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: DisableKeepAlive: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.2478.67.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Method: Default.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Uri: https://sourceforge.net/projects/winscp/best_release.json.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: ContentType: application/json; charset=utf-8.
VERBOSE: GET https://sourceforge.net/projects/winscp/best_release.json with 0-byte payload
VERBOSE: received 3296-byte response of content type application/json; charset=utf-8
VERBOSE: Get-SourceForgeRepoRelease: Validating SourceForge release object.
VERBOSE: Get-SourceForgeRepoRelease: Property validation succeeded.
VERBOSE: Get-SourceForgeRepoRelease: Capture version number from: /WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe.
VERBOSE: Get-SourceForgeRepoRelease: Found filename: [WinSCP-6.3.3-Setup.exe].
VERBOSE: Get-SourceForgeRepoRelease: Found folder:   [/WinSCP/6.3.3/].
VERBOSE: Get-SourceForgeRepoRelease: Found version:  [6.3.3].
VERBOSE: Resolve-SystemNetWebRequest: Attempting to resolve: http://downloads.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe?ts=gAAAAABmMgl9a1Zt1kVMVnF_CIMgT-ibMKhz3cOXDbV7AWpXZ4HkCBN8q_KTi3Wgx53aLrwoSeUp7xdQIjPMWOIyiSczc9dc9g%3D%3D.
VERBOSE: Resolve-SystemNetWebRequest: Resolved to: [https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/download?use_mirror=cytranet-dal].
VERBOSE: Get-SourceForgeRepoRelease: Resolve mirror to: sourceforge.net.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: MaximumRedirection: 2.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: DisableKeepAlive: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36 Edg/124.0.2478.67.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Method: Default.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Uri: https://sourceforge.net/projects/winscp/rss?path=/WinSCP/6.3.3/.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: ContentType: application/rss+xml; charset=utf-8.
VERBOSE: GET https://sourceforge.net/projects/winscp/rss?path=/WinSCP/6.3.3/ with 0-byte payload
VERBOSE: received 7236-byte response of content type application/rss+xml; charset=utf-8
VERBOSE: Get-SourceForgeRepoRelease: found 2 items.
VERBOSE: Get-SourceForgeRepoRelease: matched: https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3.msi/download.
VERBOSE: Get-SourceForgeRepoRelease: file: /WinSCP/6.3.3/WinSCP-6.3.3.msi
VERBOSE: Get-Architecture: Architecture not found in https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/WinSCP-6.3.3.msi, defaulting to x86.
VERBOSE: Get-SourceForgeRepoRelease: matched: https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/download.
VERBOSE: Get-SourceForgeRepoRelease: file: /WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe
VERBOSE: Get-Architecture: Architecture not found in https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/WinSCP-6.3.3-Setup.exe, defaulting to x86.

Version      : 6.3.3
Architecture : x86
Type         : exe
Size         : 11588560
Md5          : 91dbb70daebfd07c13cc7bf5bb8898a0
URI          : https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/WinSCP-6.3.3-Setup.exe

Version      : 6.3.3
Architecture : x86
Type         : msi
Size         : 27959296
Md5          : 1289dddf9fceb42565a4d3bf70862427
URI          : https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/WinSCP-6.3.3.msi
chezzer64 commented 5 months ago

Here is a sample of the correct URI's being returned when running Evergreen 2404.928

Get-EvergreenApp -name winscp -verbose
VERBOSE: Function path: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2404.928\Apps\Get-winscp.ps1
VERBOSE: Function exists: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2404.928\Apps\Get-winscp.ps1.
VERBOSE: Dot sourcing: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2404.928\Apps\Get-winscp.ps1.
VERBOSE: Get-FunctionResource: read application resource strings from [C:\Program Files\WindowsPowerShell\Modules\Evergreen\2404.928\Manifests\winscp.json]
VERBOSE: Calling: Get-winscp.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: MaximumRedirection: 2.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: DisableKeepAlive: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0..
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Method: Default.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Uri: https://sourceforge.net/projects/winscp/best_release.json.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: ContentType: application/json; charset=utf-8.
VERBOSE: GET https://sourceforge.net/projects/winscp/best_release.json with 0-byte payload
VERBOSE: received 3296-byte response of content type application/json; charset=utf-8
VERBOSE: Get-SourceForgeRepoRelease: Validating SourceForge release object.
VERBOSE: Get-SourceForgeRepoRelease: Property validation succeeded.
VERBOSE: Get-SourceForgeRepoRelease: Capture version number from: /WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe.
VERBOSE: Get-SourceForgeRepoRelease: Found filename: [WinSCP-6.3.3-Setup.exe].
VERBOSE: Get-SourceForgeRepoRelease: Found folder:   [/WinSCP/6.3.3/].
VERBOSE: Get-SourceForgeRepoRelease: Found version:  [6.3.3].
VERBOSE: Resolve-SystemNetWebRequest: Attempting to resolve:
http://downloads.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe?ts=gAAAAABmMgl9a1Zt1kVMVnF_CIMgT-ibMKhz3cOXDbV7AWpXZ4HkCBN8q_KTi3Wgx53aLrwoSeUp7xdQIjPMWOIyiSczc9dc9g%3D%3D.
VERBOSE: Resolve-SystemNetWebRequest: Resolved to: [http://kumisystems.dl.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe?viasf=1].
VERBOSE: Get-SourceForgeRepoRelease: Resolve mirror to: kumisystems.dl.sourceforge.net.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: MaximumRedirection: 2.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: DisableKeepAlive: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0..
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Method: Default.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Uri: https://sourceforge.net/projects/winscp/rss?path=/WinSCP/6.3.3/.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: ContentType: application/rss+xml; charset=utf-8.
VERBOSE: GET https://sourceforge.net/projects/winscp/rss?path=/WinSCP/6.3.3/ with 0-byte payload
VERBOSE: received 7236-byte response of content type application/rss+xml; charset=utf-8
VERBOSE: Get-SourceForgeRepoRelease: found 2 items.
VERBOSE: Get-SourceForgeRepoRelease: matched: https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3.msi/download.
VERBOSE: Get-SourceForgeRepoRelease: file: /WinSCP/6.3.3/WinSCP-6.3.3.msi
VERBOSE: Get-Architecture: Architecture not found in https://kumisystems.dl.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3.msi, defaulting to x86.
VERBOSE: Get-SourceForgeRepoRelease: matched: https://sourceforge.net/projects/winscp/files/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe/download.
VERBOSE: Get-SourceForgeRepoRelease: file: /WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe
VERBOSE: Get-Architecture: Architecture not found in https://kumisystems.dl.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe, defaulting to x86.

Version      : 6.3.3
Architecture : x86
Type         : exe
Size         : 11588560
Md5          : 91dbb70daebfd07c13cc7bf5bb8898a0
URI          : https://kumisystems.dl.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3-Setup.exe

Version      : 6.3.3
Architecture : x86
Type         : msi
Size         : 27959296
Md5          : 1289dddf9fceb42565a4d3bf70862427
URI          : https://kumisystems.dl.sourceforge.net/project/winscp/WinSCP/6.3.3/WinSCP-6.3.3.msi
DanGough commented 5 months ago

Hmm, I may have to modify my PsDownload code to try a blank user agent first:

image

aaronparker commented 5 months ago

I've updated Resolve-SystemNetWebRequest and Get-SourceForgeRepoRelease to not send a user agent, so it should now resolve the right URL

Version Architecture Type Size    Md5                              URI
------- ------------ ---- ----    ---                              ---
23.01   x86          exe  584704  58fc6de6c4e5d2fda63565d54feb9e75 https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7zr.exe
23.01   x64          msi  1933312 7025502de20d7b39fb06870ab06d015b https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301-x64.msi
23.01   x64          exe  1589510 e5788b13546156281bf0a4b38bdd0901 https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301-x64.exe
23.01   ARM64        exe  1527518 3c5917f4da614ef892f055c697744b77 https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301-arm64.exe
23.01   ARM32        exe  1596691 d98f54ca52882376fd8b39b29628a90a https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301-arm.exe
23.01   x86          msi  1501184 49780e0517aa248e7fd90f50674a2645 https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301.msi
23.01   x86          exe  1301195 1cfb215a6fb373ac33a38b1db320c178 https://ixpeering.dl.sourceforge.net/project/sevenzip/7-Zip/23.01/7z2301.exe
xenadmin commented 4 months ago

Thank you very much for the very fast fix! I wasn't able to install 7-Zip and WinSCP Friday. Do you have an ETA for when the fix will be GA? No pressure! Just curious.

aaronparker commented 4 months ago

Pushed to the PowerShell gallery just now

xenadmin commented 4 months ago

Evergreen Version 2405.994

Something is still off for me. On Azure VMs a command like this, for 7-zip and WinSCP, always results in a 100KB exe file, that is malformed and/or broken.

$Evergreen = Get-EvergreenApp -Name 7zip | Where-Object { $_.Architecture -eq 'x64' -and $_.Type -eq 'exe' }
Save-EvergreenApp -InputObject $Evergreen -CustomPath "D:\Temp\"
$Evergreen = Get-Evergreenapp -Name WinSCP | Where-Object { $_.Type -eq "exe" } | Select-Object -First 1
Save-EvergreenApp -InputObject $Evergreen -CustomPath "D:\Temp\"

Any ideas? I have the feeling, it's not an Evergreen, but an Sourceforge issue. Because it works sometimes, but more ofter it fails. Maybe depending on the Sourceforge Hoster?

aaronparker commented 4 months ago

The issue could be with Save-EvergreenApp which sends a user agent. I'll test with it not sending a user agent for SourceForge.

Have I mentioned how much I dislike SourceForge?

aaronparker commented 4 months ago

I have also been wondering whether Evergreen should normalise the SourceForge URLs with a single host name, instead of returning one of the mirrors each time.

aaronparker commented 4 months ago

See #677 re:SourceForge