[Bug]: SourceForge sourced apps returning 403

chezzer64 commented 3 weeks ago

What happened?

All Evergreen apps hosted on SourceForge are currently returning '403 Forbidden'

Version

2410.1527

What PowerShell edition/s are you running Evergreen on?

Windows PowerShell

Which operating system/s are you running Evergreen on?

Windows 10+

Have you reviewed the documentation?

[X] Troubleshooting at: https://stealthpuppy.com/evergreen/troubleshoot/
[X] Known issues at: https://stealthpuppy.com/evergreen/issues/

Verbose output

Get-EvergreenApp -Name keepass -verbose
VERBOSE: Function path: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2410.1527\Apps\Get-keepass.ps1
VERBOSE: Function exists: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2410.1527\Apps\Get-keepass.ps1.
VERBOSE: Dot sourcing: C:\Program Files\WindowsPowerShell\Modules\Evergreen\2410.1527\Apps\Get-keepass.ps1.
VERBOSE: Get-FunctionResource: read application resource strings from [C:\Program Files\WindowsPowerShell\Modules\Evergreen\2410.1527\Manifests\keepass.json]
VERBOSE: Calling: Get-keepass.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UseBasicParsing: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: MaximumRedirection: 2.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: DisableKeepAlive: True.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 Edg/127.0.2651.105.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Method: Default.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: Uri: https://sourceforge.net/projects/keepass/best_release.json.
VERBOSE: Invoke-EvergreenRestMethod: Invoke-RestMethod parameter: ContentType: application/json; charset=utf-8.
VERBOSE: GET with 0-byte payload
WARNING: Invoke-EvergreenRestMethod: The remote server returned an error: (403) Forbidden., with: https://sourceforge.net/projects/keepass/best_release.json.
WARNING: Invoke-EvergreenRestMethod: For troubleshooting steps see: https://stealthpuppy.com/evergreen/troubleshoot/.
WARNING: Run 'Get-EvergreenApp -Name "keepass" -Verbose' to review additional details for troubleshooting.
Invoke-RestMethod : Just a moment...*{box-sizing:border-box;margin:0;padding:0}html{line-height:1.15;-webkit-text-size-adjust:100%;color:#313131;font-family:system-ui,-apple-system,BlinkMacSystemFont,Segoe
UI,Roboto,Helvetica Neue,Arial,Noto Sans,sans-serif,Apple Color Emoji,Segoe UI Emoji,Segoe UI Symbol,Noto Color
Emoji}body{display:flex;flex-direction:column;height:100vh;min-height:100vh}.main-content{margin:8rem auto;max-width:60rem;padding-left:1.5rem}@media (width Enable JavaScript and cookies to
continue(function(){window._cf_chl_opt={cvId: '3',cZone: "sourceforge.net",cType: 'managed',cRay: '8d9aa8455c8520a6',cH:
'6Iku2Pm7upZOD.u.sSrNv48bLBUigY5W6tA7i_njR2k-1730115692-1.2.1.1-o3CSn369Bp8JHNPSbLAczOE45PnYF2pqVdErUQP6ZtWhf_L6RElVryto262PDN.N',cUPMDTk:
"\/projects\/keepass\/best_release.json?__cf_chl_tk=Id1r7.eFrKYslEc58PdkykMDHC0R_e5Hz_vUNSH3te4-1730115692-1.0.1.1-YEE4Uyds0s.Xi9k6rPNOSKItH3EFizcbB3WDvuM_lq0",cFPWv: 'b',cITimeS: '1730115692',cTTimeMs:
'1000',cMTimeMs: '390000',cTplV: 5,cTplB: 'cf',cK: "",fa:
"\/projects\/keepass\/best_release.json?__cf_chl_f_tk=Id1r7.eFrKYslEc58PdkykMDHC0R_e5Hz_vUNSH3te4-1730115692-1.0.1.1-YEE4Uyds0s.Xi9k6rPNOSKItH3EFizcbB3WDvuM_lq0",md: "cFcGXILbnKjqjgULz3Z2D57PLuYUucZQYn8F2y1fR
EA-1730115692-1.2.1.1-tzj2WwhgzgR1e8GphAgciF5HTZsHTD1NEzcextqngjTqOYq5p_kshKhEMSHYHtGszeFvq7CzU8lHSgeTMghxHp8Yi9oOLqNym8zAfUPfc9OXKH_ZKxDhaMzxpcOvCPt47c1x7IW.RzMre3fp7vlpqWoOk2IuC5BuZ1ppL7JfrLEoMR7X9D8Z3enbk7
JD_QcKsWxo_0Z3xnHu9d0faKy5xzrYw2kZGX7Dz5BjjdfAKy.uK.VCHvt6YqwXTU0eO3vXfA4ero12Z7Emr4V6CYf76umkoIkJl3RK83QT_3AqdjdAWlMU1xhE0nhMP7UObkKtq3Qd4oI4vOzLBCCW0EPCIH0LXtWj7TTqx_csgTXXskowyqIPJhrt7NMnKNZjJ2M6ccckzBkZyR
PXIRZUEOk2GFk10TYZKhrHynHAFeHH_OZO.sf1GMxAJfSTp6sESeixxQheOB7pGpP4w8M_SdNN08XePc4juq2nbz8AX23Iq013KMUDoJg7sGAG126uJ1YcZjqB7GdmykCwq0ZtBWXiuOrSwI7OMkLiZRtGEykUnNqgpjAOQXk665V4aWNkKmvQkHrgonGLXCF1h9joRJu2_0QkNJ
uyhvKAqOC.90zMRHZcZgMr76nWQJC9IPwnqwYQMosLZ6E2dcrQ3u4i4NjAwTalNvWBoWBP7EqubRUqqJg5Bu0mA2L7Ush43V0tdI9W9SzOuWxYENSTkV4Q1d1eODWdJFJK4jOYNALVD5Ck_ZhMn7y.AmvLGhuw2PDOcGmHMOQvxxFbXggIMb5p6so.T0nZYr648Y7QvCEIRTLA6Y
99Yqbs2dVvhXN1h8abEh0H0391.n_ZeHjml1LZzlA08tEnHdZiWEeydECwulwiE96M7HzVaZqMCIHi8tbs0BGGMT6wKlGaBFpyEYI.xSG5AiEYC92IEzIBK5Sj7Ic16ecaX.SjyDLKW_NxY2XH27sDzxEwmbRo91.WoS90vabSJN2vfqrUbACYvNt1Od2.iY4NvG.d2Fkcj1iB_z
mgjDSo1ERo4DP0L.6MfDiZU89EO.nUzpl1XwYQn5zAWt7Uz2Mw.ILZmDmhwHhhPyqsQByFWonfCTLcI4nUjlFdx.N7WLfucaUNXTqqwILLLkj.bGL5QFJYkXyxq3eEV6PI9vAjoHt34z__ZtcgXINkaByjuE8LGIubmVaX178BqMrAcgivquduCE1Fg7vV3pWxta6esBAaZ2eT_T
_v1vEQClGmJotC2EIdv.GZFDRMCderCfxi0DEvfvQGJyNlNRcOoAkmrIpk7TqksnpKTFodRu8Uhh2yQ_IRrYq9zQAZfHeUNWkA_Kjgvm62AF3KX7DUyggXDfE0CuAShSFm5WA5MqHszeQyBhp2xbU4fG9WsVsEtvp41HjzV1GL0M0nFY4eU88K89LgzL2WpXmtOsYOVtnDTrduw1
MaYMbiPTkR3jVHRZ42g8zujEctNW685CtSX1l1kRqWrcfkBlX32cNECfFAh9MdAr_BBvYC2HG8lLwsAlZhqNSbbC4ThwDjCSkTX.pzxUtSmtB9TwrkvdS92smCcK09VGG36xdGLMW_NnXj6WdXqRnj6WFCZvpwN9zwKmkF5L5grVHwkBQP5ZLZdVG0bbFXtuW1ZOcflm8yJUKtN8
GX._XB_71TZxfK7D2zDE3bTe1bVprpfZGmaubeMOf2MN.FK13YPeygQVhk1zQvzwP7RTHh10SbG3JbHLhJOOHSqkA4c.HHFLRmETYBh6sjSsr5GOPySHyvhjeMm2tWGsY2DVkd2OGIU2hVSCPSYfR.LTN4ufYDW4JkkuUWVKnJZOzOM2NpiyX_uqLyLFqNi4lS.ckHtAbdqnPEcH
Un_7qWvdb63HRe8N0f7RH2KMD5wGRM11vHrq3gcHTgqG1ZtrF0Ni_1MZKWpHLOQEThJ1px_E1DzkPW.2sxqoBDuNygRKYm4nyy3uEkEUtvHVyYbo8fkeJYuLnoE174AuvsyxmyS3LtffO6CIweHrT6dZtejU4PbUyEoUFN1EyGL6khBL.qNrgewcMFRiGIaM6PpfVsY1EiJgEsS7
pQyK2TVk0XRRaPKBCBIUUWzt7awVbqnWHxdpM2ISw8iiJvAauXvTyFq8YyJeajYc1dQsZx0.J7S05WXpS9LPOm8U5jc7KeuWs4h49_DeKQ_bQ7eHiWYxwEFIIZ1cR391axzqeBgZTpDttkv.8W8aPa3sx4ibdH6IhysxQqUmB_JdVNbVlD7veeoHMJDl7Gx9_ClUP.ZKr5ws.K37
OyDZ89z.D9cFR2hi9pcwu0QzJgN.tqEKwxHG_WH3d8meCsTPvt4dZ5bTqURKl9KPYLdmTbOlrFYAyP8Yyo5kQ_fLh8Ka5bE_QnSYga_ukOA9kofXeubNDUSXpEdJdYt.AXbBkpWmF5bBa0jyVufTIu6oKaFKC6WzA_a1k70fHsJVwLOAu_UNe8iq4RLDMQBhOcFNxyE_ze7kZNRk
44Bpm314j0rw9ldtc4Ppoo67B4sY8QinvDnQP_767w4_xPkjfUyaihzSFv.4GQ6axUa7_0BX4wCaEPouw2ISeSWDn5n9Gh80hdjk1dB_hpOv92n4lQCrbPyyubQRUu1Kehnfpm2QFvS.LEA6iBP37Vc9GQqmTUbq2SK2sVTi0iQ4GewYW8ufkK2u11pEg1KPdxbinvmG7AyJll.C
C2bWf3TJNY7l_Dtot4qjRTAv3QC9bTJ8S3dCoX87Gva2mBbXSspsmmj3uUZEmyHfgkr6LUT4WPgvarC5Xv376jpUzQUQgCzGxRmUg4uAiy8lDlv7eykhZxvIuBwRGQ_uV7gofANRpq5vGXVygsx43HYaunWKBvVRkucKZLvLp1lrSexvyCf2_1Ughj7MRR_6Fgix3T",mdrd: "l
Aycaml31tG.x4miRC0F8jkWYyog.P.we1E5H8ODPEU-1730115692-1.2.1.1-XeEhukXN8q8H7bUDHvLZT.esf6JdNJUJj6Qfv2muFCceZBPycEAzRmxducePNGLQXNc.N.UO84dRDBZUAcUwgVKB2m0INboZah.BxjzLg2zCGrLLaVLBWdgg4Bhwdf6iJeTxKonHwjevk_zqlI
owIgSmA3eCGT.DInv0uH7amSureUqSQNuqj9pPzo_UWz_Uutwkd5NIwNprSLRw1QVV1P4QrJWQnjRH.270B8U5ltlT.QutdQ2aeAmXPlIpWSXrVUudNURPgwiMtAJRRpUJvQCnfaPKZcXCGUdCPDSC0XSFU_sgkJJjPPB5MCV43EGu9v39hhf_0pjLeNvxj9YJ6DGwhsxLZd27uW
dhgfoW7AaW7T7Jv2XctduqYkRbTqTE4AgH81SW0Wh4qKFFZ.AAOyQOLieGWddmPxwHwHEYw9PiEb3Y7oXr4.BO8m3W7RL7nD2_2Xe5CLtoOerP.I0O470Pagg4Ct5.eH2lRcLP5uq7fl8Reb.QnhdWnlOqzKDPGnMybREP2dxZer07k7ooecvL399_Bs3IhAJEvzJJ0NkYJ1g0LK
upvqLpN9mUqSTIyhYiJAOSbwSrR8m1fnn2k2nGXQkJ74UpcW2Rz8flZb9DFxjQN6MVwZ3m1NzOUC66kbg0sd2rlgC89toxMddar9Z979hNOrxcFD2wkFDGCkjao85SzAy_l5rQny6p38A46LCYsjCS4VyGC3hC0MSkweeVyOhGUNSOKUX6qdqbrs7IidrkFLJdALYmfA1BYcC45i
9F52c.rvZkSW.Yw0QKeKv1rQbj95XgPhbpZxZFHDkUnWcocgZa9WEFmdUjenxH0BXAzyUvxQX3.otyUuck2YZ8CaBZMuCzJvKmzJbFKm2jdg6pO._iPcp2CuM9aGhDgWro5VagdM75vLxe8fvsK3rMTKxZyRJWIsNVR7uJGUwdht3ocSDqJw2ZQa9tGIVUlL5i3xGIy_KJopufpg
jiIZt.H6UFXGl38oRTfOYEQoEpqAYJC4RVJgrLDINQ2CIq9eEhPTW6IyP84lpgBTpVA1PaoD_kJbl1P637tPTAx6fDGezT.Ab6fNmRUqMz_9fYcB_RjSh.zFiI3mL7OCeJIn6ZmJiNCuOLOHJUvN_JsJDcW8a1RSiZCh4yV_GSoEHMSPe4MWUj2W4IhTEkG66Ff.jtLay0XJ9Lsp
ZsrPSN1CXBEZUpMr4Bo4PRzsiDeNiKCAHpIpMOgPQCY1muxxZKSOZw9h0kepxm63Sv27F2hYbCQpIc0hLiQaOmjtZVZuSeRkcfGmJ8BDbHFcpkTsfE6yj84_1rF7U1bZ950YNRF2P7cnjwvUaPllwwfv.chauX9HwhTM1zus38GrnkNK4kjPMF_JqMOzE4VnEzFe1LeZRtqv35.1
1Bsw.DRjG.NY.5z9_oHfMzmQ435gtWCsmD68c5C8QwFZI8ln15f_8AsKObSaDUtMFM6e2LK4Bqd0aShH4CwT47bpsaf5Ee2ves_DzvDEKuwVy1g2KasmOfhFZocnBerVhOUvfCXfqPdwZfyepk14C1KjssuujCPVo0SyolPIt4fIBO.T832vGOMt7UubDQMvGvhoVdfNyCpODz5r
Eoi865MsEoerBUoqKxWqacgUV2hwNb4_Rl9QKce8.XGqzXufi1io3sLjwdk_.p4.xR9GHhW9517z8en8gMCkzZxUXbpBiIZVDjZzEo_oT1hVnbgRxEPVYnVGOxDFNS6eE26DkwEWc21Jn2hGZP0HhvzjLV8dQmCX35UImBbGp6fTsZ8Scsr771ivV5PnFil_8Egpe6oEKsllJbHd
U3ZED4eokFx8McqQbmR_f6TxfP9nt90A9ttWmpTOEWKY6Kvc82CF4SvBLSlLH9fyioRVNUu7CClTV5Qs7DIHUsagmcvSDEfW3RqfxMpvddX2tmBz9clCf8ug4j6ipvZ34UVPfTxYGzM5sV8OwdI7iqkFaG641Jr5x7E2i6ysoDlBDyNoak.80I48tM8_ZRVqs_ZjK.6YO5uSzDrK
VoL1oTe9RebWLyOyGuB.gBLmbLv30Z4wPnZApFoRIT5cZkttDwUyLH47Fe.hvcRNl5YKY3rTpvFsF98FRH7jFR4VKrZVaFMuf.wibe7wPgcI5yOneZL.9L9_mAE9wZ6VlOJVvNNISrnwHlCmJvj3Qh.hJAPLS6kYeIe1Ms021XKT6qjkCcyRUzL_M3l2xkA9T44.cFTLtK3HDNFK
eTVQbyNmdDFqrr4j5.AIfdrjFStP4ZvfhR.g"};var cpo = document.createElement('script');cpo.src = '/cdn-cgi/challenge-platform/h/b/orchestrate/chl_page/v1?ray=8d9aa8455c8520a6';window._cf_chl_opt.cOgUHash =
location.hash === '' && location.href.indexOf('#') !== -1 ? '#' : location.hash;window._cf_chl_opt.cOgUQuery = location.search === '' && location.href.slice(0, location.href.length -
window._cf_chl_opt.cOgUHash.length).indexOf('?') !== -1 ? '?' : location.search;if (window.history && window.history.replaceState) {var ogU = location.pathname + window._cf_chl_opt.cOgUQuery +
window._cf_chl_opt.cOgUHash;history.replaceState(null, null,
"\/projects\/keepass\/best_release.json?__cf_chl_rt_tk=Id1r7.eFrKYslEc58PdkykMDHC0R_e5Hz_vUNSH3te4-1730115692-1.0.1.1-YEE4Uyds0s.Xi9k6rPNOSKItH3EFizcbB3WDvuM_lq0" + window._cf_chl_opt.cOgUHash);cpo.onload =
function() {history.replaceState(null, null, ogU);}}document.getElementsByTagName('head')[0].appendChild(cpo);}());
At C:\Program Files\WindowsPowerShell\Modules\Evergreen\2410.1527\Private\Invoke-EvergreenRestMethod.ps1:123 char:21
+         $Response = Invoke-RestMethod @params
+                     ~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (System.Net.HttpWebRequest:HttpWebRequest) [Invoke-RestMethod], WebException
    + FullyQualifiedErrorId : WebCmdletWebResponseException,Microsoft.PowerShell.Commands.InvokeRestMethodCommand

chezzer64 commented 3 weeks ago

Think it looks like SourceForge has added some 'anti-web scraping' protection?

Have tested a workaround by adding some additional headers to the request but not sure how long they will continue to be successful?

$headers = @{ 'Accept' = "text/html,application/xhtml+xml,application/xml;q=0.9" 'Accept-Language' = "en-US,en;q=0.9" 'Sec-Ch-Ua-Platform' = "Windows" }

Update: Workaround appears to have already stopped working unfortunately.

aaronparker commented 3 weeks ago

I don't have a kind word to say about SourceForge, so I'll attempt some restraint. I do however, wish developers would stop using it.

andre-sicking commented 3 weeks ago

Hi everyone,

at the moment it works for me when I use TLS 1.3 in my Windows PowerShell (5.1) scripts. So just adding "[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls13" at the beginning should work already. I guess .NET 4.8 should be installed upfront when using this on older systems as Windows Server 2016. But I did not test this on older OS. My Server 2022 deployments just work fine now.

@aaronparker I absolutly agree about SF. But nobody knows what Microsoft will do, if everybody moves things to Github. Possible Vendor-lock-in incoming ....

Andre

aaronparker commented 3 weeks ago

Using something like the below code was working about two days ago, however, this too is failing.

I'll test with TLS 1.3

$session = New-Object Microsoft.PowerShell.Commands.WebRequestSession
$session.UserAgent = "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36 Edg/130.0.0.0"
$session.Cookies.Add((New-Object System.Net.Cookie("VISITOR", "010663f5-784c-4b89-95b3-b94caeb35ef9", "/", "sourceforge.net")))
$session.Cookies.Add((New-Object System.Net.Cookie("__cf_bm", "757znJGmuebOAc5M59hKsO28h_ubXpvGGcDYocIlwX0-1730205433-1.0.1.1-VpbSE2aiIXI17YraYEUFZL6BKHo73NgKSDsxGrYdMyjwIphBVXE43T05YKDzuRWP9sjXYzacGHBMHYrFVUOxTQ", "/", ".sourceforge.net")))
Invoke-WebRequest -UseBasicParsing -Uri "https://sourceforge.net/projects/keepass/best_release.json" `
-WebSession $session `
-Headers @{
"authority"="sourceforge.net"
  "method"="GET"
  "path"="/projects/keepass/best_release.json"
  "scheme"="https"
  "accept"="text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7"
  "accept-encoding"="gzip, deflate, br, zstd"
  "accept-language"="en-AU,en-GB;q=0.9,en;q=0.8,en-US;q=0.7"
  "cache-control"="max-age=0"
  "dnt"="1"
  "priority"="u=0, i"
  "sec-ch-ua"="`"Chromium`";v=`"130`", `"Microsoft Edge`";v=`"130`", `"Not?A_Brand`";v=`"99`""
  "sec-ch-ua-mobile"="?0"
  "sec-ch-ua-platform"="`"macOS`""
  "sec-fetch-dest"="document"
  "sec-fetch-mode"="navigate"
  "sec-fetch-site"="none"
  "sec-fetch-user"="?1"
  "upgrade-insecure-requests"="1"
}

andre-sicking commented 3 weeks ago

Maybe they had too many issues with regular user requests using specific filters or probably a "scoring system". I bet they saw in the logs that many PowerShell-Sessions used TLS 1.2 and use this as a filter now.

To be safe one could determine a current Chrome version to assemble a custom UserAgent (and setting $DownloadUserAgent upfront) and just use that with TLS 1.3 on top so they can't detect PowerShell directly :-)

aaronparker commented 3 weeks ago

I've pushed a change to the Apps branch that includes a custom useragent for SourceForge requests. Looks OK to me here in PowerShell 5.1 on Windows 10, and PowerShell on macOS.

aaronparker commented 3 weeks ago

I've added this change to a new module version to see whether it helps, so should be ready to test now

aaronparker / evergreen