AMD URLs Failing Validation

[dmariogatto]

https://chocolatey.org/packages/amd-ryzen-chipset/2.04.28.626

All URLs in the package above are failing automated testing,

• DocsUrl
• LicenseUrl
• ProjectUrl
• ReleaseNotes

[gep13]

@dmariogatto thanks for reaching out regarding this package.

These URL's are resulting in a time out when running on the package-validator machine. I am going to investigate why this is the case, but it will likely not be until tomorrow before I have more information to share with you.

[chtof]

https://chocolatey.org/packages/webex-meetings/40.6.5.5

mailingListUrl is failing automated testing: https://twitter.com/webex and I can't execute a "rerun".

[ferventcoder]

We've just added the execution of a rerun earlier today @chtof.

[chtof]

Maybe an issue when using twitter as link for malinglistUrl, another example: https://chocolatey.org/packages/debotnet/0.7.800#files

The MailingListUrl element in the nuspec file should be a valid Url. Please correct this More... Url is available: https://twitter.com/Mirinsoft

[chtof]

We've just added the execution of a rerun earlier today @chtof.

I didn't understand. Are you really talking about webex-meetings package (as I can't rerun the package despite of a valid url marked as no valid?).

[gep13]

@chtof said... I didn't understand. Are you really talking about webex-meetings package (as I can't rerun the package despite of a valid url marked as no valid?).

Re-running the package-valiator is something that only moderators can do. Can I ask that you create another issue for the package that is causing you problems, and I will follow up with you there. This specific package seems to have a very unique problem.

[gep13]

@chtof actually, looks like you package has been approved, so looks like it got taken care of.

[gep13]

@dmariogatto I have went round the houses a couple times trying to get the URL's associated with this package to correctly validate as part of the package-validator. I now think I know exactly what the problem is, but I don't have any clear idea on how to fix it.

When running the code to validate one of the URL's the request that is sent out contains the following headers:

GET https://www.amd.com/en/products/chipsets-am4 HTTP/1.1
Connection: keep-alive,Keep-Alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, image/apng, */*; q=0.8, application/signed-exchange; v=b3; q=0.9
Sec-Fetch-Mode: navigate
Sec-Fetch-Dest: document
Sec-Fetch-Site: cross-site
Sec-Fetch-User: ?1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB, en-US; q=0.8, en; q=0.6, de-DE; q=0.4, de; q=0.2
Host: www.amd.com

On my machine, this works exactly as expected. However, when running the same code on the package-validator machine, a similar request, with the same headers, doesn't work properly. However, if I change the request header order manually to the following:

GET https://www.amd.com/en/products/chipsets-am4 HTTP/1.1
Host: www.amd.com
Connection: keep-alive,Keep-Alive
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36
Accept: text/html, application/xhtml+xml, application/xml; q=0.9, image/webp, image/apng, */*; q=0.8, application/signed-exchange; v=b3; q=0.9
Sec-Fetch-Mode: navigate
Sec-Fetch-Dest: document
Sec-Fetch-Site: cross-site
Sec-Fetch-User: ?1
Accept-Encoding: gzip, deflate, br
Accept-Language: en-GB, en-US; q=0.8, en; q=0.6, de-DE; q=0.4, de; q=0.2

i.e. I have moved the host header to be the second one in the list, then the request does validate correctly.

The Web Server that is hosting the AMD site uses https://www.akamai.com/ to serve the content, and I believe the rules that are in place there are forcing a requirement that the header order be specific.

Initially, I thought that this was something Operating Specific, as the same request running locally on my machine, Windows 10, worked, where as running on Windows Server 2012 R2, didn't work. However, I have ruled out this as the issue, and I have ran the same code, in the same location, with Windows Server 2019, and the results are the same.

I then ran another instance of the package-validator machine in a different hosting provider, and things started working. So it looks like it is something to do with "where" we are running the package-validator machine, and for now, this isn't something that is going to change. As I mentioned in the notes of your package moderation, we have added the ability to exempt a package from validation, and I have done this for your package. Hopefully that will buy us some time to decide what we can do going forward.

Ideally, I would alter the package-valdiator code, which now uses HttpClient, rather than HttpWebRequest to place the Host header as the second header that is sent out, but that doesn't seem to be possible, as I have tried various ways of doing it.

Anyway, long story short, this issue isn't fixed, but I do believe that we have a path forward.

[chtof]

@chtof actually, looks like you package has been approved, so looks like it got taken care of.

@gep13 I removed links to twitter.com

[dmariogatto]

@gep13 Thanks for diagnosing the issue and your thorough description!

[gep13]

@chtof ah! So do you think Twitter urls would still cause a problem through the validation?

[chtof]

@gep13 Yes, 3 of my packages have failed package validator tests for Twitter.com links considerated as unavailable: webex-meetings, debotnet and today with webex-teams: https://chocolatey.org/packages/webex-teams/3.0.15711.0

For webex-teams, I haven't yet removed the Twitter link (malinglistUrl). I let you to have a look.

[chtof]

On other hand in moderator queue: https://chocolatey.org/packages/uniflash/6.0.0.2710 Maintainer asks why validator tests fails every time (valid urls).

[chtof]

https://chocolatey.org/packages/sublimemerge/0.0.2027 The package is waiting for maintainer due to an "invalid" BugTrackerUrl. The BugTrackerUrl is valid : https://github.com/sublimehq/sublime_merge/issues And I can't launch a re-run. (the feature is not displayed, I can only reject the package).

[chtof]

https://chocolatey.org/packages/spacedesk-client/0.9.53 The package is waiting for maintainer due to an "invalid" DocsUrl. The DocsUrl is valid: https://spacedesk.net/user-manual And I can't launch a re-run. (the feature is not displayed, I can only reject the package)

[pauby]

@chtof You are the maintainer of spackdesk-client and sublimemerge which is why you can't see the options to re-run the Validator. You cannot moderate your own packages.

[chtof]

I agree I cannot moderate packages but it should be nice to be able to just do a rerun when a package fails due to URL checks. Or if technically feasible, to improve these URL checks.

[pauby]

@chtof

Or if technically feasible, to improve these URL checks.

This is why this issue exists, to improve the URL checks but they take time to troubleshoot and workaround / fix. This hans't been forgotten about or ignored. It is being worked on.

[ferventcoder]

Might be best to move urls down to guidelines instead of requirements until they absolutely work or don't work when there are true failures. I think folks might be split on this, but I prefer we capture the issues without holding up valid requests unnecessarily. There should be a happy medium somewhere here.

[cmorty]

A assume #235, #240, #242, #243, #244, #245, #246 and #247 are dups? Also https://chocolatey.org/packages/gitlab-runner/13.5.0 has this issue.

[gep13]

I have just tested this package through validation again, and it now works as expected. I am going to assume that it was a dupe of #243 which has been addressed.

[gep13]

@cmorty the issues that you mentioned didn't all have the same root cause, no. But they have been addressed where possible.

[gep13]

Actually, reading back, this isn't a duplicate of #243, but rather was fixed as a result of a change in infrastructure. Thanks to @pauby for getting this done.