ibmi-fix-check HTTP Error 521 on https://render-prd.support-drupal.cis.ibm.net/support/pages

[psyntium]

Not sure related to issue https://github.com/IBM/ansible-for-i/issues/157 or not.

But still related to ibmi-fix-check.py and ibmi-fix-group-check.py

If the URL is in https://render-prd.support-drupal.cis.ibm.net/support/pages//uid/nas4SF99738 for example, I'm getting 521 error. Sample:

{
        "changed": false,
        "count": 1,
        "elapsed_time": "0:00:02.872193",
        "end": "2023-09-05 09:41:19.546050",
        "failed": false,
        "group_info": [
            {
                "description": "SF99738 740 Group Security",
                "ptf_group_level": 50,
                "ptf_group_number": "SF99738",
                "ptf_list": [
                    {
                        "error": "HTTP Error 521: ",
                        "url": "https://render-prd.support-drupal.cis.ibm.net/support/pages//uid/nas4SF99738"
                    }
                ],
                "release": "R740",
                "release_date": "08/22/2023",
                "url": "https://www.ibm.com/support/pages/uid/nas4SF99738"
            }
        ],
        "rc": 0,
        "start": "2023-09-05 09:41:16.673857",
        "stderr": "",
        "stderr_lines": []
}

However, if I replace https://render-prd.support-drupal.cis.ibm.net/support/pages// to https://www.ibm.com/support/pages/ instead, it works fine.

Not sure what should be changed on the urls.open_url.

Any thoughts?

[robgjertsen1]

Yes, I noticed that issue when testing the updated PSP group PTF URL. The error 521 is intended to indicate no server or server not accessible, but you can pull up the web page fine outside of python or as you found with using standard IBM support prefix for URL in python, so seems like an odd python issue. See one reference on this type of issue to a user agent. Defined an http agent recently for recent http 403 error with fix check modules, but this may need some additional tweaking.

[robgjertsen1]

I was hoping that changing the user agent to look more like a browser would help here in call to open_url, but it did not. With something like the following:

    "http_agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36",

versus recently added "ansible/ibm.power_ibmi" for http_agent that will fill in the user agent value/string in the header.

I still suspect that server may be flagging the request because thinks this is a bot and have to get more sophisticated with request, but speculation at this point. I haven't seen much about http error 521 versus recent issue with http 403 error where we had to change the default user agent value provided by urllib.

[psyntium]

After playing around with the redirects that is happening, you would notice that this URL: https://render-prd.support-drupal.cis.ibm.net/support/pages//uid/nas4SF99738

Redirects to: http://render-prd.support-drupal.cis.ibm.net/support/pages/uid/nas4SF99738

Notice the http vs https. If you open up in incognito mode, or directly curl to: http://render-prd.support-drupal.cis.ibm.net/support/pages/uid/nas4SF99738

You will get the same 521 error coming from cloudflare. Here's a screenshot:

Sample curl command and output:

$ curl -I 'http://render-prd.support-drupal.cis.ibm.net/support/pages/uid/nas4SF99738'
HTTP/1.1 521 
Date: Thu, 14 Sep 2023 05:55:10 GMT
Content-Length: 0
Connection: keep-alive
Cache-Control: no-store, no-cache
CF-Cache-Status: DYNAMIC
Set-Cookie: __cflb=0H28vhTuMhHWuZ7nn1ZJXBQEWFVrZZaMqdHtkDtg8Tf; SameSite=Lax; path=/; expires=Thu, 14-Sep-23 06:25:10 GMT; HttpOnly
Server: cloudflare
CF-RAY: 80666123999a0430-HKG

So I guess the site admin needs to fix this issue.

By any chance, do you know who owns those links?

[psyntium]

Another thought, while redirecting, I suspect it sets the cookie but python request module doesn't handle the set-cookie automatically during redirection, causing the request to fail.

Example curl request:

$ curl -IL 'https://render-prd.support-drupal.cis.ibm.net/support/pages//uid/nas4SF99738'
HTTP/2 302 
date: Thu, 14 Sep 2023 08:15:31 GMT
content-type: text/html; charset=UTF-8
location: http://render-prd.support-drupal.cis.ibm.net/support/pages/uid/nas4SF99738
cache-control: max-age=1800, public
cache-control: private
x-ua-compatible: IE=edge
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
expires: Sun, 19 Nov 1978 05:00:00 GMT
vary: Cookie
x-generator: Drupal 9 (https://www.drupal.org)
x-drupal-cache: MISS
last-modified: Thu, 14 Sep 2023 08:15:31 GMT
strict-transport-security: max-age=31536000
set-cookie: 8cfd9b5613f6b72c0ffaeee0933933b9=3c8fc3bf15bd1449a417e807611e9a59; path=/; HttpOnly; Secure; SameSite=None
cf-cache-status: DYNAMIC
set-cookie: __cflb=0H28vhTuMhHWuZ7nn1NeGHid59M98HRsWk7cHh16Dqw; SameSite=Lax; path=/; expires=Thu, 14-Sep-23 08:45:31 GMT; HttpOnly
server: cloudflare
cf-ray: 80672eb8c9230502-HKG

HTTP/2 301 
date: Thu, 14 Sep 2023 08:15:33 GMT
content-type: text/html; charset=UTF-8
location: http://render-prd.support-drupal.cis.ibm.net/support/pages/sf99738-740-group-security-level-51
x-redirect-id: 12753616
x-ua-compatible: IE=edge
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
x-generator: Drupal 9 (https://www.drupal.org)
x-drupal-cache: HIT
strict-transport-security: max-age=31536000
set-cookie: 5a61ae4084b0bc452f75faecc0727e22=489c4472a607c950333575aedb7beb0f; path=/; HttpOnly; Secure; SameSite=None
cache-control: private
cf-cache-status: DYNAMIC
set-cookie: __cflb=0H28vhTuMhHWuZ7nn1NWzzepctJ349Wf3S6q5hYmwRw; SameSite=Lax; path=/; expires=Thu, 14-Sep-23 08:45:33 GMT; HttpOnly
server: cloudflare
cf-ray: 80672ebeed570502-HKG

HTTP/2 200 
date: Thu, 14 Sep 2023 08:15:35 GMT
content-type: text/html; charset=UTF-8
cache-control: max-age=1800, public
cache-control: private
x-drupal-dynamic-cache: MISS
x-ua-compatible: IE=edge
content-language: en
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
expires: Sun, 19 Nov 1978 05:00:00 GMT
vary: Cookie
x-generator: Drupal 9 (https://www.drupal.org)
x-drupal-cache: HIT
last-modified: Thu, 14 Sep 2023 05:40:09 GMT
strict-transport-security: max-age=31536000
set-cookie: 5a61ae4084b0bc452f75faecc0727e22=a50a591d89858f47d9fdb9ace5ddd5fc; path=/; HttpOnly; Secure; SameSite=None
cf-cache-status: DYNAMIC
set-cookie: __cflb=0H28vhTuMhHWuZ7nn1Nj3BTxyVPrp2njjN6j5iosi5K; SameSite=Lax; path=/; expires=Thu, 14-Sep-23 08:45:35 GMT; HttpOnly
server: cloudflare
cf-ray: 80672ecdc8850502-HKG

[psyntium]

Hi @robgjertsen1 .. Seems like the links in the page has been resolved. Please help to confirm before I close this issue.

Links in https://www.ibm.com/support/pages/ibm-i-group-ptfs-level now no longer points to https://render-prd.support-drupal.cis.ibm.net/support/pages

[robgjertsen1]

Thanks for digging into this more. I was observing the same issue with igcognito mode browser with Cloudflare server (Dallas). Looking at this web page https://www.mondoze.com/guide/kb/error-521-web-server-is-down , I am guessing that the server at render-prd.support-drupal.cis.ibm.net was blocking the requests from Cloudflare server IPs (traffic level too high or what not). However, also agree that it now seems to be resolved with testing on my end, so no longer referencing that server did the trick.