Request to WAF/CDN with enhanced bot protection using url module gets HTTP Error 403: Forbidden

Summary

The specific scenario is checked with Keycloak Module, where my Keycloak Instance is hosted with Cloudflare Proxy.

When I try run keycloak related task with ansible playbook, it fails with HTTP Error 403: Forbidden. The reason being the bot protection in CDN Services, Cloudflare in my case. The request made by the keycloak module, doesn't add User-Agent header, which is blocked. I came across this issue when I was testing out keycloak realm creation with ansible playbook, It worked for my local test server, gave the following error for my cloudflare protected keycloak.

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Could not obtain access token from https://kc.korifi.run/realms/master/protocol/openid-connect/token: HTTP Error 403: Forbidden"}

I debugged down to the module and tried out the snippet to get token with a smaller python script. Just adding the Useragent header with appropriate default value, and it worked

Many of the WAF Protected servers won't allow Requests without User-Agent headers, considering them as bot generated and block.

Adding User-Agent header to the request will work for most of the scenarios that use ansible.module_uitls.urls library Following Script demonstrates the exception and fix for the same with snippet taken from https://github.com/ansible-collections/community.general/blob/1c167ab8948afe429fd4591ea4434f14ab40a862/plugins/module_utils/identity/keycloak/keycloak.py#L140


from __future__ import absolute_import, division, print_function

__metaclass__ = type

import json
import traceback
import urllib
import ssl
from ansible.module_utils.urls import open_url
from ansible.module_utils.six.moves.urllib.parse import urlencode, quote
from ansible.module_utils.common.text.converters import to_native, to_text
ctx = ssl.create_default_context()
ctx.check_hostname = False
ctx.verify_mode = ssl.CERT_NONE

request_url = "https://kc.korifi.run/realms/master/protocol/openid-connect/token" #Your Keycloak URL

## Taking out Snipper from 
temp_payload = {
            'grant_type': 'password',
            'client_id': "admin-cli",
            'username': "admin",
            'password': "admin",
}
payload = dict(
    (k, v) for k, v in temp_payload.items() if v is not None)

try:  #Request without User-Agent header will return 403 Forbidden
    r = json.loads(to_native(open_url(request_url, method='POST',
                                              validate_certs='False',
                                              data=urlencode(payload)).read()))
    print(r)
except Exception as e:
    print(e,"\n")

try: #Request with User-Agent header will work
    rh = json.loads(to_native(open_url(request_url, method='POST',
                                              validate_certs='False',
                                              data=urlencode(payload), headers={"User-Agent": "Ansible"}).read()))
    print(rh)
except Exception as e:
    print(e,"\n")

### Demonstrating Behavior with URL Lib

data=urllib.parse.urlencode(payload)
data=data.encode('ascii')

try: #Request without User-Agent header will return 403 Forbidden
    req = urllib.request.Request(url=request_url, data=data)

    with urllib.request.urlopen(req, context=ctx) as f:
        print("Response with no User-Agent Header")
        print(json.loads(to_native(f.read())))
except Exception as e:
    print("Exception with no User-Agent Header")
    print(e,"\n")

try: #Request with User-Agent header will work
    req_header = urllib.request.Request(url=request_url,headers={"User-Agent": "Ansible"},data=data)

    with urllib.request.urlopen(req_header, context=ctx) as f:
        print("Response with User-Agent Header\n")
        print(f.read())
except Exception as e:
    print(e,"\n")

One suggestion is to implicitly set User-Agent headers value to something like Ansible- I will be working on applying the fix myself will create a PR once completed.

Issue Type

Feature request

Component Name

community.general(Keycloak)

Ansible Version

$ ansible --version
ansible [core 2.12.7]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/home/dishant/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.10/site-packages/ansible
  ansible collection location = /home/dishant/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.10.5 (main, Jun  9 2022, 00:00:00) [GCC 12.1.1 20220507 (Red Hat 12.1.1-1)]
  jinja version = 3.0.3
  libyaml = True

Community.general Version

$ ansible-galaxy collection list community.general

# /usr/lib/python3.10/site-packages/ansible_collections
Collection        Version
----------------- -------
community.general 4.8.2  

# /home/dishant/.ansible/collections/ansible_collections
Collection        Version
----------------- -------
community.general 5.3.0

Configuration

$ ansible-config dump --only-changed

DEFAULT_GATHERING(/etc/ansible/ansible.cfg) = smart

OS / Environment

Linux fedora 5.18.13-200.fc36.x86_64

Steps to Reproduce

Pre-requisites:

Keycloak Server Hosted with Cloudflare Proxy

- name: Create or update Keycloak realm (minimal example)
  community.general.keycloak_realm:
    auth_client_id: "{{auth_client_id}}"
    auth_keycloak_url: "{{auth_keycloak_url}}"
    auth_realm: "{{auth_realm}}"
    auth_username: "{{auth_username}}"
    auth_password: "{{auth_password}}"
    id: "{{realm}}"
    realm: "{{realm}}"
    state: "{{state}}"
    validate_certs: false

Expected Results

It shoud be successful

Actual Results

fatal: [localhost]: FAILED! => {"changed": false, "msg": "Could not obtain access token from https://kc.korifi.run/realms/master/protocol/openid-connect/token: HTTP Error 403: Forbidden"}

Code of Conduct

[X] I agree to follow the Ansible Code of Conduct

ansible-collections / community.general