XML-RPC failure: Required Header Missing

[vstinner]

My https://github.com/vstinner/python-security/ script download metadata of bugs.python.org issues using XML-RPC no longer works. I use it rarely, like once a month. So I'm not sure when it stopped working.

Example of Python 3 script:

#!python
import xmlrpc.client

class SpecialTransport(xmlrpc.client.SafeTransport):
    def send_content(self, connection, request_body):
        connection.putheader("Referer", "https://bugs.python.org/")
        connection.putheader("Origin", "https://bugs.python.org")
        connection.putheader("X-Requested-With", "XMLHttpRequest")
        super().send_content(connection, request_body)

server = xmlrpc.client.ServerProxy('https://bugs.python.org/xmlrpc',
                                   allow_none=True,
                                   verbose=True,
                                   transport=SpecialTransport())
with server:
    data = server.display('issue34623')
    print("DATA", data)

Output:

send: b'POST /xmlrpc HTTP/1.1\r\nHost: bugs.python.org\r\nAccept-Encoding: gzip\r\nContent-Type: text/xml\r\nUser-Agent: Python-xmlrpc/3.7\r\nReferer: https://bugs.python.org/\r\nOrigin: https://bugs.python.org\r\nX-Requested-With: XMLHttpRequest\r\nContent-Length: 161\r\n\r\n'
send: b"<?xml version='1.0'?>\n<methodCall>\n<methodName>display</methodName>\n<params>\n<param>\n<value><string>issue34623</string></value>\n</param>\n</params>\n</methodCall>\n"
reply: 'HTTP/1.1 200 OK\r\n'
header: Server: nginx
header: Date: Mon, 21 Jan 2019 15:59:23 GMT
header: Content-Type: text/xml
header: Content-Length: 324
header: Connection: keep-alive
header: Strict-Transport-Security: max-age=63072000; includeSubdomains; preload
header: X-Frame-Options: deny
header: X-Xss-Protection: 1; mode=block
header: X-Content-Type-Options: nosniff
header: X-Permitted-Cross-Domain-Policies: none
body: b"<?xml version='1.0'?>\n<methodResponse>\n<fault>\n<value><struct>\n<member>\n<name>faultCode</name>\n<value><int>1</int></value>\n</member>\n<member>\n<name>faultString</name>\n<value><string>&lt;class 'roundup.exceptions.UsageError'&gt;:Required Header Missing</string></value>\n</member>\n</struct></value>\n</fault>\n</methodResponse>\n"
Traceback (most recent call last):
  File "bpo_xmlrpc.py", line 15, in <module>
    data = server.display('issue34623')
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1112, in __call__
    return self.__send(self.__name, args)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1452, in __request
    verbose=self.__verbose
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1154, in request
    return self.single_request(host, handler, request_body, verbose)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1170, in single_request
    return self.parse_response(resp)
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 1342, in parse_response
    return u.close()
  File "/usr/lib64/python3.7/xmlrpc/client.py", line 656, in close
    raise Fault(**self._stack[0])
xmlrpc.client.Fault: <Fault 1: "<class 'roundup.exceptions.UsageError'>:Required Header Missing">

I copied headers from http://roundup.sourceforge.net/docs/xmlrpc.html#advanced-python-client-adding-anti-csrf-headers Roundup XML-RPC documentation.

The "Migrating from 1.5.1 to 1.6.0" documentation says:

Alternatively change your xmlrpc client to add appropriate headers to the request including the:

    X-Requested-With:

header as well as any other required csrf headers (e.g. referer, origin) configured in config.ini.

http://roundup.sourceforge.net/docs/upgrading.html#errors-and-troubleshooting-xmlrpc-required-header-missing

My script uses X-Requested-With. What are other required csrf headers (e.g. referer, origin) configured in config.ini?

Note: I simplified my script. My real script uses HTTP Basic authentication using login+password. But I don't think that it matters here.

[vstinner]

Example of Roundup code which raises the error: https://hg.python.org/tracker/roundup/rev/44f7e6b958fe

[ewdurbin]

This appears to be an issue of compatibility with the roundup.cgi.wsgi_handler and Gunicorn. It was triggered by moving from the builtin roundup-server for running the web service to gunicorn.

Code is looking for HTTP_X-REQUESTED-WITH, but it appears that Gunicorn is populating the environment with HTTP_X_REQUESTED_WITH.

I've temporarily disabled this check in https://github.com/python/psf-salt/commit/4807d2ae3a731471f996c65e3fe9aeeb3fe812ad, and verified that allows XMLRPC through.

[vstinner]

I've temporarily disabled this check in python/psf-salt@4807d2a, and verified that allows XMLRPC through.

Thanks. I confirm that it works again the issue :-)

[rouilj]

@ewdurbin this is a bug. See:

changeset:   5632:8e3df461d316
branch:      maint-1.6
user:        John Rouillard <rouilj@ieee.org>
date:        Wed Feb 27 21:47:39 2019 -0500
files:       CHANGES.txt roundup/cgi/client.py roundup/scripts/roundup_server.py
 test/test_cgi.py
description:
issue2551023: Fix CSRF headers for use with wsgi and cgi. The
env variable array used - separators rather than _. Compare:
HTTP_X-REQUESTED-WITH to HTTP_X_REQUESTED_WITH. The last is
correct. Also fix roundup-server to produce the latter form. (Patch
by Cédric Krier)

https://sourceforge.net/p/roundup/code/ci/8e3df461d316

[berkerpeksag]

The fix John pointed out has also been applied to Roundup 1.6.1 and since our fork has been upgraded to 1.6.1, this issue can be closed now:

https://github.com/psf/bpo-roundup/blob/68573d196f9a01786414d3b235252b9c857c3e08/CHANGES.TXT#L25-L29