Failed to handle valid XML request body larger than XML_MAX_LOOKUP_LIMIT

How to reproduce:

Generate some valid XML larger than XML_MAX_LOOKUP_LIMIT (https://github.com/GNOME/libxml2/blob/master/include/libxml/parserInternals.h#L73, 10000000 bytes by default) e.g. with this script: https://gist.github.com/defanator/e06a66a3275fd75b39c37d6a00f7b7a4

vagrant@vagrant:~$ ./generate-xml.sh 26700 > test.xml
vagrant@vagrant:~$ stat test.xml 
  File: test.xml
  Size: 10333790    Blocks: 20184      IO Block: 4096   regular file
Device: 803h/2051d  Inode: 3418121     Links: 1
Access: (0664/-rw-rw-r--)  Uid: ( 1000/ vagrant)   Gid: ( 1000/ vagrant)
Access: 2021-05-05 10:30:46.459915604 +0000
Modify: 2021-05-05 10:30:44.127915604 +0000
Change: 2021-05-05 10:30:44.127915604 +0000
 Birth: -

Push it to nginx running with ModSecurity-nginx connector:

vagrant@vagrant:~$ curl -v -d @test.xml -H "Content-Type: text/xml" http://localhost/modsec-full/upload/
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 80 (#0)
> POST /modsec-full/upload/ HTTP/1.1
> Host: localhost
> User-Agent: curl/7.58.0
> Accept: */*
> Content-Type: text/xml
> Content-Length: 10013375
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 400 Bad Request
< Server: nginx/1.19.10
< Date: Wed, 05 May 2021 10:30:56 GMT
< Content-Type: text/html
< Content-Length: 158
< Connection: close
< 
<html>
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx/1.19.10</center>
</body>
</html>
* Closing connection 0

The logs will contain something like this:

2021/05/05 10:30:56 [error] 20269#20269: *4 [client 127.0.0.1] ModSecurity: Access denied with code 400 (phase 2). Matched "Operator `Eq' with parameter `0' against variable `REQBODY_ERROR' (Value: `1' ) [file "/etc/nginx/modsec/modsecurity.conf"] [line "10"] [id "200002"] [rev ""] [msg "Failed to parse request body."] [data "XML parsing error: XML: Failed parsing document."] [severity "2"] [ver ""] [maturity "0"] [accuracy "0"] [hostname "127.0.0.1"] [uri "/modsec-full/upload/"] [unique_id "1620210656"] [ref "v156,1"], client: 127.0.0.1, server: , request: "POST /modsec-full/upload/ HTTP/1.1", host: "localhost"

But the actual error message from libxml2 will be:

body.xml:1: parser error : internal error: Huge input lookup
      <time>2009-10-17T18:37:34Z</time>      </trkpt>    </trkseg>  </trk></gpx>
                                                                               ^

(see https://github.com/SpiderLabs/ModSecurity/issues/2565 for details on how to get the actual message)

I see that the library has some functions to process XML payload chunk by chunk, so I suspect there's some limitation at the connector's side (thus I'm filling this issue in the connector project).

owasp-modsecurity / ModSecurity-nginx

Failed to handle valid XML request body larger than XML_MAX_LOOKUP_LIMIT #246