Closed rsbrisci closed 4 years ago
rsbrisci
commented
4 years ago The question buried in here is:
Is there any significant set of rules which runs by default on JSON, but not XML-parsed fields? If not (and I couldn't find any), I am at a total loss to explain the roughly 650% increase in processing time in JSON compared to XML requests
rsbrisci
commented
4 years ago UPDATE!
I've been able to confirm that the latency with JSON has something to do with the inclusion of a large list within the payload.
rsbrisci
commented
4 years ago Question for the CRS team now:
Why do JSON Lists cause so much latency with CRS rules?
rsbrisci
commented
4 years ago Payload for the "Fast" JSON test outlined above:
{
"lorem": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem1": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum1":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or e",
"lorem2": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum2":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem3": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum3":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains.",
"lorem4": "At vero eos et accusamus et iusto odio dignissimos ducimus qui blanditiis praesentium voluptatum deleniti atque corrupti quos dolores et quas molestias excepturi sint occaecati cupiditate non provident, similique sunt in culpa qui officia deserunt mollitia animi, id est laborum et dolorum fuga. Et harum quidem rerum facilis est et expedita distinctio. Nam libero tempore, cum soluta nobis est eligendi optio cumque nihil impedit quo minus id quod maxime placeat facere possimus, omnis voluptas assumenda est, omnis dolor repellendus. Temporibus autem quibusdam et aut officiis debitis aut rerum necessitatibus saepe eveniet ut et voluptates repudiandae sint et molestiae non recusandae. Itaque earum rerum hic tenetur a sapiente delectus, ut aut reiciendis voluptatibus maiores alias consequatur aut perferendis doloribus asperiores repellat.",
"ipsum4":"On the other hand, we denounce with righteous indignation and dislike men who are so beguiled and demoralized by the charms of pleasure of the moment, so blinded by desire, that they cannot foresee the pain and trouble that are bound to ensue; and equal blame belongs to those who fail in their duty through weakness of will, which is the same as saying through shrinking from toil and pain. These cases are perfectly simple and easy to distinguish. In a free hour, when our power of choice is untrammelled and when nothing prevents our being able to do what we like best, every pleasure is to be welcomed and every pain avoided. But in certain circumstances and owing to the claims of duty or the obligations of business it will frequently occur that pleasures have to be repudiated and annoyances accepted. The wise man therefore always holds in these matters to this principle of selection: he rejects pleasures to secure other greater pleasures, or else he endures pains to avoid worse pains."
}
airween
commented
4 years ago I'm afraid we can't help with the available information.
The JSON contents above are very different: the first one has so many keys, and all child have more child items. The second one has 10 keys (as I see) without any sub-child.
If a rule has an argument XML:/*, it means all keys will checked. That could be so many time...
Yes, you're right - the XML content doesn't trigger this latency - I have no idea why. But I think this means this issue is not CRS related.
You should start to turn off each rule set, start with 901. If this modification has no effect, turn back and take next one. Iterate this while you get a better result. Then you found the source of your problem, and can check each rule in that file.
rsbrisci
commented
4 years ago Thanks @airween !
Agree, there's not enough info yet to draw conclusions.
For now, my current theory is that something about lists with multiple items, multiple nested keys, might cause high latency in some particular rule(s).
I'm pretty sure XML vs JSON doesn't really matter. I just happened to run my original test with a particularly "bad" JSON. I ran a "List" version of this payload earlier today too, and that did not produce similar high latency.
I also attempted running Modsecurity with CRS disabled on the original "bad" JSON, and that too did not produce high latency - that's the only reason I was eyeing something in CRS.
Will attempt to identify which rule(s) seem to take excess time on the "bad" JSON.
airween
commented
4 years ago I made some researches, let me share with you the results. I sent all of three payloads above to my test Nginx with curl, and reviewed the modsec_debug.log.
The first column in the line is the line number in file (all requests logged into same file). The first line of three pairs is the beginning of random rule, the last is same with next one. Just see the difference between the first and last line numbers. That means, how many steps required to execute a rule.
'High-latency JSON' payload
407 [158573518154.925648] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
4268 [158573518154.925648] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.3861 lines.
XML payload
129999 [158573526493.497825] [/] [4] (Rule: 942100) (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
130014 [158573526493.497825] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.15 lines.
JSON with 10 keys
132705 [158573628522.775075] [/] [4] (Rule: 942100) Executing operator "DetectSQLi against REQUEST_COOKIES|...|ARGS|XML:/*.
132834 [158573628522.775075] [/] [4] (Rule: 942140) Executing operator "Rx" with param "(?i:\b(...)" against REQUEST_COOKIES|REQUEST_COOKIES_NAMES|ARGS_NAMES|ARGS|XML:/*.129 lines.
The other lines between two quotes lines above contains all steps, including transformations, and all operator evaluations.
Please check these values at your side - I think you can see the reason, why is so slowly the JSON payload what you included.
Hope this helps.
rsbrisci
commented
4 years ago @airween thank you for the analysis!
Just one last Q - is this behavior consistent between all rules for the "High Latency" JSON? Or just for the 942100 rule?
airween
commented
4 years ago No, not just for the rule 942100 - that's just a random chosed :).
It's consistent between all rules (which affected on your chosed PL), just check your log.
The "problem" is that JSON parser converts the tree hierarchy into a flat structure, and the engine handles it as ARGS and ARGS_NAMES. Any rule which contains these variables will check all 'flatted' list.
Just check your modsec_debug.log.
Describe the bug
In comparison testing of different payload types while running the OWASP CRS Ruleset 3.2, with Modsecurity 3.0.4 on nginx; we've noticed a pretty huge discrepancy in processing times between JSON and XML payloads.
Please find a screenshot of a reasonable comparison, as well as the raw JSON and XML payloads below.
Summary of above graphic:
Important Note: We've ran a test of Modsecurity without the CRS enabled, but still including Modsecurity request parsing of both MIMEs and have confirmed little difference between the parsing times for JSON and XML; so that leads us to the CRS.
Steps to reproduce
With both
XMLandJSONrequest body processing types enabled, send requests using the following XML and JSON payloads to an Nginx webserver running OWASP CRS Ruleset 3.2, with Modsecurity 3.0.4.The discrepancy in processing times between JSON and XML payloads is reproducible with any throughput.
XML Payload
JSON Payload