search

PortSwigger / turbo-intruder

Turbo Intruder is a Burp Suite extension for sending large numbers of HTTP requests and analyzing the results.
https://portswigger.net/blog/turbo-intruder-embracing-the-billion-request-attack
Apache License 2.0
1.42k stars 207 forks source link

CLI runtime breaks for versions > 1.0.19 #113

Closed laluka closed 1 year ago

laluka commented 1 year ago

Description

The CLI runtime breaks on the most recent versions of turbo-intruder due to high dependencies on burp utils.

I've also experienced issues for building with the repo's current gradlew, thus advising a generic and reproductible use of sdkman:

# https://sdkman.io/ official method:
curl -s "https://get.sdkman.io" | bash
sdk install java 16.0.2.7.1-amzn
sdk use java 16.0.2.7.1-amzn
sdk install gradle 7.5.1
sdk use gradle 7.5.1

Suggestions

Runs & Logs

OK on 1.0.19 (expected behavior)

~/Downloads/turbo-intruder (f390c1a) » gradle build fatjar

> Task :compileKotlin
w: /home/lalu/Downloads/turbo-intruder/src/ThreadedRequestEngine.kt: (203, 36): 'isEmpty(): Boolean' is deprecated. This member is not fully supported by Kotlin compiler, so it may be absent or have different signature in next major version
w: /home/lalu/Downloads/turbo-intruder/src/ThreadedRequestEngine.kt: (354, 17): 'isEmpty(): Boolean' is deprecated. This member is not fully supported by Kotlin compiler, so it may be absent or have different signature in next major version

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.

See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings

BUILD SUCCESSFUL in 9s
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/turbo-intruder (f390c1a) » java -jar ./build/libs/turbo-intruder-all.jar /opt/turbo-intruder/resources/examples/basic.py /opt/turbo-intruder/resources/examples/request.txt http://127.0.0.1:5000 foo
Please note that Turbo Intruder's SSL/TLS handling may differ slightly when run outside Burp Suite.
TURBO NOTICE: The input request appears to be using \n instead of \r\n as a line-ending. Consider changing your text-editor settings. Normalising...
ID | Word | Status | Wordcount | Length | Time
Starting attack...
Warming up...
HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.8.10
[...]
HTTP/1.1 200 OK
Server: Werkzeug/2.1.2 Python/3.8.10
Date: Thu, 22 Sep 2022 12:52:26 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 131
Connection: close

<!DOCTYPE html>
<html>
  <head>
    <meta charset="UTF-8">
    <title>title</title>
  </head>
  <body>
POST:|GET:
  </body>
</html>

Completed attack on http://127.0.0.1:5000
Sent 5 requests in 0.19818087 seconds
RPS: 5

Reqs: 5 | Queued: 0 | Duration: 1 | RPS: 5 | Connections: 5 | Retries: 0 | Fails: 0 | Next: null | Completed

KO on master (suffered behavior)

~/Downloads/turbo-intruder (master) » gradle build fatjar

Deprecated Gradle features were used in this build, making it incompatible with Gradle 8.0.                                                                                                    

You can use '--warning-mode all' to show the individual deprecation warnings and determine if they come from your own scripts or plugins.                                                      

See https://docs.gradle.org/7.5.1/userguide/command_line_interface.html#sec:command_line_warnings                                                                                              

BUILD SUCCESSFUL in 11s                                                                                                                                                                        
6 actionable tasks: 6 executed
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
~/Downloads/turbo-intruder (master) » java -jar ./build/libs/turbo-intruder-all.jar /opt/turbo-intruder/resources/examples/basic.py /opt/turbo-intruder/resources/examples/request.txt http://1
27.0.0.1:5000 foo                                                                                                                                                                              
Please note that Turbo Intruder's SSL/TLS handling may differ slightly when run outside Burp Suite.                                                                                            
TURBO NOTICE: The input request appears to be using \n instead of \r\n as a line-ending. Consider changing your text-editor settings. Normalising...                                           
ID | Word | Status | Wordcount | Length | Time                                                                                                                                                 
Starting attack...                                                                                                                                                                             
Establishing 5 connection to http://127.0.0.1:5000 ...                                                                                                                                         
Autorecovering first-request error during 'pouet'                                                                                                                                              
Autorecovering first-request error during 'lang'                                                                                                                                               
Autorecovering first-request error during 'id'                                                                                                                                                 
Autorecovering first-request error during 'ln'          
Autorecovering first-request error during 'ln'                                                                                                                                        [48/1293]
Autorecovering first-request error during 'test'                                                                                                                                               
Autorecovering first-request error during 'id'                                                                                                                                                 
Autorecovering first-request error during 'ln'                                                                                                                                                 
Skipping word due to multiple failures: -887004842                                                                                                                                             
Autorecovering first-request error during 'test'                                                                                                                                               
java.lang.NullPointerException: Cannot invoke "burp.IExtensionHelpers.stringToBytes(String)" because "burp.Utils.helpers" is null                                                              
Autorecovering first-request error during 'id'                                                                                                                                                 
        at burp.Request.getRequestAsBytes(Request.kt:99)                                                                                                                                       
Autorecovering first-request error during 'ln'                                                                                                                                                 
        at burp.ThreadedRequestEngine.sendRequests(ThreadedRequestEngine.kt:212)                                                                                                               
        at burp.ThreadedRequestEngine.access$sendRequests(ThreadedRequestEngine.kt:17)                                                                                                         
        at burp.ThreadedRequestEngine$1.invoke(ThreadedRequestEngine.kt:50)                                                                                                                    
        at burp.ThreadedRequestEngine$1.invoke(ThreadedRequestEngine.kt:49)                                                                                                                    
Skipping word due to multiple failures: 1275215925                                                                                                                                             
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)                                                                                                                       
Ignoring error: java.lang.NullPointerException: Cannot invoke "burp.IExtensionHelpers.stringToBytes(String)" because "burp.Utils.helpers" is null                                              
java.lang.NullPointerException: Cannot invoke "burp.IExtensionHelpers.stringToBytes(String)" because "burp.Utils.helpers" is null                                                              
Skipping word due to multiple failures: 1956063102                                                                                                                                             
        at burp.Request.getRequestAsBytes(Request.kt:99)                                                                                                                                       
        at burp.ThreadedRequestEngine.sendRequests(ThreadedRequestEngine.kt:212)                                                                                                               
Skipping word due to multiple failures: 2045426533                                                                                                                                             
        at burp.ThreadedRequestEngine.access$sendRequests(ThreadedRequestEngine.kt:17)                                                                                                         
        at burp.ThreadedRequestEngine$1.invoke(ThreadedRequestEngine.kt:50)                                                                                                                    
        at burp.ThreadedRequestEngine$1.invoke(ThreadedRequestEngine.kt:49)                                                                                                                    
        at kotlin.concurrent.ThreadsKt$thread$thread$1.run(Thread.kt:30)                                                                                                                       
Ignoring error: java.lang.NullPointerException: Cannot invoke "burp.IExtensionHelpers.stringToBytes(String)" because "burp.Utils.helpers" is null                                              
java.lang.NullPointerException: Cannot invoke "burp.IExtensionHelpers.stringToBytes(String)" because "burp.Utils.helpers" is null                                                              
Autorecovering first-request error during 'lang'                                                                                                                                               
        at burp.Request.getRequestAsBytes(Request.kt:99)                         
[...]

End Node

By reading the logs, only stringToBytes seems to be a problem for now, but the others members of this class might bite us later:

~/Downloads/turbo-intruder (master) » grep -rF Utils.helpers
src/BurpRequestEngine.kt:                req.response = Utils.helpers.bytesToString(respBytes)
src/BurpRequestEngine.kt:                req.response = Utils.helpers.bytesToString(resp.response) // , StandardCharsets.UTF_8
src/ThreadedRequestEngine.kt:                                    val pauseBytes = Utils.helpers.stringToBytes(pauseMarker)
src/ThreadedRequestEngine.kt:                                    pausePoint = Utils.helpers.indexOf(byteReq, pauseBytes, true, i, byteReq.size)
src/Request.kt:        return fixContentLength(Utils.helpers.stringToBytes(getRequest()))
src/Request.kt:        return Utils.helpers.stringToBytes(response)

Side-files for test purpose

/opt/turbo-intruder/resources/examples/basic.py

def queueRequests(target, wordlists):
    engine = RequestEngine(endpoint=target.endpoint,
                           concurrentConnections=5,
                           requestsPerConnection=100,
                           pipeline=False
                           )

    for word in open('/opt/sulfateuse/tests/test_params'):
        engine.queue(target.req, word.rstrip())

def handleResponse(req, interesting):
    print(req.response)

/opt/sulfateuse/tests/test_params

id
test
lang
pouet
ln

/opt/turbo-intruder/resources/examples/request.txt

GET /both?%s=foo HTTP/1.1
Host: example.net
Connection: keep-alive

Debug server (similar but not same): python3 -m http.server 5000

laluka commented 1 year ago

Might also be related to this early issue if you use locally a different build process than the one in your research article :) https://github.com/PortSwigger/turbo-intruder/issues/4

albinowax commented 1 year ago

Thanks for the high quality report, I should be able to resolve this next week by eliminating the dependencies between ThreadedRequestEngine and Utils.helpers / Utils.callbacks.

albinowax commented 1 year ago

I've just pushed some updates which should fix this, let me know if you have any further issues. Thanks!

laluka commented 1 year ago

Compiles, works like a charm! Might be nice to see the table element move to a dict ? Or have another structured way to store only interesting results while the scan runs, instead of having a big filter at the end

Doing something like this works, but might be memory consuming for big wordlists :sweat: 

def completed(table):
    findings = dict()
    for finding in table:
        key = "%d:%d" % (finding.status, finding.length)
        findings[key] = finding

    for status, req in findings.items():
        print "\n\n\n\n\n*****", status, "*****"
        print req.request
        print req.response

Thanks again for this efficient bugfix! :tulip:

albinowax commented 1 year ago

The only requests that are stored are those that you place in the table by using table.add() inside handleResponse(). For larger/long-running attacks I suggest doing your filtering at runtime in handleResponse(), simply by being selective about calling table.add(). Many simple filters are easily applied using the decorator system: https://github.com/PortSwigger/turbo-intruder/blob/master/decorators.md

laluka commented 1 year ago

I totally agree on the fact that the filtering should be done in handleResponse, but as it's a list, and to ensure only interesting results are added, I usually work with a dict that use concat(resp.status_code, ":", resp.len) as a key. As this uses a hashtable it's complexity remains low. But adding only unique elements to a lists seems to force the use of iterators and thus pushes the complexity to n^2

A workaround (for me) would be to declare a global variable and make it available in both these functions and leaving the table aside, or storing my dict in table[0], but it feels flaky, thus discussing another approach for the "findings bag" and its format! :)

albinowax commented 1 year ago

It sounds like you're optimising for CPU use, but I think you'll find Turbo Intruder is bottlenecked by network speed, and eventually RAM for long running attacks that store lots of data.

I might have misunderstood something, but for your use-case I would

I'm pretty rusty but to my mind that should be reasonably efficient.

laluka commented 1 year ago

Both for CPU & memory use, but you're totally right for the throttle on the network... :sweat_smile:

The idea you're sharing totally works, like the one I suggested, but it requires having a global object on the two function handlers to place stuff in, keep some context, and filter

The question was, "Should we keep only the table as a long-lived data holder or something more structured?"

But it's flexible enough, so there is no real need to change how things work, it would just improve a bit the readability, but who cares it's offensive stuff! :heart_decoration:

I think we're good here, thanks again for taking the time to discuss this!