KeyStore DoS vulnerability to crash the node [BugBounty][Security Issue]

[Shashank-In]

Describe the bug The keystore has an API to create a user with input values username and password It was noticed that the username and password have an upper limit of 1024 characters. However, 1024 characters are too much for a password and is causing a denial of service bug

The request to create an account is

curl -X POST --data '{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "username",
         "password": "StrongPass"
     }
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/keystore

It was noticed that when a 635 character password is used it takes almost 12 seconds however when a 1024 character password is used it took 42 seconds which is too much.

A malicious user can make multiple such requests and make the service unavailable.

To Reproduce

1. Setup everything on a localserver.
2. Start the node ./build/ava
3. Now create an automated request with the different user where the value of the user name is changed every request to create a new account.

POST /ext/keystore HTTP/1.1
Host: 127.0.0.1:9650
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 1188
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/create

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "12test",
         "password": "JunT1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameuserna"
     }
}

4. Withing 5 secs the application will be Dos'ed

This can be confirmed by making any keystore request like

curl -X POST --data '{
    "jsonrpc":"2.0",
    "id"     :1,
    "method" :"keystore.listUsers"
}' -H 'content-type:application/json;' 127.0.0.1:9650/ext/keystore

We will notice we won't get any response back.

Note: The node has to be restarted to get make everything work again.

Expected behavior The node should be able to handle such concurrent request as well as the max length of password should be reduced

Screenshots

Operating System MacOS, ubuntu

Additional context It was tested on my localhost. After 5 seconds the server was stuck and won't be responsive

Suggested Fix Since we are already doing following strong password policy. A password length of even 25 is enough.

By submitting this issue I agree to the Terms and Conditions of the Developer Accelerator Program.

[Shashank-In]

Some additional note why I think 1024 limit is too high and not worth it at the cost of security. Since we have password policies of minimum 8 characters with a symbol number and upper alpha by default even for 8 character password, the total possibility is

(26{lower alpha}+26{Upper}+10{numbers}+30{special charaters})^8 = 92^8 = 5.1321887e+15

If we go for 25 as upper limit as submitted in PR 1.2436429e+49

It's still quite a huge/impossible possibility to brute force.

[swdee]

If one accepts the current maximum user/pass length, this is 2kb in size which is not much data at all. So if the RPC server is being DOS'd from a ~2kb request this means there is some other underlying issue. By reducing the maximum field length, this does NOT fix the underlying issue, rather it avoids it. Therefore you would investigate further please to find the real cause as other types of RPC calls could be effected similarly.

[swdee]

Also for your Reproduction step 3, how many POST requests are you making per second? How many requests are made in the 5 seconds the RPC server becomes unresponsive?

[Shashank-In]

Hi @swdee Sorry for not making it clear. But the DoS has no relation to the length of username. It is just limited to the length of the password. This is happening because the time is taken by the application to encrypt the password is proportional to the length of the password. Hence the fix would be lowering down the number of characters of the password.

Regarding step 3: I tried with 50 threads with passwords of length 1024 and user name only of 5 to 6 characters. As mentioned before, the length of the username is has nothing to do with the bug, only with the length of the password.

I realized the fix I submitted to reduce the value of maxUserPassLen is a bit wrong because the same is used for username length as well. I will correct that.

[swdee]

@Shashank-In Hi,

Thanks for explaining what your seeing further, however I disagree that the password length has any effect on the hashing of the password.

The hashing method used is Argon2 at the code point here https://github.com/ava-labs/gecko/blob/83502ee59e19bd93a2205753d3ff317bcde4c4a8/api/keystore/user.go#L26

I put together a quick benchmark comparing hashing of a password of length 1024 characters versus 25 and you can see there is no real difference. Code here https://github.com/swdee/argonbench with results

BenchmarkAHash1024-8         316          37207613 ns/op        67116783 B/op         33 allocs/op
BenchmarkAHash25-8           315          39111558 ns/op        67115769 B/op         33 allocs/op

A single run irrespective of the password length takes around 60ms on my workstation.

As Argon2 was designed to be resistant to GPU cracking, it requires the allocation of a large amount of memory, which has been configured to be 64MB.

If you are firing 50 threads, this would require [64*50] 3.2GB RAM to support, so the slow down your seeing I suspect is your RAM is exhausted and your system has gone into Swap which is putting a huge i/o load on the system.

So the proper fix would be to rate limit the user creation API to avoid the DOS and not change the password length which weakens the security without any effect.

[Shashank-In]

@swdee Sorry I am still confused why did a password with 1024 characters takes 42 seconds and 635 characters take 12 seconds?

Note: I am not firing any threads. Just a single request.

Here is my observation. (I have 8GB ram and the observation was done multiple times for each request)

1. username: test102 Password length: 512 Characters Response time: 5.4 seconds

Request

POST /ext/keystore HTTP/1.1
Host: 127.0.0.1:9650
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 678
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/create

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "test102",
         "password": "JunT1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernam"
     }
}

2. username: test103 Password length: 667 Characters Response time: 11.7 seconds

Request

POST /ext/keystore HTTP/1.1
Host: 127.0.0.1:9650
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 834
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/create

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "test103",
         "password":  "Junt1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameus"
     }
}

3. username: test104 Password length: 812 Characters Response time: 20.8 seconds

Request

POST /ext/keystore HTTP/1.1
Host: 127.0.0.1:9650
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 979
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/create

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "test104",
         "password":  "Junt1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameuse"
     }
}

4. username: test100 Password length: 1024 Characters Response time: 40.8 seconds

Request

POST /ext/keystore HTTP/1.1
Host: 127.0.0.1:9650
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 1189
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/create

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "test100",
         "password": "JunT1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameuserna"
     }
}

I am still unsure if rate-limiting would be very helpful. I guess first the problem is why the response time is increasing exponentially when password length is increased.

[Shashank-In]

Also @swdee

I had the same observation with a slight difference on avawallet website.

Note: I went really cautious and did not cause any disruption.

Password length: 512 Response time: 7 seconds approx

POST /ext/keystore HTTP/1.1
Host: bootstrap.ava.network:21000
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:76.0) Gecko/20100101 Firefox/76.0
Accept: application/json, text/plain, */*
Accept-Language: en-GB,en;q=0.5
Accept-Encoding: gzip, deflate
Content-Type: application/json;charset=UTF-8
Content-Length: 678
Origin: https://wallet.ava.network
Connection: close
Referer: https://wallet.ava.network/wallet

{
     "jsonrpc": "2.0",
     "id": 1,
     "method": "keystore.createUser",
     "params": {
         "username": "test102",
         "password": "JunT1237#usernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernameusernam"
     }
}

In localhost it was 5.5 seconds for me and on the wallet website, it was 7seconds and all the response time was quite similar.

I would be glad to know your views.

[swdee]

@Shashank-In I have done a quick profile of the full API execution path and now see the slow down your describing, it is actually the library doing the password strength test here https://github.com/ava-labs/gecko/blob/83502ee59e19bd93a2205753d3ff317bcde4c4a8/api/keystore/service.go#L151

I haven't dug any deeper yet to see why it slows, but have just updated this here to identify the cause.

[swdee]

Going back to the source library it has a note about latency here https://github.com/dropbox/zxcvbn#runtime-latency

zxcvbn operates below human perception of delay for most input: ~5-20ms for ~25 char passwords on modern browsers/CPUs, ~100ms for passwords around 100 characters. To bound runtime latency for really long passwords, consider sending zxcvbn() only the first 100 characters or so of user input.

So if we truncate the password to the first 100 characters that gets passed to zxcvbn() we can avoid the slow down and still test password strength using this library.

So by changing;

https://github.com/ava-labs/gecko/blob/83502ee59e19bd93a2205753d3ff317bcde4c4a8/api/keystore/service.go#L151

To the following would provide a suitable fix.

if zxcvbn.PasswordStrength(args.Password[:100], nil).Score < requiredPassScore { 

Limiting the check to 100 chars consumes around 20ms of execution time on my workstation.

Try it out and let me know if it solves your issue?

[Shashank-In]

Waoo that's quite helpful. I will update the code in my localhost and comeback with a new PR and my observations.

Thank you @swdee for your insights.

[Shashank-In]

Hi @swdee This was not the perfect fix as zxcvbn.PasswordStrength(args.Password[:100] it would lead to an error if the length is less than 100.

runtime error: slice bounds out of range [:100] with length 27

I have added checks for pass length and submitted a PR