Multi-threaded concurrent testing,i don't know the reason

[sxj1991]

[drakkan]

You probably exceed the maximum allowed open files, you can increase this limit using os commands/configurations,

however you cannot report something like this via 2 screenshots, you should at least:

1) share your load testing script 2) post some sftpgo logs

thank you

[sxj1991]

thanks,my friend. i test file 30M size,Multi-threaded concurrent 20 Thank you very much!!!

[drakkan]

20 concurrent uploads should not cause any issue, can you please share an executable, something I can use with java -jar or a compilable source code? Thank you

[sxj1991]

sftpgojar.zip thanks,if know why,please tell me,i will be learn more Programming knowledge

[drakkan]

I tryed your jar but I see no connection attempts to the sftpgo instance running on port 2022.

I did a quick test myself:

• I created 20 users, usernames are user1 to user20 and home dir /tmp/user1 to /tmp/user20
• I created a test file

ls -la file.img 
-rw-r--r-- 1 nicola nicola 81920000 19 apr 16.16 file.img

and finally executed 20 concurrent SFTP uploads this way:

for i in {1..20}; do
    curl -k -T file.img sftp://127.0.0.1:2022/ -u user$i:password &
done

no errors here:

ls -la /tmp/user*/
/tmp/user1/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.23 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user10/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user11/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user12/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user13/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user14/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user15/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user16/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user17/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user18/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user19/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user2/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user20/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.16 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user3/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user4/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user5/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user6/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user7/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user8/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

/tmp/user9/:
totale 80000
drwxr-xr-x  2 nicola nicola       60 19 apr 16.15 .
drwxrwxrwt 55 root   root       1460 19 apr 16.16 ..
-rw-r--r--  1 nicola nicola 81920000 19 apr 16.16 file.img

RSS usage and file descriptors for the sftpgo process are ok too.

Please try to debug the issue yourself and if something does not work for you please share a way to reproduce the issue with clear instructions, thank you

[sxj1991]

i will be try,thanks friend.

[drakkan]

Hi,

I did some more tests myself. I'm able to replicate the reported issue using a script like this one:

for i in {1..30}; do
    for i in {1..20}; do
        curl --verbose -k -T file.img sftp://127.0.0.1:2022/ -u user$i:password &
    done
done

so about 600 concurrent connections.

The problem here is the memory cost for the password authentication. It is by default 64MB, so you have to divide your available memory by 64MB to get the max concurrent authentications that your system can handle.

Once you know this number you can easily configure an appropriate rate limit. Rate limiting is not supported in any released version, you have to build yourself git main or download a recent build artificat or use the edge Docker image variant.

You can also decrease the password hashing memory cost. So this is basically a configuration issue.

As future enhancement I'll evaluate a configurable memory cache for passwords in order to reduce the memory cost for passwords authentication

[sxj1991]

hello, dear drakkan , i keep try in the Local environment,system is windows. I see my program 100 concurrent uploads is sccess. maybe is Remote server problem,maybe is my sftpgo config problem,but i still do not know yet

[drakkan]

Hi,

please read the above explaination for understand the reason. Your remote system probably has less RAM available than your local system. If you don't have many users, you can try the build artificat you find at the following link:

https://github.com/drakkan/sftpgo/actions/runs/766321007

I'll add an option to use bcrypt as password hashing too since bcrypt has an high CPU cost but a low memory cost and could help on memory constrainted system if you don't want to enable passwords caching

[drakkan]

You can also try to download the latest build artifact from the following link:

https://github.com/drakkan/sftpgo/actions/runs/767022358

and, in sftpgo.json, change:

"password_hashing_algo": "argon2id",

to

"password_hashing_algo": "bcrypt",

After you change the password_hashing_algo you need to remove and recreate any existing user (or change their passwords) to apply the new setting.

This could help, anyway you should understand what is happening on your server and configure SFTPGo appropriately. It is quite hard to have a default configuration that works for all use cases. Thank you

[sxj1991]

hi,Thank you for your help drakkan. you are right, my server is less RAM. so i change the other serve.This time i see sftpgo work is great. I didn't know what to do for a while,Luckily there's you.

[drakkan]

Thank you for confirming, much appreciated.

Please note that SFTPGo works just fine also on memory constrained systems, you only have to change some configuration options as explained above, thank you!