search

kiwix / kiwix-build

Kiwix & openZIM build engine
GNU General Public License v3.0
90 stars 43 forks source link

Use a scp cipher which goes faster #622

Closed kelson42 closed 1 year ago

kelson42 commented 1 year ago

Attempt to speed-up scp uploads

kelson42 commented 1 year ago

@rgaudin scp uploads fail sporadically with client_loop: send disconnect: Broken pipe. This is a challenge we have worked on this the past with the Zimfarm uploader. Do you see other things we should/could do to improve the situation (if we could speed-up the upload this would welcome as well).

rgaudin commented 1 year ago

We use the same aes128-ctr cipher in our zimfarm uploader.

I don't think there's much more we can do regarding speed ; and we should keep in mind that this service is receiving files from multiple ends, and serving a lot of things, all that from a mechanical disk.

Regarding the sporadic failures, I believe those are caused by the server, as previously discussed. Here's the sshd_config with the following interesting options

#MaxAuthTries 6
#MaxSessions 10
MaxSessions 20

# Send a alive requests every 5 seconds
#ClientAliveInterval 0
ClientAliveInterval 30

# Disconnect after 10 alive requests unasnwered
#ClientAliveCountMax 3
ClientAliveCountMax 10

#MaxStartups 10:30:100
MaxStartups 100

Many interpretations of the doc regarding those have been tried but it's difficult to evaluate properly because we haven't been able to willingly reproduce the issue that is mostly tied to certains conditions and a number of simultaneous clients.