Slow execution due to low entropy in /dev/random

[cyberduck]

e3d6251 created the issue

• OS: Ubuntu 16.04
• Java: openjdk version "11.0.1" 2018-10-16
• Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

[cyberduck]

e3d6251 commented

Note: in rare cases the response is immediate

[cyberduck]

2ad25be commented

Replying to [10604 jkvbe]:

It takes about 38 minutes.

$ time duck --help
Usage:duck [options...]

URLs must ...
... Purchase a donation key to support the development of this software.

real    38m37.287s
user    0m2.524s
sys 0m0.564s
$

I confirm behaviour since version 6.9.1.29776. Current testing with ubuntu 18.04 and version 6.9.3.30061 is still the same

On version 6.8.3.29107 duck CLI was running fine. It is independent from OS version (16.04 or 18.04). No java installed since this is no dependency.

• OS: Ubuntu 16.04
• Java: openjdk version "11.0.1" 2018-10-16
• Cyberduck version: 6.9.1 (29776)

duck -h takes more than 5 minutes to execute. I have the impression it runs against a timeout, but I don't see what resources duck needs to print the help information.

This happens for each duck command.

[cyberduck]

a1dfdd4 commented

Hey, we ran into this today and it appears as though the issues is with the use of /dev/random in certain calls that are being made.

If the system in use is low on entropy then /dev/random will block until there are enough bits to satisfy whatever request is being made.

How to test
On a system w/low entropy (Linux VMs that don't see a lot of CPU I/O are good candidates):
In one terminal or screen session run:
time duck -h

In another run:
lsof |grep duck|grep random
You will see calls to both /dev/urandom and /dev/random

Now on that same system (as root) do the following:
(if necessary "ctrl-c" your previously running duck session)
"mv /dev/random /dev/random.orig"
"cp -a /dev/urandom /dev/random"

Once you've replaced /dev/random with /dev/urandom run:
"time duck -h"
It should return a whole lot quicker.
In our case it went from 11m to 8s.
Don't forget to put your system back the way it was:
"rm -f /dev/random"
"mv /dev/random.orig /dev/random"

You can tell how much entropy is available (in bits) for /dev/random by:
"cat /proc/sys/kernel/random/entropy_avail"

In our test case where "duck -h" took 11m to run, we had about 118 bits of available entropy.

Some Suggestions for the devs:

1. Only be use /dev/random (or libraries that invoke it) if absolutely necessary; otherwise use /dev/urandom
   
2. Provide a run-time option that would allow the user to choose the use /dev/urandom exclusively
   

[cyberduck]

@dkocher commented

Because of this we actually run with -Djava.security.egd=file:/dev/./urandom. We will have to look into when/what libraries still use /dev/random directly.

[cyberduck]

a1dfdd4 commented

Suggestion for End-Users:

If you need duck to work now, and cannot wait for the devs (sometimes this happens).

You can increase the entropy of your systems' /dev/random by using something like haveged.

This acts as a quite decent source of randomness to feed /dev/random in a very efficient (non-CPU-intensive) way. And works excellently for VM/containers that have no access to more traditional sources of randomness.

There are some other projects and methods but something like haveged should be fairly available and infrastructure independent, so no need to worry about virtio / kvm / qemu ...

Also, a big thank you to the cyberduck team for their support.

-*Edit:* To add running duck -h* with haveged running with a low-watermark of 2000 (general recommendations but the debian/ubuntu default of 1024 is probably good enough for this case) caused the execution time to go from the 11m area to 3.5s.

[cyberduck]

@dkocher commented

Replying to [comment:10 frwessling]:

Thanks for sharing this!

[cyberduck]

a1dfdd4 commented

Replying to [comment:11 dkocher]:

Replying to [comment:10 frwessling]:

Thanks for sharing this!

My pleasure, glad to help.

[cyberduck]

@dkocher commented

Now documented.