Closed codeurimpulsif closed 1 year ago
It seems this issue is Debian-specific. I'm unable to reproduce this using Ubuntu.
To reproduce it, you can use either the docker image savonet/liquidsoap:v2.1.4 or the virtual machine with Debian bullseye.
It's possible to use self-signed certs.
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -sha256 -days 365 -subj '/CN=localhost' -nodes
settings.harbor.ssl.certificate.set("cert.pem")
settings.harbor.ssl.private_key.set("key.pem")
live = input.harbor.ssl(
"live",
port=8005,
password="test",
icy=true,
)
output.dummy(mksafe(live))
The problem happens because the icy=true
parameter is set, causing liquidsoap to begin listening on
port=8005
with icy=false
port=8006
with icy=true
Then use nc -z localhost 8005
from the openbsd-netcat
package to connect to the server. It can be performed from anywhere.
The same problem exists in 2.2.0+git@81016089f.
ssl_transport = http.transport.ssl(
certificate="cert.pem",
key="key.pem",
)
live = input.harbor(
"live",
transport=ssl_transport,
port=8005,
password="test",
icy=true,
)
output.dummy(mksafe(live))
For now I just try to remove all icy
parameters and try again with netcat but the issue is the same.
I will try with the Docker image later maybe.
Ok, I have tried with the Debian Docker image and the issue seem still here: lot of CLOSE_WAIT
state connections never flushed.
But I can still connect because it never hit the limit of the log message Too many open files in accept()
.
So I don't know, maybe I will try to compile the binary without using the Debian package on the machine where the issue occur.
@codeurimpulsif, Can you check if liquidsoap has rights to read the cert and key files?
settings.harbor.ssl.certificate.set("#{TLS_CERTIFICATE_PATH}/fullchain.pem")
settings.harbor.ssl.private_key.set("#{TLS_CERTIFICATE_PATH}/privkey.pem")
@vitoyucepi Yes rights are ok, it's not a permission issue
Found the same problem after I changed the rights for key.pem
file and reduced the nofile limit to 1024.
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: SSL: Privte key error: error:8000000D:system library::Permission denied
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: SSL: Privte key error: error:8000000D:system library::Permission denied
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: SSL: Privte key error: error:8000000D:system library::Permission denied
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: SSL: Certificate error: error:80000018:system library::Too many open files
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: Too many open files in accept()
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: Too many open files in accept()
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: Too many open files in accept()
2023/05/09 16:03:25 [harbor:2] Failed to accept new client: Too many open files in accept()
Check ulimit -n
on your raspberry.
Also spelling errors https://github.com/savonet/ocaml-ssl/blob/dfba6793d939b46f55f0452bcbe04759286aa3fe/src/ssl.ml#L105 https://github.com/savonet/ocaml-ssl/blob/dfba6793d939b46f55f0452bcbe04759286aa3fe/src/ssl.ml#L108
@vitoyucepi Yes on the raspberry pi the ulimit is 1024.
Of course I can increase it but I think it will just hide the real problem (the CLOSE_WAIT
connections who are never flushed)?
Reproduction
Get debian 11 image. I use genericcloud-amd64 from https://cloud.debian.org/images/cloud/bullseye/20230501-1367/ With cloud-init config.
wget https://github.com/savonet/liquidsoap/releases/download/v2.1.4/liquidsoap_2.1.4-debian-bullseye-1_amd64.deb
sudo apt-get install --no-install-recommends ./liquidsoap_2.1.4-debian-bullseye-1_amd64.deb
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -sha256 -days 365 -subj '/CN=localhost' -nodes
Write config
liquidsoap 1.liq
netcat-openbsd
.
sudo apt-get install netcat-openbsd
.nc -z localhost 8005
.ss -tapn | grep CLOSE
Thanks for reporting. This could point to file descriptor leak from libssl
but, at any rate, we were instantiating a new context for each new connection, which doesn't seem standard according to: https://wiki.openssl.org/index.php/Simple_TLS_Server
I pushed a PR fixing this here: https://github.com/savonet/liquidsoap/pull/3071 Any chance you could test? This would be with the v2.2.x
API..
2.2.0+git@27ec13733 looks the same
ss -tapn | grep CLOSE
This should be fixed now!
@vitoyucepi what do you use to generate certificate and key?
In the tests, we use:
openssl req -x509 -newkey rsa:4096 -keyout ssl.key -out ssl.cert -sha256 -days 3650 -nodes -subj /C=XX/ST=StateName/L=CityName/O=CompanyName/OU=CompanySectionName/CN=localhost
Then:
transport = http.transport.ssl(
certificate="./ssl.cert",
key="./ssl.key"
)
port = random.int(min=8000, max=10000)
s = sine()
output.icecast(
port=port,
mount="ssl_test",
transport=transport,
%vorbis,
s)
i = input.harbor(
buffer=2.,
port=port,
transport=transport,
"ssl_test")
i = source.on_track(i, fun (_) -> test.pass())
output.dummy(fallible=true, i)
@vitoyucepi what do you use to generate certificate and key?
Generally I use this command
openssl req -x509 -newkey rsa:2048 -keyout key.pem -out cert.pem -sha256 -days 365 -subj '/CN=localhost' -nodes
There's another test.
[harbor:2] Failed to accept new client: SSL: Privte key error: error:8000000D:system library::Permission denied
To get this you have to chmod 0000 key.pem
, or run from a user, that has no access to key file.
icy=true
input.harbor.ssl(
"live",
port=8005,
password="test",
icy=true,
)
I'll check the release artifacts from https://github.com/savonet/liquidsoap/actions/runs/4972860470
Liquidsoap 2.2.0+git@4af8db175
The same problem as before.
ss -tapn | grep CLOSE
If I change the rights, then server will shutdown after a bunch of errors.
Thanks for insisting @vitoyucepi This is confirmed fixed in https://github.com/savonet/liquidsoap/commit/57d489745242256b4aaca581eacc1542dd8c8660 and in the latest rolling-release-v2.2.x
Describe the bug
I use
input.harbor.ssl
to input external stream but after some time (I don't know exactly how many time, few days for sure) I can't connect anymore.Note: This issue seem similar to #1403 where I leave a comment.
To Reproduce
The script:
A netstat (
netstat -altupn | grep <harbor-port> | grep "CLOSE_WAIT"
) or ss (ss -tap state CLOSE-WAIT | grep "liquidsoap"
) command show me there is a lot of open connections (250 exactly) to the harbor port inCLOSE_WAIT
state.I can reproduce the issue by sending lot of connections (using
nc -z <harbor-ip> <harbor-port>
, curl don't seem to reproduce), after some time I can't connect anymore on harbor port and open connections in CLOSE_WAIT state are still here, never flushed.Liquidsoap logs shows
Failed to accept new client: SSL accept() error: error:00000000:lib(0):func(0):reason(0)
each time I open a connection (and of course add a connection inCLOSE_WAIT
state to the previous netstat or ss list).Then at some point (I can't say when exactly) a normal streaming source (I use Butt) can't connect anymore.
Then it reach the limit with log message
Failed to accept new client: Too many open files in accept()
, but only when I try to reproduce with this test. In normal conditions it take days to append and there is noToo many open files in accept()
log messages, only theSSL accept() error
ones.Expected behavior
Version details
2.1.4-debian-bullseye-1
Install method
Debian package from this Github repository