Open borjam opened 2 months ago
I was just researching this last week. What i found is that if you're able to configure net.core.rmem_max
you can make the error go away:
# The maximum socket receive buffer size which may be set by using the SO_RCVBUF socket option
# Applies immediately, does not persist after a reboot
sudo sysctl -w net.core.rmem_max=1048576
# persists on reboot, but does not apply immediately
echo 'net.core.rmem_max=1048576' | sudo tee -a /etc/sysctl.conf
I validated the above resolves this.
This may be more of a documentation issue, what do you think?
Sorry, different issues.
You are talking about Linux. I am talking about FreeBSD. Not sure about other systems because the problem might be on the netty library or the JDK itself.
So, what happened to me on FreeBSD is this:
Either setting the graylog.conf variable or the input buffer size for a given input to a value greater than 65536 bytes results in the buffer size being actually capped.
I confirmed that it is not a problem in my system writing a silly C program. I will attach it below.
Suspecting some kind of buffer size capping I changed the UDP buffer size parameter to 0 on graylog.conf. It is an illegal value of course. But the call fails, as it should, and Graylog just logs a warning and goes on.
The result of the workaround: The buffer size is the one I specified using sysctl net.inet.udp.recvspace.
Now, a small clarification. Not intending to be disrespectful at all, but some people are not aware that FreeBSD is not a Linux distribution. It's an entirely different operating system. Linux also behaves in a really odd way because when you set a buffer size with setsockopt() and you read it back with getsockopt() it returns the specified buffer size multiplied by two.
So, what I think would make all of this work better is:
It would be nice to document the issue anyway. There should be a way to ensure that Graylog does not mess with the buffer size and trusts whatever size has been configured by the system aministrator. It can work with the workaround I found (udp_recvbuffer_sizes=0) which is an ugly kludge anyway, or it can be done better by changing Graylog's behavior when the udp_recvbuffer_sizes variable is omitted.
Now I am running Graylog with udp_recvbuffer_sizes=0 and making sure I don't specify any buffer size for the UDP inputs and it shows the buffer size I wanted (1 MB).
That 65536 byte limit is worth investigating, anyway. Maybe it happens on other platforms.
Checking program: checkbuf.c
/*
* checkbuf.c
* Verifies UDP buffer size
*/
#include <stdio.h>
#include <stdlib.h>
#include <stddef.h>
#include <sys/types.h>
#include <sys/socket.h>
int main(int argc, char **argv)
{
int s, r;
unsigned int bufsiz = 0, actual_bufsiz = 0, len = 0;
if ( argc > 1 )
bufsiz = atoi(argv[1]);
s = socket(PF_INET, SOCK_DGRAM, 0);
if ( s < 0 ) {
perror("socket()");
exit(1);
}
if ( bufsiz > 0 ) {
r = setsockopt(s, SOL_SOCKET, SO_RCVBUF, &bufsiz, sizeof(bufsiz));
if ( r < 0 ) {
perror("setsockopt()");
exit(1);
}
}
/* Now verify */
len = sizeof(actual_bufsiz);
r = getsockopt(s, SOL_SOCKET, SO_RCVBUF, &actual_bufsiz, &len);
if ( r < 0 ) {
perror("getsockopt()");
exit(1);
}
if ( bufsiz > 0 )
printf("SO_RCVBUF: requested: %d, actual: %d\n", bufsiz, actual_bufsiz);
else
printf("SO_RCVBUF: %d\n", actual_bufsiz);
exit(0);
}
My apologies, you did say BSD :)
Also for reference, noting the original discussion via the community forum https://community.graylog.org/t/udp-buffer-size-problem-despite-system-configuration/32726
Just in case it was a JDK bug I "checked" the netty source code and I don't find anything suspect of capping the value to 16 bit. Relevant variables in the associated C code are defined as "int" which should be 32 bit.
And looking at the OpenJDK 17 source code, /usr/local/openjdk17/include/freebsd/jni_md.h also defines "jint" as "int". As it should be.
At least the workaround is working until someone decides to abort if a setsockopt() with an illegal value returns an error :)
Thanks for your attention! 👍
Expected Behavior
When setting a buffer size for UDP inputs, either on the Graylog config file (udp_recvbuffer_sizes parameter), the value should be provided to the setsockopt() call. The operating system should decide whether the value is legal or not.
This should work for a value greater than 65536 bytes.
Current Behavior
There are two different issues at play here:
The behavior on FreeBSD is rather confusing. Moreover I have found a workaround that suggests a potential fix for any operating system.
On a FreeBSD system configured with a UDP buffer size of 1048576 bytes (1 MB), Graylog gives the following warning:
2024-06-14 12:05:08,300 WARN o.g.i.t.UdpTransport [netty-transport-0] receiveBufferSize (SO_RCVBUF) for input RawUDPInput{title=Alertas DNS ML, type=org.graylog2.inputs.raw.udp.RawUDPInput, nodeId=XXXXXX} (channel [id: 0x75163fc7, L:/10.0.0.1:555]) should be >= 262144 but is 65536.
And the UDP buffer size is indeed 65536 bytes, checking with netstat -x -p udp (not sure these options work on Linux)
This happens in any of these cases:
Possible Solution
Workaround on FreeBSD:
When doing this, Graylog makes a setsockopt(SO_RCVBUF) with 0 specified as a buffer length. The call will fail, so the actual buffer size set by the system to the default value will not change. Graylog issues a warning but it moves forward.
2024-06-15 08:32:31,391 WARN i.n.b.Bootstrap [inputs-0] Failed to set channel option 'SO_RCVBUF' with value '0' for channel '[id: 0x30026dd0]'
Suggested fix:
There is a cap to 65536 bytes somewhere. It may be netty, Graylog or OpenJDK. I don't know.
Anyway in my opinion the sane, foolproof option is to change the default behavior:
Doing this, Graylog will trust whatever default UDP buffer size has been configured by the system administrator which is probably the best option. I am not sure about other systems but at least on FreeBSD there are two system variables controlling this:
Steps to Reproduce (for bugs)
Context
This issue is important because the UDP buffer size is critical for high volume UDP packet reception.
Your Environment