UDP Buffer Size reset on restart of node

[jasonkeller]

Expected Behavior

When a UDP buffer setting is changed on an input, the expectation would be that the setting would be saved in MongoDB and saved across restarts.

Current Behavior

Currently the udp_recvbuffer_sizes setting in the server.conf overrides this, even if not set it defaults to 1MB.

Steps to Reproduce (for bugs)

Simply resize the buffer on a UDP input, then restart the node.

Context

Small buffers = dropped messages. I can get bursts of UDP datagrams that flood 1MB buffers easily.

Your Environment

• Graylog Version: 2.3.2
• Elasticsearch Version: 5.6.3-1
• MongoDB Version: 0:3.4.9-1.el7
• Operating System: RHEL7
• Browser version: Chrome 62.0.3202.89 on Windows 10

[joschi]

I'm unable to reproduce that with Graylog 2.3.2. The settings for the (GELF) UDP input are persisted and correctly loaded on restart.

Example screenshot (GELF UDP input with 512k receive buffer size):

Relevant data from MongoDB:

$ mongo 127.0.0.1/graylog
MongoDB shell version v3.2.10
connecting to: mongodb://127.0.0.1/graylog
MongoDB server version: 3.2.10
> db.inputs.find().pretty()
{
    "_id" : ObjectId("5a05760d2ab79c0001a5ce55"),
    "creator_user_id" : "admin",
    "configuration" : {
        "override_source" : null,
        "recv_buffer_size" : 524288,
        "bind_address" : "0.0.0.0",
        "port" : 12201,
        "decompress_size_limit" : 8388608
    },
    "name" : "GELF UDP",
    "created_at" : ISODate("2017-11-10T09:49:01.794Z"),
    "global" : false,
    "type" : "org.graylog2.inputs.gelf.udp.GELFUDPInput",
    "title" : "GELF UDP (512k)",
    "content_pack" : null,
    "node_id" : "f20ddb81-c96c-4629-ad42-734ffc8b8907"
}

docker-compose.yml for test setup:

version: '2'
services:
  mongodb:
    image: mongo:3
    ports:
      - 27017:27017
    mem_limit: 64m
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.3
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      - xpack.security.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 512m
  graylog:
    image: graylog/graylog:2.3.2-1
    mem_limit: 2g
    environment:
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      - 9000:9000
      - 12201:12201/udp

The custom configuration for the socket receive buffer size is stored in MongoDB for each input.

https://github.com/Graylog2/graylog2-server/blob/3df951e9c88a5e3a4cfe82f8c4861d05075051b5/graylog2-server/src/main/java/org/graylog2/plugin/BaseConfiguration.java#L108-L109

https://github.com/Graylog2/graylog2-server/blob/3df951e9c88a5e3a4cfe82f8c4861d05075051b5/graylog2-server/src/main/java/org/graylog2/plugin/inputs/transports/NettyTransport.java#L96-L98

@jasonkeller Make sure to use the same node ID (see node_id_file) throughout node restarts. Local inputs (in contrast to global inputs) are using the node ID to "bind" to a specific node.

[jasonkeller]

The difference here being you used a Local input; with a cluster you'll be using more than likely Global inputs, which are not persisted it would seem.

[joschi]

@jasonkeller I just tried to reproduce your issue with a global GELF UDP input on Graylog 2.3.2 with the exactly same result. The UDP receive buffer size is stored in MongoDB and applied when starting the global input, also after a Graylog node restart.

$ mongo 127.0.0.1/graylog
MongoDB shell version v3.2.10
connecting to: mongodb://127.0.0.1/graylog
MongoDB server version: 3.2.10
> db.inputs.find().pretty()
{
    "_id" : ObjectId("5a0c5ac82ab79c00017d0682"),
    "creator_user_id" : "admin",
    "configuration" : {
        "override_source" : null,
        "recv_buffer_size" : 524288,
        "bind_address" : "0.0.0.0",
        "port" : 12201,
        "decompress_size_limit" : 8388608
    },
    "name" : "GELF UDP",
    "created_at" : ISODate("2017-11-15T15:18:32.424Z"),
    "global" : true,
    "type" : "org.graylog2.inputs.gelf.udp.GELFUDPInput",
    "title" : "GELF UDP",
    "content_pack" : null
}