search

Moocar / logback-gelf

Logback plugin to send GELF messages to graylog2 server
Apache License 2.0
147 stars 59 forks source link

Chuncking issue #42

Closed cloon77 closed 9 years ago

cloon77 commented 9 years ago

Hi, we are using graylog2 (0.20.x) and will move to 1.x really soon but we noticed that we are not able to send message bigger than the chuncktreshold(they get "lost"). On this site https://www.graylog.org/resources/gelf-2/ its said the library should be sending "1.1" as version and various other elements for the chucking to work.
Can this library work with newer version than 0.9.6 ?

Regards, Cloon_

Moocar commented 9 years ago

Thanks for pointing out the new gelf docs. From the documentation, "A message MUST NOT consist of more than 128 chunks.". So I'm not aware of any way to support chunking of more than 128 chunks.

I just read through the latest 1.1 GELF spec. I can't actually spot any differences between 1.1 and 1.0. I'm checking with the mailing list to see what would be required to upgrade to GELF 1.1, but from what I've seen so far, everything should just work with the latest graylog server.

cloon77 commented 9 years ago

I guess the version thing should be fixed right? 1.0 and 1.1 are using the same "magic" number? I didn,t get to see the official spec or badly checked because it seems to be new from the link I pasted

cloon77 commented 9 years ago

About the 128 chunks.. in one Gelf message no but what about generating more than one gelf message.. just an idea.. maybe it can be dangerous but at least everything would get loggued.

Moocar commented 9 years ago

I've just massively reworked logback-gelf on a new branch. The new version should be fully compatible with Graylog 1.0 (GELF 1.1). The default chunk threshold for chunking is now 512 bytes which is actually less than the previous version. I used this default as it seems to be the safest for general use. 512 * 128 chunks leaves a log size of 64kb. Unzipped. With Gzip (highly recommended for UDP), you'd be able to fit much more than that.

If you're operating within one datacenter with sane networking, then you can increase the chunkThreshold to whatever you like using the GelfLayout.maxPacketSize field. According to Wikipedia, the theoretical maximum size is 65'607 bytes, which gives you an overall log size of ~ 8 MB unzipped. Have a play with the settings and see how you go.

I just finished the branch, any chance you'd be interested in testing it? I've done a fair bit, but it would be great to get some feedback. You'll need to build both socket-encoder-appender and logback-gelf from scratch as they're not on maven central yet

https://github.com/Moocar/socket-encoder-appender https://github.com/Moocar/logback-gelf/tree/the-great-appender-rejig

Moocar commented 9 years ago

logback-gelf 0.2beta1 is released an has more thorough docs around chunking. Hopefully it will avoid confusion in future. Closing this issue.