Open pixelshaded opened 8 years ago
Can you create a repro for this?
How would I go about doing that? My graylog server isn't public. Do you have a graylog server to run against?
You can't reproduce against localhost?
What sort of messages are you sending?
The concern that "logging should never throw" is also a bit of a red herring – if your process is running out of memory, and it just happens to be when gelf-stream is allocating, then that's not an issue with gelf-stream – there's nothing gelf-stream can do to prevent that (especially if you're sending very large messages/objects). It's only an issue with gelf-stream if it's leaking memory in some way.
I am getting the same problem, it seems to happen on complex nested objects, because on simple string or object it works
@Migo2014 is it a recursive object of some kind?
yes, I wrote too early, sorry. It's mongoose objects that make it crash, since it worked on regular bunyan, I didn't think about it.
After crashes in production and a long investigation I found that the flatten() does not detect circular references and generates a infinite recursion. This is a big problem because as @pixelshaded mentioned the logger should never throw but crashing your process is even more serious.
Basically there are no checks of circular references in flatten():
function flatten(obj, into, prefix, sep) {
if (into == null) into = {}
if (prefix == null) prefix = ''
if (sep == null) sep = '.'
var key, prop
for (key in obj) {
if (!Object.prototype.hasOwnProperty.call(obj, key)) continue
prop = obj[key]
if (typeof prop === 'object' && !(prop instanceof Date) && !(prop instanceof RegExp))
flatten(prop, into, prefix + key + sep, sep)
else
into[prefix + key] = prop
}
return into
}
I think is important to be able to opt-in for flatten and not be obliged to use it. I prefer using a Graylog extractor instead. This is related: https://github.com/mhart/gelf-stream/issues/11
Here is a workaround:
+function safeCycles() {
+ var seen = [];
+ return function (key, val) {
+ if (!val || typeof val !== 'object') {
+ return val;
+ }
+ if (seen.indexOf(val) !== -1) {
+ return '[Circular]';
+ }
+ seen.push(val);
+ return val;
+ };
+}
- var stream = gelfStream.forBunyan(logOptions.server);
+ var bunyanGelfLevels = {
+ '10': 7, // bunyan.TRACE - gelfling.DEBUG
+ '20': 7, // bunyan.DEBUG - gelfling.DEBUG
+ '30': 6, // bunyan.INFO - gelfling.INFO
+ '40': 4, // bunyan.WARN - gelfling.WARNING
+ '50': 3, // bunyan.ERROR - gelfling.ERROR
+ '60': 0, // bunyan.FATAL - gelfling.EMERGENCY
+ };
+ var stream = gelfStream.create(logOptions.server, {
+ map: function (log) {
+ return {
+ host: log.hostname,
+ timestamp: +new Date(log.time) / 1000,
+ short_message: log.msg,
+ facility: log.name,
+ level: bunyanGelfLevels[log.level],
+ full_message: JSON.stringify(log, safeCycles(), 2)
+ };
+ }
+ });
It seems that bunyan-gelf has the same problem: https://github.com/hughsk/flat/issues/24#issuecomment-273439342
@pixelshaded Can you please change the title to something like "Crashes process when serializing objects with circular references: process out of memory" to be more precise?
Getting this in stdout
This scares me only because logging should never throw and it brings down the whole server.