my script ends and i'm missing some messages on graylog

[navihtot]

Hi,

When my script executes, few last messages are not pushed to graylog. i.e. last message "All done" is never delivered. What can I do to fix this?

[navihtot]

Doing this for now:

                    setTimeout(function() { //sleep for graylog to get last messages
                        callback(null);
                    }, 10);

[ronkorving]

How does your script finish? Do you call process.exit()?

[navihtot]

Hmm... not really sure. my code runs on AWS Lambda and I create a handler function which is then runned. Don't really now how they implemented it. If it helps function definition looks like this:

exports.handler = function(event, context, callback)

more docs: http://docs.aws.amazon.com/lambda/latest/dg/nodejs-prog-model-handler.html

[ronkorving]

Can you show me a more complete example of what you're running?

[ronkorving]

I do think I understand the reason though. Our log functions don't have callback arguments, but they do asynchronous work. If you bail out (in your case, calling your callback which may kill a process in that tick) there are 2 things that won't have had a chance to complete: gzip compressing of your log message, UDP-send of your message.

This is a fundamental issue in this module. We need to work on this.

[navihtot]

Well really nothing special. Here is my log.write function from log module:

module.exports.write = function (title, message) {
    if (typeof message === 'undefined') {
        message = "";
    }
    if ( process.env.NODE_ENV !== 'test' ) {
        console.log(module.exports.timestamp()+'\t'+title+'\t'+message);
        if (message!=="") {
            if (title=='Error') logger.error(message, message);
            else if (title=='Warning') logger.warning(message, message);
            else if (title=='Info') logger.info(message, message);
            else logger.error(title, message);
        }

    }
}

and my main program function:

/**
 * Main event handler for Lambda
 *
 * @param {Object} event AWS Lambda event
 * @param {Object} context AWS Lambda context
 * @callback callback function(err)
 *      @param {Error} err
 */
exports.handler = function(event, context, callback) {
    var operation = event.operation;
    delete event.operation;
    if (operation) {
        log.write('Info', 'Operation ' + operation +' requested ');
    }
    //create tmp folder for this lambda session
    fs.createFolderLocal('/tmp/'+common.getGuid());    

    switch (operation) {
        case 'transform':                
            exports.transform(event,context,  function (err) {
                if (err) context.fail(err);
                else {
                    setTimeout(function() { //sleep for graylog to get last messages
                        callback(null);
                    }, 10);
                    //context.done();
                }
            });
            break;
        default:
            context.fail(new Error('Unrecognized operation "' + operation + '"'));
    }
};

trough the rest of the code i just call log.write. But yeah i suppose the problem is that lib functions should have async and sync way of calling (with callback or without) like its a common style with node.js

[ronkorving]

There is no sync way to call functions that do UDP, but we need to allow callbacks to wait for gzipping and sending to complete (although iirc the callback on UDP sends is really a callback post-DNS-resolve).

[Fl4m3Ph03n1x]

Hello,

I have this exact issue on my code right now. I need to wait for the logs to finish processing, but my app exits before that.

I see you are working on a new implementation. Is there any ETA? Will it be backwards compatible?