Timed out

[mweel1]

I had to bring down the log server, and my go app stopped responding.

Any idea why this would happen? Should my go log calls be wrapped in a go routine?

Thank you

[nblumhardt]

Hi @mweel1!

I am not up to speed on the internals of this package, so apologies if this isn't useful info, but just in case you're stuck, Seq also accepts events in the GELF format via its Seq.Input.Gelf plug-in, and I think GELF transports are just about always non-blocking.

There are a couple of GELF hooks/formatters for Logrus out there.

HTH!

[mweel1]

@nblumhardt Hi,

Thanks for getting back to me, yeah it looked like it was blocking/sync on the http requests.

I implemented this, if you ever need it.

It basically uses a message queue and worker processes, and drops messages if the queue size reaches 5000.

We saw a huge performance boost in doing this.

import (
    "bytes"
    "fmt"
    "io/ioutil"
    "net/http"
    "time"

    "github.com/LyricTian/queue"
    "github.com/sirupsen/logrus"
)

// SeqHook sends logs to Seq via HTTP.
type SeqHook struct {
    endpoint string
    apiKey   string
    levels   []logrus.Level
    q        *queue.Queue
}

// NewSeqHook creates a Seq hook for logrus which can send log events to the
// host specified, for example:
//     logruseq.NewSeqHook("http://localhost:5341")
// Optionally, the hook can be used with an API key, for example:
//     logruseq.NewSeqHook("http://localhost:5341",
//         logruseq.OptionAPIKey("N1ncujiT5pYGD6m4CF0"))
// Optionally, which levels to log can be specified:
//     logruseq.NewSeqHook("http://localhost:5341",
//         logruseq.OptionLevels([]logrus.Level{
//             logrus.WarnLevel,
//             logrus.ErrorLevel,
//             logrus.FatalLevel,
//             logrus.PanicLevel,
//         }))
func NewSeqHook(host string, queue *queue.Queue, opts ...func(*SeqHookOptions)) *SeqHook {
    sho := &SeqHookOptions{
        levels: []logrus.Level{
            logrus.TraceLevel,
            logrus.DebugLevel,
            logrus.InfoLevel,
            logrus.WarnLevel,
            logrus.ErrorLevel,
            logrus.FatalLevel,
            logrus.PanicLevel,
        },
    }

    for _, opt := range opts {
        opt(sho)
    }

    endpoint := fmt.Sprintf("%v/api/events/raw", host)

    return &SeqHook{
        endpoint: endpoint,
        apiKey:   sho.apiKey,
        levels:   sho.levels,
        q: queue,
    }
}

func (h *SeqHook) copyEntry(e *logrus.Entry) *logrus.Entry {
    entry := logrus.NewEntry(e.Logger)
    entry.Data = make(logrus.Fields)
    entry.Time = e.Time
    entry.Level = e.Level
    entry.Message = e.Message
    for k, v := range e.Data {
        entry.Data[k] = v
    }
    return entry
}

// Fire sends a log entry to Seq.
func (hook *SeqHook) Fire(entry *logrus.Entry) error {  

    entry = hook.copyEntry(entry)

    // if our queue has 5000 items were overloading, lets start dropping logs       
    if (hook.q.GetJobCount() > 5000) {
        return nil;
    }

    hook.q.Push(queue.NewJob(entry, func(v interface{}) {
        hook.postSeq(v.(*logrus.Entry))
    }))

    return nil
}

func (h *SeqHook) postSeq(entry *logrus.Entry) {

    formatter := logrus.JSONFormatter{
        TimestampFormat: time.RFC3339Nano,
        FieldMap: logrus.FieldMap{
            logrus.FieldKeyMsg:   "@mt",
            logrus.FieldKeyLevel: "@l",
            logrus.FieldKeyTime:  "@t",
        },
    }

    data, _ := formatter.Format(entry)

    http.DefaultClient.Timeout = 10 * time.Second

    req, err := http.NewRequest("POST", h.endpoint, bytes.NewReader(data))

    if (err != nil) {
        fmt.Println("Error sending log to seq: ", err)      
        return;
    }

    req.Header.Set("Content-Type", "application/vnd.serilog.clef")

    if h.apiKey != "" {
        req.Header.Add("X-Seq-ApiKey", h.apiKey)
    }

    resp, err := http.DefaultClient.Do(req)

    if (err != nil) {
        fmt.Println("Error sending log to seq: ", err)

        return;
    }

    defer resp.Body.Close()

    if resp.StatusCode != http.StatusCreated {
        data, _ = ioutil.ReadAll(resp.Body) 
        fmt.Errorf("error creating seq event: %v", string(data))
    }

}

// Levels returns the levels for which Fire will be called.
func (hook *SeqHook) Levels() []logrus.Level {
    return hook.levels
}

// SeqHookOptions collects non-default Seq hook options.
type SeqHookOptions struct {
    apiKey string
    levels []logrus.Level

}

// OptionAPIKey sets the Seq API key option.
func OptionAPIKey(apiKey string) func(opts *SeqHookOptions) {
    return func(opts *SeqHookOptions) {
        opts.apiKey = apiKey
    }
}

// OptionLevels sets the levels for which Fire will be called.
func OptionLevels(levels []logrus.Level) func(opts *SeqHookOptions) {
    return func(opts *SeqHookOptions) {
        opts.levels = levels
    }
}```

[nblumhardt]

Awesome!

Unless you plan to use message templates with named holes in your Logrus messages, you might get slightly better handling on the Seq side by using the @m rather than @mt property for the message.

Thanks for sharing your solution! 😎

[alxyng]

Hey @mweel1 thanks for raising this. It looks like we need an option for a timeout on the http client, it's probably a good idea to use a dedicated client instead of the built in one too.