Consider publishing tailer separately

[sgtsquiggs]

Hi,

I'm currently using a chunk of mtail as a replacement for hpcloud/tail. I was previously pulling google/mtail/[tailer,watcher,logline], but with the latest changes on master this isn't possible anymore. I agree with the move to internal and I can always maintain my own copy and make updates but it would be very useful to have the tail implementation in mtail exposed as a standalone library. It's outside of the scope of mtail to expose a library, but maybe useful to the golang community as a whole :)

[jaqx0r]

Do you use all of tailer and watcher? I'm not really a fan of them at the moment, but if you can tell me what you like then we can figure out something.

I expected someone to want to do this, so I'm glad you asked.

On Fri, 21 Dec 2018 at 06:48, Matthew Crenshaw notifications@github.com wrote:

Hi,

I'm currently using a chunk of mtail as a replacement for hpcloud/tail https://github.com/hpcloud/tail. I was previously pulling google/mtail/[tailer,watcher,logline], but with the latest changes on master this isn't possible anymore. I agree with the move to internal and I can always maintain my own copy and make updates but it would be very useful to have the tail implementation in mtail exposed as a standalone library. It's outside of the scope of mtail to expose a library, but maybe useful to the golang community as a whole :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/199, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-wagTDNbivFdSmdvvFZOLpkxak5oks5u6-mdgaJpZM4Zczvc .

[sgtsquiggs]

My usage is very roughly like:

func (t *Tail) Start() error {
    t.Lock()
    defer t.Unlock()
    t.lines := make(chan *logline.LogLine)
    t.watcher := watcher.NewLogWatcher(pollDuration, !poll)
    if err != nil {
        defer close(t.lines)
        return err
    }
    var opts []func(*tailer.Tailer) error
    if t.FromBeginning {
        opts = append(opts, tailer.OneShot)
    }
    t.tailer, err := tailer.New(t.lines, afero.NewOsFs(), t.watcher, opts...)
    if err != nil {
        defer t.watcher.Close()
        defer close(t.lines)
        return err
    }
    t.wg.Add(1)
    go t.processLines()
    return t.tailNew()
}

func (t *Tail) Stop() error {
    t.Lock()
    defer t.Unlock()

    err := t.tailer.Close()
    t.wg.Wait()
    return err
}

// never errors but that's ok for now
func (t *Tail) tailFiles() error {
    for _, filepath := t.Files {
        // snip globbing stuff i'm doing myself
        for file := range globbed {
            if _, ok := t.files[file]; ok {
                continue
            }
            err := t.tailer.TailPath(file)
            if err != nil {
                t.log.Error(err)
                continue
            }
            t.log.Debug("start tailing %s", file)
            t.files[file] = true
        }
    }
    return nil
}

func (t *Tail) processLines() {
    defer t.wg.Done()

    for line = range t.lines {
        // do stuff with line.Line depending on line.Filename
    }

    for _, file := range t.files {
        t.log.Debug("stop tailing %s", file)
    }
}

Other methods add/remove files to be tailed on the fly

[jaqx0r]

Ok so you use the same core functions as mail does. I think I'm happy to move it out of internal and let you depend on it directly.

I don't want to require use of fsnotify though, so the watcher will change, and I don't want to force glog on anyone so that'll change too.

I don't think I'm comfortable with the obligation of making it a public API but I'm sure it'll work out.

On Sat., 22 Dec. 2018, 06:31 Matthew Crenshaw, notifications@github.com wrote:

My usage is very roughly like:

func (t Tail) Start() error { t.Lock() defer t.Unlock() t.lines := make(chan logline.LogLine) t.watcher := watcher.NewLogWatcher(pollDuration, !poll) if err != nil { defer close(t.lines) return err } var opts []func(tailer.Tailer) error if t.FromBeginning { opts = append(opts, tailer.OneShot) } t.tailer, err := tailer.New(t.lines, afero.NewOsFs(), t.watcher, opts...) if err != nil { defer t.watcher.Close() defer close(t.lines) return err } t.wg.Add(1) go t.processLines() return t.tailNew() } func (t Tail) Stop() error { t.Lock() defer t.Unlock()

err := t.tailer.Close() t.wg.Wait() return err } // never errors but that's ok for nowfunc (t Tail) tailFiles() error { for , filepath := t.Files { // snip globbing stuff i'm doing myself for file := range globbed { if , ok := t.files[file]; ok { continue } err := t.tailer.TailPath(file) if err != nil { t.log.Error(err) continue } t.log.Debug("start tailing %s", file) t.files[file] = true } } return nil } func (t Tail) processLines() { defer t.wg.Done()

for line = range t.lines { // do stuff with line.Line depending on line.Filename }

for _, file := range t.files { t.log.Debug("stop tailing %s", file) } }

Other methods add/remove files to be tailed on the fly

— You are receiving this because you commented.

Reply to this email directly, view it on GitHub https://github.com/google/mtail/issues/199#issuecomment-449478473, or mute the thread https://github.com/notifications/unsubscribe-auth/AC5b-64vEvkRT1F-VsQ97ox_1fREKCzyks5u7TcPgaJpZM4Zczvc .

[sgtsquiggs]

Hi @jaqx0r, I started work on separating out the lib here: https://github.com/sgtsquiggs/mtail

Other than removing unrelated mtail code: I removed the dependency on glog. It should still work with glog though :smile:

Please let me know if I'm missing anything license-wise. I need to add a README.

[jnovack]

As another use-case, I've been dying for a simple-evcorr replacement. I grew up on perl, so it's hard for me to bad-mouth it; but in 2019, using perl is not cute anymore.

I love the simple-evcorr idea (and I use it exclusively on my syslog servers for alerting), but it's very primitive and archaic, and mtail has a lot of the core pieces of a replacement.

[jaqx0r]

I think the tailer API needs some more work before I'm willing to move it out of internal:

• The logstream.New() functoin and its implementation methods need to return the lines channel, not take one as argument. This means goroutine shutdown is easier and more idiomatic.
• I like glog but I don't love it, so what's the current practice for injecting your own logger?
• Can I remove waker entirely and just use have blocking reads? The Stop() method is also unidiomatic and context cancellation should just trigger this, except that Stop means "stop after the next EOF" for filestreams, so should context cancellation try to read to an EOF or is that madness on a high-write-rate log? (Or in another way, how do we synchronise the code under test with the test to make sure the code behaves?)