tsaikd / gogstash

Logstash like, written in golang
MIT License
647 stars 106 forks source link

the inputfile problem #30

Open R7R8 opened 7 years ago

R7R8 commented 7 years ago

There is a situation that the code below may fail.

    if fpath, err = filepath.EvalSymlinks(fpath); err != nil {
        err = errutil.New("Get symlinks failed: "+fpath, err)
        return
    }

For example do operations Write, Rename, Create to the fpath.

When the waitWatchEvent() have processed the Write event, it will enter the waitWatchEvent() again. But now, the fpath doesn't exit or the fpath is creating. Under this situation, the filepath.EvalSymlinksmay fail.

It is expected to do:

 waitWatchEvent have processed Write
        enter waitWatchEvent again
        waitWatchEvent processes Rename
        waitWatchEvent processed Create

but the reality is:

 waitWatchEvent have processed Write
        enter waitWatchEvent again
        filepath.EvalSymlinks(fpath)  fail

I fix this problem temporarily by adding below codes before filepath.EvalSymlinks(fpath) .

    for ;; {
        _, err := os.Stat(fpath)
        if err != nil && os.IsNotExist(err) {
            fmt.Println("IsNotExist")
            continue
        }
        break
    }
tsaikd commented 7 years ago

thx, please check 32f2b15 commit work or not

R7R8 commented 7 years ago

Thank you for your response.I will check.

R7R8 commented 7 years ago

It seems 32f2b15 commit can solve a part of problem, but sometimes it may still fail. The situation failed is that futil.IsExist(fpath) is true, but filepath.EvalSymlinks(fpath)can not return.

Maybe we should take restricted control to check if the fpath is ready.

tsaikd commented 7 years ago

How about 0253f9f ? I add one more checking before filepath.EvalSymlinks(fpath).

R7R8 commented 7 years ago

It seems that you put the new codes in the wrong place. filepath.EvalSymlinks(fpath) fails in the function waitWatchEvent, not in the function Start

R7R8 commented 7 years ago

Today I remove the code filepath.EvalSymlinks(fpath) in the function waitWatchEvent, then the application goes well. So I can ensure that the problem happens in filepath.EvalSymlinks(fpath).

tsaikd commented 7 years ago

OK, I use a wrap function for all filepath.EvalSymlinks(fpath) in 8360d5a .

R7R8 commented 7 years ago

I have two machines to test gogstash. Tomcat logs rotate one hour in one, one minute in the other. The former goes well, but the latter still has problem. The problem is still that filepath.EvalSymlinks(fpath) can not return

R7R8 commented 7 years ago

Sometimes filepath.EvalSymlinks(fpath) fails like this: Error: Get symlinks failed: ; lstat /XXX/XXX/XXX/localhost_access_log.log: no such file or directory