srinivascodes commented 6 years ago

Whenever I am running the command "./grok_exporter -config ./example/config.yml", I am getting following error.

SIGILL: illegal instruction PC=0x438d4fa m=8 sigcode=1 signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x4383910, 0xc420051600, 0x4485c19) /usr/local/Cellar/go/1.9.2/libexec/src/runtime/cgocall.go:132 +0xe4 fp=0xc4200515c0 sp=0xc420051580 pc=0x4003614 github.com/fstab/grok_exporter/exporter._Cfunc_onig_new(0xc4204d45e0, 0x6000000, 0x600076e, 0x0, 0x46ed318, 0x46ed1b8, 0xc4204d6cc0, 0x0) github.com/fstab/grok_exporter/exporter/_obj/_cgo_gotypes.go:294 +0x4d fp=0xc420051600 sp=0xc4200515c0 pc=0x437bbcd github.com/fstab/grok_exporter/exporter.(OnigurumaLib).Compile.func1(0xc4204d45e0, 0x6000000, 0x600076e, 0x0, 0x46ed318, 0x46ed1b8, 0xc4204d6cc0, 0x53) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x19f fp=0xc420051660 sp=0xc420051600 pc=0x437e1ef github.com/fstab/grok_exporter/exporter.(OnigurumaLib).Compile(0xc4204fa0a8, 0xc42017f800, 0x76e, 0x0, 0x0, 0x0) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x179 fp=0xc4200516f0 sp=0xc420051660 pc=0x437c4a9 github.com/fstab/grok_exporter/exporter.Compile(0xc420168000, 0x6d, 0xc42011a050, 0xc4204fa0a8, 0x0, 0x1, 0x4042fb4) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/grok.go:31 +0x9c fp=0xc4200517a8 sp=0xc4200516f0 pc=0x4372bfc main.createMetrics(0xc420158000, 0xc42011a050, 0xc4204fa0a8, 0x0, 0x0, 0x0, 0x0, 0xc420064970) /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:178 +0x15c fp=0xc420051b10 sp=0xc4200517a8 pc=0x43819ec main.main() /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:62 +0x14d fp=0xc420051f80 sp=0xc420051b10 pc=0x438000d runtime.main() /usr/local/Cellar/go/1.9.2/libexec/src/runtime/proc.go:195 +0x226 fp=0xc420051fe0 sp=0xc420051f80 pc=0x402da76 runtime.goexit() /usr/local/Cellar/go/1.9.2/libexec/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420051fe8 sp=0xc420051fe0 pc=0x4059551

rax 0x0 rbx 0x1 rcx 0x5 rdx 0x700000392be4 rdi 0x46ed318 rsi 0x1 rbp 0x700000392d90 rsp 0x700000392aa0 r8 0x0 r9 0x0 r10 0x467bfb0 r11 0x6ffffbd165fc r12 0x700000392bb0 r13 0x1 r14 0xc4204d6cc0 r15 0x4d00020 rip 0x438d4fa rflags 0x10297 cs 0x2b fs 0x0 gs 0x0

kindly look into this issue.

fstab commented 6 years ago

Thanks for the bug report. I must say I have never seen this error.

I have a Mac Book as my development machine, it never occurred there.
For each commit I run automated OS X tests on https://travis-ci.org/fstab/grok_exporter, the error never occurred there.
I even set up a new virtual machine with an original high sierra os, to make sure there are no unexpected dependencies. There was no error running grok_exporter there.

Can you provide some more info that could help me reproduce this error?

renweibo commented 6 years ago

similiar error

grok_exporter-0.2.4.darwin-amd64 renweibo$ ./grok_exporter -config ./example/config.yml SIGILL: illegal instruction PC=0x438e59a m=10 sigcode=1 signal arrived during cgo execution

goroutine 1 [syscall, locked to thread]: runtime.cgocall(0x4384990, 0xc420053600, 0x4486e6f) /Users/fabian/golang-1.9.3/go/src/runtime/cgocall.go:132 +0xe4 fp=0xc4200535c0 sp=0xc420053580 pc=0x40033d4 github.com/fstab/grok_exporter/exporter._Cfunc_onig_new(0xc420049200, 0x5800000, 0x580076e, 0x0, 0x46ee2d8, 0x46ee178, 0xc4202b4a40, 0x0) github.com/fstab/grok_exporter/exporter/_obj/_cgo_gotypes.go:294 +0x4d fp=0xc420053600 sp=0xc4200535c0 pc=0x437cc2d github.com/fstab/grok_exporter/exporter.(OnigurumaLib).Compile.func1(0xc420049200, 0x5800000, 0x580076e, 0x0, 0x46ee2d8, 0x46ee178, 0xc4202b4a40, 0x53) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x19f fp=0xc420053660 sp=0xc420053600 pc=0x437f24f github.com/fstab/grok_exporter/exporter.(OnigurumaLib).Compile(0xc42000e088, 0xc420209800, 0x76e, 0x0, 0x0, 0x0) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x179 fp=0xc4200536f0 sp=0xc420053660 pc=0x437d509 github.com/fstab/grok_exporter/exporter.Compile(0xc420064070, 0x6d, 0xc42000e0e0, 0xc42000e088, 0x0, 0x1, 0x4043224) /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/grok.go:31 +0x9c fp=0xc4200537a8 sp=0xc4200536f0 pc=0x4373c5c main.createMetrics(0xc420132000, 0xc42000e0e0, 0xc42000e088, 0x0, 0x0, 0x0, 0x0, 0xc420013e70) /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:178 +0x15f fp=0xc420053b10 sp=0xc4200537a8 pc=0x4382a4f main.main() /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:62 +0x14d fp=0xc420053f80 sp=0xc420053b10 pc=0x438106d runtime.main() /Users/fabian/golang-1.9.3/go/src/runtime/proc.go:195 +0x226 fp=0xc420053fe0 sp=0xc420053f80 pc=0x402dc56 runtime.goexit() /Users/fabian/golang-1.9.3/go/src/runtime/asm_amd64.s:2337 +0x1 fp=0xc420053fe8 sp=0xc420053fe0 pc=0x40597b1

rax 0x0 rbx 0x1 rcx 0x5 rdx 0x700003543be4 rdi 0x46ee2d8 rsi 0x1 rbp 0x700003543d90 rsp 0x700003543aa0 r8 0x0 r9 0x0 r10 0x467cfb0 r11 0x6ffffeec65fc r12 0x700003543bb0 r13 0x1 r14 0xc4202b4a40 r15 0x4c00020 rip 0x438e59a rflags 0x10297 cs 0x2b fs 0x0 gs 0x0

renweibo commented 6 years ago

grok_exporter-0.2.4.darwin-amd64 renweibo$ go version go version go1.10 darwin/amd64

renweibo commented 6 years ago

grok_exporter-0.2.4.darwin-amd64 renweibo$ cgo -V
cgo version go1.10

fstab commented 6 years ago

This is weird, I have never seen the error on my system. Could you provide the output of uname -a, and if you have otool installed also the output of otool -L ./grok_exporter?

renweibo commented 6 years ago

otool -L ./grok_exporter
./grok_exporter:
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1451.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.41.2)

Here is uname

uname -a
Darwin WeideMBP.lan 17.4.0 Darwin Kernel Version 17.4.0: Sun Dec 17 09:19:54 PST 2017; root:xnu-4570.41.2~1/RELEASE_X86_64 x86_64

renweibo commented 6 years ago

The problem seems weird to me. If you need more information, please give more instruction. I need this tool to work with some of the server.

fstab commented 6 years ago

This is very strange. You have exactly the same OS and library versions as the machine I am using for developing grok_exporter. Do you have any idea how I could reproduce this error? The error doesn't occur on my machine.

renweibo commented 6 years ago

Currently for me to reproduce it, it's very simple. Download the archive, unzip it, execute it then failed. I'm not familiar with golang. What's your idea about following questions?

It seems that these lines in grok_exporter.go cause problem

    metrics, err := createMetrics(cfg, patterns, libonig)  # line 62
    exitOnError(err)                                                              # line 63

Q1: Is the code I pointed is where the real error occurs? Q2: What's the real work in createMetrics? Q3: Can you provide more information about the error code itself?

fstab commented 6 years ago

You are right, from the stack trace we can see that createMetrics(...) is the call where the error occurs. However, the actual error seems to be a little further down the stack, in oniguruma.go, line 77, near r := C.onig_new(...).

A little background on what this does: Go's built-in regular expressions don't support backtracking (see re2 for more info). As grok patterns heavily rely on backtracking, grok_exporter use an external regular expression library called Oniguruma. Oniguruma is built into the Ruby programming language, which is used for Logstash, which is the origin of grok patterns.

Oniguruma is a C library. Go supports calling C libraries from go programs using cgo. This is the magic C in C.onig_new(...). The function onig_new is not a go function, it's a C function defined in oniguruma.h.

For some reason, the call to onig_new fails with the illegal instruction error. The error message sounds like there is an illegal assembly code involved, but it could be something completely different. I don't know why this happens, especially as I have never seen it on my machine.

fstab commented 6 years ago

You could try to build grok_exporter from source, as described in README.md. This will create a grok_exporter binary that is dynamically linked against Oniguruma on your system. Maybe this helps. (the binary release is statically linked, so that there is no need for having Oniguruma installed)

renweibo commented 6 years ago

@fstab Get it. I's clear to me now. I'll try to build it later on my system and ping here back for anything useful to dig the problem.

renweibo commented 6 years ago

@fstab Here is the update. Build from source works. Following is the steps

brew install fstab/oniguruma/oniguruma-5.9.6
go get github.com/fstab/grok_exporter
cd $GOPATH/src/github.com/fstab/grok_exporter
git submodule update --init --recursive
./release.sh darwin-amd64  # at first, it failed for the requirement of go 1.9.3 but I have go 1.10
# manually disable go version check in release.sh
./release.sh darwin-amd64
cd dist
unzip grok_exporter-0.2.5-SNAPSHOT.darwin-amd64.zip
cd grok_exporter-0.2.5-SNAPSHOT.darwin-amd64
./grok_exporter -config example/config.yml

Then I check http://localhost:9144/metrics. It show some output in browser.

renweibo commented 6 years ago

@fstab Here are some questions:

I don't found the detail step to build dynamically linked version grok-exporter. How to get it?
Now with this new result, what conclusion can we get?
Do you have any suggestion what's the next thing we can try to find the reason?

fstab commented 6 years ago

The release is built exactly with the same commands as you described above. I have no idea why it works when you run these commands on your machine and it doesn't work when I run the commands on my machine.

The dynamically linked version is created when you just run go install github.com/fstab/grok_exporter instead of using the release.sh script. You can run otool -L $GOPATH/bin/grok_exporter to verify that it's dynamically linked against oniguruma.

It would be interesting to see if the release you built on your machine crashes on my machine. Could you send me the ZIP file to fabian@fstab.de ?

renweibo commented 6 years ago

The archive is sent to you and let's see what happen on your machine.

What's your meaning of 'it doesn't work when I run the commands on my machine.'

renweibo commented 6 years ago

Dynamically linked version here. And http://localhost:9144/metrics show output.

$ go install github.com/fstab/grok_exporter
$ otool -L $GOPATH/bin/grok_exporter
/Users/renweibo/project/go/bin/grok_exporter:
    /usr/local/opt/oniguruma-5.9.6/lib/libonig.2.dylib (compatibility version 3.0.0, current version 3.0.0)
    /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1252.0.0)
    /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1451.0.0)
    /System/Library/Frameworks/Security.framework/Versions/A/Security (compatibility version 1.0.0, current version 58286.41.2)
$ cd dist/grok_exporter-0.2.5-SNAPSHOT.darwin-amd64
$ $GOPATH/bin/grok_exporter -config example/config.yml
Starting server on http://localhost:9144/metrics

fstab commented 6 years ago

Your release works fine on my machine.

With 'it doesn't work when I run the commands on my machine.' I meant that if I build a release on my machine using exactly the same commands that you described above, the release doesn't work on your machine.

I have no idea why this happens.

renweibo commented 6 years ago

Please ping me when you release next version, I will try it again. if the problem disappear, I suggest you close the issue and move on.

fstab commented 6 years ago

@renweibo released v0.2.5. If you are still looking into it, I would be interested if the problem is resolved. Updated to golang 1.10.2, maybe that helped.

renweibo commented 6 years ago

It still report error. I'll try go 1.10.2 later.

grok_exporter-0.2.5.darwin-amd64 renweibo$ cgo -V
cgo version go1.10
grok_exporter-0.2.5.darwin-amd64 renweibo$ go version
go version go1.10 darwin/amd64
grok_exporter-0.2.5.darwin-amd64 renweibo$ ./grok_exporter -config ./example/config.yml
SIGILL: illegal instruction
PC=0x437be9a m=8 sigcode=1

goroutine 0 [idle]:
runtime: unknown pc 0x437be9a
stack: frame={sp:0x700000e89ac0, fp:0x0} stack=[0x700000e0a290,0x700000e89e90)
0000700000e899c0:  0000000000000000  0000000000000000
0000700000e899d0:  0000000000000000  0000000000000000
0000700000e899e0:  0000000000000000  0000000000000000
0000700000e899f0:  0000000000000000  0000000000000000
...
0000700000e89bb0:  4000000000000000  0000700000e89ae0

goroutine 1 [syscall]:
runtime.cgocall(0x43720c0, 0xc420077668, 0x29)
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/cgocall.go:128 +0x64 fp=0xc420077628 sp=0xc4200775f0 pc=0x40034f4
github.com/fstab/grok_exporter/exporter._Cfunc_onig_new(0xc42056c3b0, 0x7000000, 0x700076e, 0x0, 0x46e5e58, 0x46e5d00, 0xc420164800, 0xc400000000)
    _cgo_gotypes.go:300 +0x4d fp=0xc420077668 sp=0xc420077628 pc=0x436ab3d
github.com/fstab/grok_exporter/exporter.(*OnigurumaLib).Compile.func1(0xc42056c3b0, 0x7000000, 0x700076e, 0x0, 0x46e5e58, 0x46e5d00, 0xc420164800, 0x43cdf80)
    /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x188 fp=0xc4200776b8 sp=0xc420077668 pc=0x436ce88
github.com/fstab/grok_exporter/exporter.(*OnigurumaLib).Compile(0xc4200b8578, 0xc4204fd800, 0x76e, 0x0, 0x0, 0x0)
    /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/oniguruma.go:77 +0x13c fp=0xc420077740 sp=0xc4200776b8 pc=0x436b3ac
github.com/fstab/grok_exporter/exporter.Compile(0xc42018a000, 0x6d, 0xc4200b80d0, 0xc4200b8578, 0x0, 0x1, 0x4874000)
    /Users/fabian/go/src/github.com/fstab/grok_exporter/exporter/grok.go:31 +0x9c fp=0xc4200777f8 sp=0xc420077740 pc=0x43624fc
main.createMetrics(0xc420182000, 0xc4200b80d0, 0xc4200b8578, 0x0, 0x0, 0x0, 0x0, 0x3cb164141fe3ad57)
    /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:178 +0x156 fp=0xc420077b00 sp=0xc4200777f8 pc=0x43703b6
main.main()
    /Users/fabian/go/src/github.com/fstab/grok_exporter/grok_exporter.go:62 +0x14d fp=0xc420077f88 sp=0xc420077b00 pc=0x436eadd
runtime.main()
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/proc.go:198 +0x212 fp=0xc420077fe0 sp=0xc420077f88 pc=0x402d302
runtime.goexit()
    /usr/local/Cellar/go/1.10.2/libexec/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420077fe8 sp=0xc420077fe0 pc=0x4056b11

rax    0x0
rbx    0x1
rcx    0x5
rdx    0x700000e89c04
rdi    0x46e5e58
rsi    0x1
rbp    0x700000e89db0
rsp    0x700000e89ac0
r8     0x0
r9     0x0
r10    0x4679fb0
r11    0x6ffffc80f61c
r12    0x700000e89bd0
r13    0x1
r14    0xc420164800
r15    0x4c00020
rip    0x437be9a
rflags 0x10297
cs     0x2b
fs     0x0
gs     0x0

renweibo commented 6 years ago

Try latest go v1.10.3, the same result

grok_exporter-0.2.5.darwin-amd64 renweibo$ go version
go version go1.10.3 darwin/amd64
grok_exporter-0.2.5.darwin-amd64 renweibo$ cgo -V
cgo version go1.10.3
grok_exporter-0.2.5.darwin-amd64 renweibo$ ./grok_exporter -config ./example/config.yml
SIGILL: illegal instruction
PC=0x437be9a m=8 sigcode=1

goroutine 0 [idle]:
runtime: unknown pc 0x437be9a
stack: frame={sp:0x70000e671ac0, fp:0x0} stack=[0x70000e5f2290,0x70000e671e90)
000070000e6719c0:  0000000000000000  0000000000000000
000070000e6719d0:  0000000000000000  0000000000000000
000070000e6719e0:  0000000000000000  0000000000000000
000070000e6719f0:  0000000000000000  0000000000000000
...

fstab commented 6 years ago

What if you build the release in your machine with ./release.sh darwin-amd64? Does it work?

renweibo commented 6 years ago

renweibo$ git clone https://github.com/fstab/grok_exporter.git
Cloning into 'grok_exporter'...
remote: Counting objects: 2534, done.
remote: Compressing objects: 100% (100/100), done.
remote: Total 2534 (delta 66), reused 68 (delta 29), pack-reused 2405
Receiving objects: 100% (2534/2534), 1.96 MiB | 337.00 KiB/s, done.
Resolving deltas: 100% (1282/1282), done.
Checking connectivity... done.
renweibo$ cd grok_exporter
renweibo$ ls
AUTHORS         LICENSE         example         logstash-patterns-core  tailer
BUILTIN.md      NOTICE          exporter        oniguruma       template
CONFIG.md       README.md       grok_exporter.go    release.sh      vendor
CONFIG_v1.md        config          integration-test.sh screenshot.png
renweibo$ go version
go version go1.10.3 darwin/amd64
renweibo$ ./release.sh darwin-amd64
Go version 1.9.3 required.
renweibo$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean

renweibo commented 6 years ago

After disable go version check, rerun release.sh

renweibo$ ./release.sh darwin-amd64
?       github.com/fstab/grok_exporter  [no test files]
ok      github.com/fstab/grok_exporter/config   0.007s
ok      github.com/fstab/grok_exporter/config/v1    0.008s
ok      github.com/fstab/grok_exporter/config/v2    0.011s
ok      github.com/fstab/grok_exporter/exporter 2.891s
ok      github.com/fstab/grok_exporter/tailer   6.716s
ok      github.com/fstab/grok_exporter/templates    0.011s
Building dist/grok_exporter-0.2.6-SNAPSHOT.darwin-amd64.zip
sed: oniguruma/oniguruma.go: No such file or directory

fstab commented 6 years ago

Oh, you tried between two of my commits while I was creating oniguruma/oniguruma.go. It should work now. I also removed the version check. Could you try again?

renweibo commented 6 years ago

It seems that oniguruma/oniguruma.go is missing.

renweibo$ git pull
Already up-to-date.
renweibo$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
renweibo$ ./release.sh darwin-amd64
?       github.com/fstab/grok_exporter  [no test files]
ok      github.com/fstab/grok_exporter/config   (cached)
ok      github.com/fstab/grok_exporter/config/v1    (cached)
ok      github.com/fstab/grok_exporter/config/v2    (cached)
ok      github.com/fstab/grok_exporter/exporter (cached)
ok      github.com/fstab/grok_exporter/tailer   (cached)
ok      github.com/fstab/grok_exporter/templates    (cached)
Building dist/grok_exporter-0.2.6-SNAPSHOT.darwin-amd64.zip
sed: oniguruma/oniguruma.go: No such file or directory

fstab commented 6 years ago

This is weird, the file is right here: https://github.com/fstab/grok_exporter/blob/master/oniguruma/oniguruma.go Could you check if you see the file on your disk after git pull?

renweibo commented 6 years ago

This time I can build it. And I found the reason for build failure. I git clone the repository in one directory not under GOPATH. So it failed for the release.sh assume it current path is under right path in GOPATH like ${GOPATH}/src/github.com/fstab/grok_exporter

For the file I build local, it can run without any problem.

fstab / grok_exporter

Unable to run darwin binary. #26

similiar error