flier
commented
8 years ago very strange, only impact OSX, seems ok on Linux
Closed flier closed 8 years ago
flier
commented
8 years ago very strange, only impact OSX, seems ok on Linux
mdb256
commented
8 years ago Yes, we've also just reproduced it on OSX, but cannot on Linux.
We don't really know anything about Go or the integration, but our own unit tests pass on OSX, so we suspect it is something to do with the Go layer. Between 4.1 and 4.2 the only change in the API was adding a new error for HS_SCRATCH_IN_USE.
jviiret
commented
8 years ago @flier: Have you had any luck tracking down what was causing this? The pointer 0xb01dfacedebac1e from the Go runtime apparently indicates a memory corruption issue on OS X -- perhaps a Hyperscan allocated pointer is getting freed by Go's GC when it shouldn't be?
flier
commented
8 years ago I have upgrade to hyperscan v4.3.0 with golang 1.7, it still crash when I reuse/scan a reseted stream.
If I replace the hs_reset_stream with a pair of hs_close_stream/hs_open_stream, it could works.
Convey("Then reset the stream", func() {
//So(hsResetStream(stream, 0, s, h.Handle, nil), ShouldBeNil)
So(hsCloseStream(stream, s, h.Handle, nil), ShouldBeNil)
stream, err := hsOpenStream(db, 0)
So(stream, ShouldNotBeNil)
So(err, ShouldBeNil)
Convey("When scan the second part, should not be matched", func() {
So(hsScanStream(stream, []byte("stdef"), 0, s, h.Handle, nil), ShouldBeNil)
So(h.matched, ShouldBeNil)
})
})When I use hs_reset_and_copy_stream in similar scene, it also could works very well.
Convey("Then copy the stream", func() {
stream2, err := hsCopyStream(stream)
So(stream2, ShouldNotBeNil)
So(err, ShouldBeNil)
Convey("When copied stream2 scan the second part, should be matched", func() {
So(hsScanStream(stream2, []byte("stdef"), 0, s, h.Handle, nil), ShouldBeNil)
So(h.matched, ShouldResemble, []matchEvent{{0, 0, 7, 0}})
Convey("When copied stream2 scan the second part again, should not be matched", func() {
h.matched = nil
So(hsScanStream(stream2, []byte("stdef"), 0, s, h.Handle, nil), ShouldBeNil)
So(h.matched, ShouldBeNil)
Convey("When copy and reset stream2", func() {
So(hsResetAndCopyStream(stream2, stream, s, h.Handle, nil), ShouldBeNil)
Convey("When copied and reset stream2 scan the second part again, should be matched", func() {
h.matched = nil
So(hsScanStream(stream2, []byte("stdef"), 0, s, h.Handle, nil), ShouldBeNil)
So(h.matched, ShouldResemble, []matchEvent{{0, 0, 7, 0}})
})
})
})
})
So(hsCloseStream(stream2, s, h.Handle, nil), ShouldBeNil)
})It seems hs_reset_stream will write some pointer and corrupt memory, because when I call hs_reset_stream twice, it also crash
Convey("Then reset the stream", func() {
So(hsResetStream(stream, 0, s, h.Handle, nil), ShouldBeNil)
So(hsResetStream(stream, 0, s, h.Handle, nil), ShouldBeNil)
Convey("When scan the second part, should not be matched", func() {
So(hsScanStream(stream, []byte("stdef"), 0, s, h.Handle, nil), ShouldBeNil)
So(h.matched, ShouldBeNil)
})
})func hsResetStream(stream hsStream, flags ScanFlag, scratch hsScratch, onEvent hsMatchEventHandler, context interface{}) error {
ctxt := &hsMatchEventContext{onEvent, context}
ret := C.hs_reset_stream_cgo(C.uintptr_t(uintptr(unsafe.Pointer(stream))), C.uint(flags),
C.uintptr_t(uintptr(unsafe.Pointer(scratch))), C.uintptr_t(uintptr(unsafe.Pointer(ctxt))))
if ret != C.HS_SUCCESS {
return HsError(ret)
}
return nil
}static inline
hs_error_t hs_reset_stream_cgo(uintptr_t id, unsigned int flags, uintptr_t scratch, uintptr_t context) {
return hs_reset_stream((hs_stream_t *) id, flags, (hs_scratch_t *) scratch, hs_event_callback, (void *) context);
}Thank you for reporting this - this turns out to be an issue with Clang builds (as found in Xcode) and the changes we had made in hs_reset_stream() - it did not affect gcc builds.
We have pushed v4.3.1 with a fix for this.
flier
commented
8 years ago Verified, it could works now, thanks
I'm working on a golang binding for Hyperscan.
After I upgrade from 4.1.0 to 4.2.0, the unit tests crashed when scan a reseted stream.
I have confirmed the same code could work with
Hyperscan 4.1.0andgo version go1.6.2 darwin/amd64Are there any changes in those API behaviors?