search

intel / hyperscan

High-performance regular expression matching library
https://www.hyperscan.io
Other
4.77k stars 708 forks source link

pcre JIT support in chimera #143

Open trythreetimes opened 5 years ago

trythreetimes commented 5 years ago

Dose chimera support pcre's JIT, looks like it is closed by default in cmake. Thanks in advance.

xiangwang1 commented 5 years ago

We don't enable pcre JIT as it conflicts with database serialization and deserialization feature shared by Chimera and Hyperscan. I think this is a useful feature by our users and we try to avoid running pcre by making Hyperscan as a fast filter in Chimera.

trythreetimes commented 5 years ago

Thank you for your response. I tried to set it, but it led to memory leak.

xiangwang1 commented 5 years ago

Did you make any code modifications?

trythreetimes commented 5 years ago

Oh, yes, i modified a lot to add character iterator and \G for chimera, and i removed the stream mode because it is useless for chimera, but i don't think my modifications leads to memory leak, everything runs well except #139 by designed.

xiangwang1 commented 5 years ago

Actually there is no stream mode but only block mode in Chimera.

trythreetimes commented 5 years ago

I know. I mean i removed stream mode and vector mode from hyperscan because chimera only support block mode. Totally about 7500 lines including test case, this make the library 10% smaller for static linking. And ll_buf, the buffer for caseful long literal support , is very hard to modify when adding character iterator. So the best way is to let it go. Everything went smoothly until #139, if chimera team can solve this problem, it would be great.

ghost commented 1 year ago

@xiangwang1 I had the same problem with hyperscan 5.2.0 & pcre 8.43 (build from source and enable jit ) 。 My program behaves fine when using chimera, but when it exits valgrind reports some memory leaks。 Details： ==99601== 528 bytes in 2 blocks are indirectly lost in loss record 3 of 3 ==99601== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==99601== by 0x2A3EB1: fast_forward_first_n_chars (in ...................) ==99601== by 0x175D189: _pcre_jit_compile (in ...................) ==99601== by 0x175EFA1: pcre_study (in ...................) ==99601== by 0xE5B606: buildPcre (ch_compile.cpp:349) ==99601== by 0xE5B606: PatternData (ch_compile.cpp:285) ==99601== by 0xE5B606: make_unique<(anonymous namespace)::PatternData, char const&, unsigned int&, unsigned int&, unsigned int&, unsigned int&, long unsigned int&, long unsigned int&, const hs_platform_info&> (make_unique.hpp:74) ==99601== by 0xE5B606: ch::ch_compile_multi_int(char const const, unsigned int const, unsigned int const, unsigned int, unsigned int, unsigned long, unsigned long, hs_platform_info const*, ch_database**) (ch_compile.cpp:499) ==99601== by 0xE5C93C: ch_compile (ch_compile.cpp:707)

ghost commented 1 year ago

@xiangwang1 I had the same problem with hyperscan 5.2.0 & pcre 8.43 (build from source and enable jit ) 。 My program behaves fine when using chimera, but when it exits valgrind reports some memory leaks。 Details： ==99601== 528 bytes in 2 blocks are indirectly lost in loss record 3 of 3 ==99601== at 0x4C2FB0F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==99601== by 0x2A3EB1: fast_forward_first_n_chars (in ...................) ==99601== by 0x175D189: _pcre_jit_compile (in ...................) ==99601== by 0x175EFA1: pcre_study (in ...................) ==99601== by 0xE5B606: buildPcre (ch_compile.cpp:349) ==99601== by 0xE5B606: PatternData (ch_compile.cpp:285) ==99601== by 0xE5B606: make_unique<(anonymous namespace)::PatternData, char const&, unsigned int&, unsigned int&, unsigned int&, unsigned int&, long unsigned int&, long unsigned int&, const hs_platform_info&> (make_unique.hpp:74) ==99601== by 0xE5B606: ch::ch_compile_multi_int(char const const, unsigned int const, unsigned int const, unsigned int, unsigned int, unsigned long, unsigned long, hs_platform_info const*, ch_database**) (ch_compile.cpp:499) ==99601== by 0xE5C93C: ch_compile (ch_compile.cpp:707)

I found a similar issue in another project, and its method solved my memory leak problem. FYI: https://github.com/SpiderLabs/ModSecurity/issues/610