Open Jingwiw opened 5 months ago
今天开始进行初步的样例测试
目前测试还存在问题,主要如下
test-suite.log
=================================
PCRE 8.45: ./test-suite.log
=================================
# TOTAL: 6
# PASS: 4
# SKIP: 0
# XFAIL: 0
# FAIL: 2
# XPASS: 0
# ERROR: 0
.. contents:: :depth: 2
FAIL: pcre_jit_test
===================
Running JIT regression tests
target CPU of SLJIT compiler: RISC-V-64 64bit (little endian + aligned)
in 8 bit mode with UTF-8 enabled and ucp enabled:
in 16 bit mode with UTF-16 enabled and ucp enabled:
in 32 bit mode with UTF-32 enabled and ucp enabled:
............................................................
............................................................
............................................................
............................................................
............................................................
............................................................
............................................................
..................................................
8 and 16 bit: Ovector[0] value differs(J8:18,I8:14,J16:18,I16:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb'
8 and 32 bit: Ovector[0] value differs(J8:18,I8:14,J32:18,I32:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb'
16 and 32 bit: Ovector[0] value differs(J16:18,I16:14,J32:18,I32:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb'
......................................
8 and 16 bit: Ovector[0] value differs(J8:12,I8:8,J16:12,I16:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc'
8 and 32 bit: Ovector[0] value differs(J8:12,I8:8,J32:12,I32:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc'
16 and 32 bit: Ovector[0] value differs(J16:12,I16:8,J32:12,I32:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc'
8 and 16 bit: Ovector[0] value differs(J8:14,I8:10,J16:14,I16:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc'
8 and 32 bit: Ovector[0] value differs(J8:14,I8:10,J32:14,I32:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc'
16 and 32 bit: Ovector[0] value differs(J16:14,I16:10,J32:14,I32:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc'
.........................................................8 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
16 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
32 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
............................................................
.......
Successful test ratio: 99% (4 failed)
FAIL pcre_jit_test (exit status: 1)
FAIL: RunTest
=============
PCRE C library tests using test data from ./testdata
PCRE version 8.45 2021-06-15
---- Testing 8-bit library ----
Test 1: Main functionality (Compatible with Perl >= 5.10)
OK
OK with study
OK with JIT study
Test 2: API, errors, internals, and non-Perl stuff (not UTF-8)
OK
OK with study
--- ./testdata/testoutput2 2024-04-14 00:21:14.688089700 +0800
+++ testtry 2024-04-27 16:43:47.291383393 +0800
@@ -10638,7 +10638,7 @@
/(?>a\Kb)z|(ab)/
ab
- 0: ab
+ 0: b
1: ab
/(?P<L1>(?P<L2>0|)|(?P>L2)(?P>L1))/
@@ -12501,7 +12501,7 @@
1: a
--->aab
^^ b
-Callout 2: last capture = -1
+Callout 2: last capture = 1
0: <unset>
--->aab
^ ^ b
@@ -12509,15 +12509,15 @@
/(?:(?1)(?C1)x|ab(?C2))((a)){0}/
aab\C+
-Callout 1: last capture = -1
+Callout 1: last capture = 2
0: <unset>
--->aab
^^ x
-Callout 1: last capture = -1
+Callout 1: last capture = 2
0: <unset>
--->aab
^^ x
-Callout 2: last capture = -1
+Callout 2: last capture = 2
0: <unset>
--->aab
^ ^ )
@@ -12531,7 +12531,7 @@
2: a
--->aab
^^ )
-Callout 1: last capture = -1
+Callout 1: last capture = 2
0: <unset>
--->aab
^^ ((a)(?C2)){0}
FAIL RunTest (exit status: 1)
根据 pcre_jit_test 中对于结果的匹配开始位置都比正确位置多 4 的规律性猜测可能是在 matching环节上存在问题,现在正在对几个案例的生成的 JIT code 进行分析以验证猜测
good job
测试案例已全部通过,完成包编译
[ 275s] Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.YMvlf6
[ 276s] + umask 022
[ 276s] + cd /home/abuild/rpmbuild/BUILD
[ 276s] + cd pcre-8.45
[ 276s] + /usr/bin/rm -rf /home/abuild/rpmbuild/BUILDROOT/pcre-8.45-5.oe2403.riscv64
[ 276s] + RPM_EC=0
[ 276s] ++ jobs -p
[ 276s] + exit 0
[ 276s] Executing(rmbuild): /bin/sh -e /var/tmp/rpm-tmp.lOOGJq
[ 276s] + umask 022
[ 276s] + cd /home/abuild/rpmbuild/BUILD
[ 276s] + rm -rf pcre-8.45 pcre-8.45.gemspec
[ 276s] + RPM_EC=0
[ 276s] ++ jobs -p
[ 276s] + exit 0
[ 276s] ... checking for files with abuild user/group
[ 276s]
[ 276s] fedora-texas finished "build pcre.spec" at Mon Apr 29 16:59:08 UTC 2024.
[ 276s]
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/SRPMS/pcre-8.45-5.oe2403.src.rpm
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/noarch/pcre-help-8.45-5.oe2403.noarch.rpm
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/riscv64/pcre-devel-8.45-5.oe2403.riscv64.rpm
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/riscv64/pcre-8.45-5.oe2403.riscv64.rpm
lgtm
一些简单的案列
测试文件 howto
来自此处 https://lh3lh3.users.sourceforge.net/reb.shtml
测试文件 input-text.txt
来自此处 https://github.com/mariomka/regex-benchmark
因为客观数据比较少所以找了一份 pcre 1 的 code 修改了下进行测试
核心代码逻辑如下,遍历整个文件并找出所有结果
int exec_offset = 0;
int matched_size = 0;
TIMER_START
do{
rc = pcre_exec(re, extra, input, len, exec_offset, 0, ovector, ovecsize);
if (rc > 0) {
exec_offset = ovector[1];
matched_size ++;
}
} while (rc > 0);
TIMER_STOP
if (rc == 0) {
fprintf(stderr, "capture size too small");
exit(2);
}
if (rc < PCRE_ERROR_NOMATCH) {
printf("error: %d", rc);
}
printf("matched_size: %d\n", matched_size);
printf("spend time: %.02lf ms elapsed.\n", elapsed);
结果如下,default 为 posix 实现, JIT 为本次移植实现
./bench '[\w\.+-]+@[\w\.-]+\.[\w\.-]+' howto
pcre default matched_size: 14077
spend time: 6529.35 ms elapsed.
pcre JIT matched_size: 14077
spend time: 225.15 ms elapsed.
./bench '[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?' howto
pcre default matched_size: 17810
spend time: 6003.11 ms elapsed.
pcre JIT matched_size: 17810
spend time: 231.41 ms elapsed.
./bench '(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])' howto
pcre default matched_size: 66
spend time: 539.08 ms elapsed.
pcre JIT matched_size: 66
spend time: 94.91 ms elapsed.
./bench '[\w\.+-]+@[\w\.-]+\.[\w\.-]+' input-text.txt
pcre default matched_size: 92
spend time: 838.50 ms elapsed.
pcre JIT matched_size: 92
spend time: 37.45 ms elapsed.
./bench '[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?' input-text.txt
pcre default matched_size: 5301
spend time: 792.29 ms elapsed.
pcre JIT matched_size: 5301
spend time: 40.48 ms elapsed.
./bench '(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])' input-text.txt
pcre default matched_size: 0
spend time: 74.73 ms elapsed.
pcre JIT matched_size: 0
spend time: 15.91 ms elapsed.
这里也能给出关于利用 pcre2_match 进行测试的方案对比, 虽然代码不太一样,但是在迭代执行部分的逻辑也是do while形式,能将对照范围缩小到 pcre 和 pcre2 的实现区别上了
pcre2 jit 在input-text.txt
中两个 pattern 的结果如下
'[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?'
37.908805 ms - 92
'[\w\.+-]+@[\w\.-]+\.[\w\.-]+'
38.379105 ms - 5301
目前粗看 pcre jit 和 pcre2 jit 相比,性能上没有什么差距,基本持平,且匹配数据上没有错误。
本次的移植性能提升还是非常明显的,虽然这是一个 legacy 软件了,不过如果说在一些场合使用的话性能还是非常的足够的
说明
问题背景: pcre1 版本需要从 pcre2 版本 backport JIT 支持,目前还有很多软件依赖旧的 pcre1 版本
涉及软件包: pcre
涉及软件包地址: src-openeuler/pcre
所需技能:
其它: