search

openeuler-riscv / oerv-team

OERV 实习生工作中心
8 stars 38 forks source link

pcre 1 支持 riscv jit #397

Open Jingwiw opened 5 months ago

Jingwiw commented 5 months ago

说明

问题背景: pcre1 版本需要从 pcre2 版本 backport JIT 支持，目前还有很多软件依赖旧的 pcre1 版本

涉及软件包: pcre

涉及软件包地址: src-openeuler/pcre

所需技能:

其它:

TexasOct commented 4 months ago

今天开始进行初步的样例测试

TexasOct commented 4 months ago

目前测试还存在问题，主要如下

test-suite.log

=================================
   PCRE 8.45: ./test-suite.log
=================================

# TOTAL: 6
# PASS:  4
# SKIP:  0
# XFAIL: 0
# FAIL:  2
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: pcre_jit_test
===================

Running JIT regression tests
  target CPU of SLJIT compiler: RISC-V-64 64bit (little endian + aligned)
  in  8 bit mode with UTF-8  enabled and ucp enabled:
  in 16 bit mode with UTF-16 enabled and ucp enabled:
  in 32 bit mode with UTF-32 enabled and ucp enabled:
............................................................
............................................................
............................................................
............................................................
............................................................
............................................................
............................................................
..................................................
8 and 16 bit: Ovector[0] value differs(J8:18,I8:14,J16:18,I16:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb' 

8 and 32 bit: Ovector[0] value differs(J8:18,I8:14,J32:18,I32:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb' 

16 and 32 bit: Ovector[0] value differs(J16:18,I16:14,J32:18,I32:14): [471] '(?>\Ka\Ka)*aaaab' @ 'aaaaaaaa aaaaaaaaaabb' 
......................................
8 and 16 bit: Ovector[0] value differs(J8:12,I8:8,J16:12,I16:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc' 

8 and 32 bit: Ovector[0] value differs(J8:12,I8:8,J32:12,I32:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc' 

16 and 32 bit: Ovector[0] value differs(J16:12,I16:8,J32:12,I32:8): [510] '(?(DEFINE)(a\Kb))(?1)+ababc' @ 'abababxabababc' 

8 and 16 bit: Ovector[0] value differs(J8:14,I8:10,J16:14,I16:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc' 

8 and 32 bit: Ovector[0] value differs(J8:14,I8:10,J32:14,I32:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc' 

16 and 32 bit: Ovector[0] value differs(J16:14,I16:10,J32:14,I32:10): [511] '(a\Kb)(?1)+ababc' @ 'abababxababababc' 
.........................................................8 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
16 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
32 bit: Mark value mismatch: [569] '(?>a(*:aa))b|ac' @ 'ac'
............................................................
.......
Successful test ratio: 99% (4 failed)
FAIL pcre_jit_test (exit status: 1)

FAIL: RunTest
=============

PCRE C library tests using test data from ./testdata
PCRE version 8.45 2021-06-15

---- Testing 8-bit library ----

Test 1: Main functionality (Compatible with Perl >= 5.10)
  OK
  OK with study
  OK with JIT study
Test 2: API, errors, internals, and non-Perl stuff (not UTF-8)
  OK
  OK with study
--- ./testdata/testoutput2  2024-04-14 00:21:14.688089700 +0800
+++ testtry 2024-04-27 16:43:47.291383393 +0800
@@ -10638,7 +10638,7 @@

 /(?>a\Kb)z|(ab)/
     ab 
- 0: ab
+ 0: b
  1: ab

 /(?P<L1>(?P<L2>0|)|(?P>L2)(?P>L1))/
@@ -12501,7 +12501,7 @@
  1: a
 --->aab
     ^^      b
-Callout 2: last capture = -1
+Callout 2: last capture = 1
  0: <unset>
 --->aab
     ^ ^     b
@@ -12509,15 +12509,15 @@

 /(?:(?1)(?C1)x|ab(?C2))((a)){0}/                                                
     aab\C+ 
-Callout 1: last capture = -1
+Callout 1: last capture = 2
  0: <unset>
 --->aab
     ^^      x
-Callout 1: last capture = -1
+Callout 1: last capture = 2
  0: <unset>
 --->aab
      ^^     x
-Callout 2: last capture = -1
+Callout 2: last capture = 2
  0: <unset>
 --->aab
      ^ ^    )
@@ -12531,7 +12531,7 @@
  2: a
 --->aab
     ^^      )
-Callout 1: last capture = -1
+Callout 1: last capture = 2
  0: <unset>
 --->aab
     ^^      ((a)(?C2)){0}
FAIL RunTest (exit status: 1)

根据 pcre_jit_test 中对于结果的匹配开始位置都比正确位置多 4 的规律性猜测可能是在 matching环节上存在问题，现在正在对几个案例的生成的 JIT code 进行分析以验证猜测

Jingwiw commented 4 months ago

good job

TexasOct commented 4 months ago

测试案例已全部通过，完成包编译

[  275s] Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.YMvlf6
[  276s] + umask 022
[  276s] + cd /home/abuild/rpmbuild/BUILD
[  276s] + cd pcre-8.45
[  276s] + /usr/bin/rm -rf /home/abuild/rpmbuild/BUILDROOT/pcre-8.45-5.oe2403.riscv64
[  276s] + RPM_EC=0
[  276s] ++ jobs -p
[  276s] + exit 0
[  276s] Executing(rmbuild): /bin/sh -e /var/tmp/rpm-tmp.lOOGJq
[  276s] + umask 022
[  276s] + cd /home/abuild/rpmbuild/BUILD
[  276s] + rm -rf pcre-8.45 pcre-8.45.gemspec
[  276s] + RPM_EC=0
[  276s] ++ jobs -p
[  276s] + exit 0
[  276s] ... checking for files with abuild user/group
[  276s] 
[  276s] fedora-texas finished "build pcre.spec" at Mon Apr 29 16:59:08 UTC 2024.
[  276s] 

/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/SRPMS/pcre-8.45-5.oe2403.src.rpm

/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/noarch/pcre-help-8.45-5.oe2403.noarch.rpm
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/riscv64/pcre-devel-8.45-5.oe2403.riscv64.rpm
/var/tmp/build-root/mainline_riscv64-riscv64/home/abuild/rpmbuild/RPMS/riscv64/pcre-8.45-5.oe2403.riscv64.rpm
Jingwiw commented 4 months ago

lgtm

TexasOct commented 4 months ago

已提交可见pr https://gitee.com/src-openeuler/pcre/pulls/28 https://gitee.com/src-openeuler/openresty-pcre/pulls/13

TexasOct commented 3 months ago

一些简单的案列 测试文件 howto 来自此处 https://lh3lh3.users.sourceforge.net/reb.shtml 测试文件 input-text.txt 来自此处 https://github.com/mariomka/regex-benchmark

因为客观数据比较少所以找了一份 pcre 1 的 code 修改了下进行测试

核心代码逻辑如下，遍历整个文件并找出所有结果

int exec_offset = 0;
int matched_size = 0;

TIMER_START
do{
    rc = pcre_exec(re, extra, input, len, exec_offset, 0, ovector, ovecsize);

    if (rc > 0) {
        exec_offset = ovector[1];
        matched_size ++;
    }
} while (rc > 0);
TIMER_STOP

if (rc == 0) {
    fprintf(stderr, "capture size too small");
    exit(2);
}

if (rc < PCRE_ERROR_NOMATCH) {
    printf("error: %d", rc);
}

printf("matched_size: %d\n", matched_size);

printf("spend time: %.02lf ms elapsed.\n", elapsed);

结果如下，default 为 posix 实现， JIT 为本次移植实现

./bench '[\w\.+-]+@[\w\.-]+\.[\w\.-]+' howto
pcre default matched_size: 14077
spend time: 6529.35 ms elapsed.
pcre JIT matched_size: 14077
spend time: 225.15 ms elapsed.
./bench '[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?' howto
pcre default matched_size: 17810
spend time: 6003.11 ms elapsed.
pcre JIT matched_size: 17810
spend time: 231.41 ms elapsed.
./bench '(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])' howto
pcre default matched_size: 66
spend time: 539.08 ms elapsed.
pcre JIT matched_size: 66
spend time: 94.91 ms elapsed.

./bench '[\w\.+-]+@[\w\.-]+\.[\w\.-]+' input-text.txt
pcre default matched_size: 92
spend time: 838.50 ms elapsed.
pcre JIT matched_size: 92
spend time: 37.45 ms elapsed.
./bench '[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?' input-text.txt
pcre default matched_size: 5301
spend time: 792.29 ms elapsed.
pcre JIT matched_size: 5301
spend time: 40.48 ms elapsed.
./bench '(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])\\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9])' input-text.txt
pcre default matched_size: 0
spend time: 74.73 ms elapsed.
pcre JIT matched_size: 0
spend time: 15.91 ms elapsed.

这里也能给出关于利用 pcre2_match 进行测试的方案对比， 虽然代码不太一样，但是在迭代执行部分的逻辑也是do while形式，能将对照范围缩小到 pcre 和 pcre2 的实现区别上了

pcre2 jit 在input-text.txt中两个 pattern 的结果如下

'[\w]+:\/\/[^\/\s?#]+[^\s?#]+(?:\?[^\s#]*)?(?:#[^\s]*)?'
37.908805 ms  - 92
'[\w\.+-]+@[\w\.-]+\.[\w\.-]+'
38.379105 ms - 5301

目前粗看 pcre jit 和 pcre2 jit 相比，性能上没有什么差距，基本持平，且匹配数据上没有错误。

本次的移植性能提升还是非常明显的，虽然这是一个 legacy 软件了，不过如果说在一些场合使用的话性能还是非常的足够的