search

AmpereComputing / ampere-lts-kernel---DEPRECATED

Linux 5.4 and 5.10 Longterm kernel (LTS) with Ampere patches
21 stars 17 forks source link

test: Revise LTP skip file list #34

Open adamliyi opened 3 years ago

adamliyi commented 3 years ago

"788 failures out of 3885 tests" on Debian 11. v.s ours: LTP TESTS: total 2186 skip 588 fail 28.

Need to find out missing cases.

Also refer to: https://testfarm.openanolis.org/#/report

adamliyi commented 3 years ago

Attached the LTP result on Debian11 by other team, there are 3883 test cases. Details in this file:

test-run-31789-results.zip

e.g, we skipped aio test. We need to edit the skipped file, and enable more test cases.

cat test-run-31789-results.csv | grep -i aio | wc -l
470
adamliyi commented 3 years ago

With the updated skip file in: https://github.com/AmpereComputing/ampere-lts-kernel/commit/2a5ea219ff5c9387a2b5aae23666e4e94c39a157

LTP TESTS: total 2187 skip 545 fail 59

Still big gap with 3883 test case.

adamliyi commented 3 years ago

PR: https://github.com/AmpereComputing/ampere-lts-kernel/pull/42 enabled all test cases. Current result: Total Tests 3875, Skipped Tests 983, Failures: 156

result_0526.txt

Compare with 'test-run-31789-results.zip', it has 788 failures out of 3885. (fail: result=1, pass: result=2). It looks we skipped too many test cases?

bobolmw commented 3 years ago

I have enabled some kernel config, and installed software package dependencies those days. Current result: Total Tests 3877, Skipped Tests 652, Failures: 138

The rest fixing progress can be find in here: https://docs.google.com/spreadsheets/d/1kNXjuFW_OxsNXR1kxfbeIOhodwRRf8S8xdy0YSKD2l8/edit?usp=sharing

bobolmw commented 3 years ago

Current result: Total Tests 3554, Skipped 223, Failures 106

output.tar.gz

adamliyi commented 3 years ago

@bobolmw 1） could you please add comments in ltp_cmdfile, why "ltp-aiodio.part1,ltp-aiodio.part3,,ltp-aio-stress.part1,ltp-aio-stress.part2" are skipped (not run)? 2) ‘Skipped 223’ - are these 223 case defined in skipfile-lkft.yaml 3) 'Failures 106' - can we add comments on why it is failed?

adamliyi commented 3 years ago

Tested using 5.10.27 kernel, using CentOS-8.3, result: LTP TESTS: total 3561 skip 830 fail 398

adamliyi commented 3 years ago

Latest result with analysis of failing or skip case: https://docs.google.com/spreadsheets/d/1kNXjuFW_OxsNXR1kxfbeIOhodwRRf8S8xdy0YSKD2l8/edit?usp=sharing

Ampere-lts ltp testcase revise_0610.zip

bobolmw commented 3 years ago

1） could you please add comments in ltp_cmdfile, why "ltp-aiodio.part1,ltp-aiodio.part3,,ltp-aio-stress.part1,ltp-aio-stress.part2" are skipped (not run)?

ltp_cmdfile is not supported to add comments. I explain it here: The aio and dio tests I have removed request some file and dirs to be present before tests run. But the scripts in ltp have referenced tmpdir and current pid runtime information. And dependencies files can't be created unless to modify ltp source code. Reference some aiodio test code below:

AD001 time aiocp -b 1k -n 1 -f DIRECT $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2 AD002 time aiocp -b 1k -n 1 -f SYNC $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2 AD003 time aiocp -b 1k -n 1 -f DIRECT -f SYNC $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2

adamliyi commented 3 years ago

1） could you please add comments in ltp_cmdfile, why "ltp-aiodio.part1,ltp-aiodio.part3,,ltp-aio-stress.part1,ltp-aio-stress.part2" are skipped (not run)?

ltp_cmdfile is not supported to add comments. I explain it here: The aio and dio tests I have removed request some file and dirs to be present before tests run. But the scripts in ltp have referenced tmpdir and current pid runtime information. And dependencies files can't be created unless to modify ltp source code. Reference some aiodio test code below:

AD001 time aiocp -b 1k -n 1 -f DIRECT $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2 AD002 time aiocp -b 1k -n 1 -f SYNC $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2 AD003 time aiocp -b 1k -n 1 -f DIRECT -f SYNC $TMPDIR/aiodio.$$/junkfile $TMPDIR/aiodio.$$/ff2

Hi Bob,

In the test result: https://github.com/AmpereComputing/ampere-lts-kernel/issues/34#issuecomment-840929785 "[Linux Test Project]: suite: ltp-aiodio.part1 test: AD301 aiocp -b 128k -n 32 -f CREAT -f DIRECT $TMPDIR/aiodio.$$/fff $TMPDIR/aiodio.$$/junkdir/fff 2 " This case test result is PASS. This is the ltp test run by other team. I don't think they modified LTP source code.

What is the error you see when creating ' dependencies files'?

bobolmw commented 3 years ago

What is the error you see when creating ' dependencies files'?

The error log of aiodio tests: /opt/ltp/ltptmp/ltp-L1cvyIuIYN/aiodio.1371720/fff: No such file or directory /opt/ltp/ltptmp/ltp-L1cvyIuIYN/aiodio.1371722/ff1: No such file or directory /opt/ltp/ltptmp/ltp-L1cvyIuIYN/aiodio.1371724/ff2: No such file or directory /opt/ltp/ltptmp/ltp-L1cvyIuIYN/aiodio.1371726/ff3: No such file or directory

As the log showed, the dirs of ltp-L1cvyIuIYN and aiodio.1371726 are generated randomly. I can't create those file before ltp run

bobolmw commented 3 years ago

There is a testscripts/ltp-aiodio.sh script under ltp source code, which run aiodio test independent. I have tried this script, it's not work either. And in this script, the dependency files is created by:

dd if=$file1 of=$TMP/aiodio/junkfile bs=8192 conv=block,sync dd if=$file1 of=$TMP/aiodio/fff bs=4096 conv=block,sync dd if=$file1 of=$TMP/aiodio/ff1 bs=2048 conv=block,sync dd if=$file1 of=$TMP/aiodio/ff2 bs=1024 conv=block,sync dd if=$file1 of=$TMP/aiodio/ff3 bs=512 conv=block,sync

It's obvious ltp-aiodio.part1 testscript have a different file path. I don't known how other team to fix this problem without modifing ltp source. Are they use the latest ltp master branch @adamliyi ?

adamliyi commented 3 years ago

@bobolmw , ok I think it is a ltp test script issue.

There is patch related with aiodio test here: https://patchwork.ozlabs.org/project/ltp/patch/20210601155427.996321-1-zlang@redhat.com/

Could you try the patch?

Also there is already an issue reported here: https://github.com/linux-test-project/ltp/issues/261

adamliyi commented 3 years ago

@bobolmw , more comments:

1) Two more test set are skipped: io_cd and io_floppy, are they require CD and Floppy disk? Could you edit wiki: https://github.com/AmpereComputing/ampere-lts-kernel/wiki/LTP-test-on-LTS-kernel , add the reason to skip?

2) In 'https://docs.google.com/spreadsheets/d/1kNXjuFW_OxsNXR1kxfbeIOhodwRRf8S8xdy0YSKD2l8/edit?usp=sharing' , you mentioned some test cases are for x86, could you add in 'skipfile-lkft.yaml' to skip those test cases?

3) Please add latest result to wiki: https://github.com/AmpereComputing/ampere-lts-kernel/wiki/LTP-test-on-LTS-kernel#current-result

Thanks.

adamliyi commented 2 years ago

LTP test failed case analysis: Ampere-lts ltp testcase revise (3).xlsx