search

kernelci / kernelci-backend

Upstream Linux Kernel Validation Project API
http://api.kernelci.org
GNU Lesser General Public License v2.1
40 stars 27 forks source link

T23580 Fix test report e-mail indentation #256

Closed mgalka closed 4 years ago

mgalka commented 4 years ago

This commit fixes test case indentation when there is more than one test case to be displayed in the regression block. It also fixes indentation when there are test cases with no log lines present.

Signed-off-by: Michal Galka michal.galka@collabora.com

gctucker commented 4 years ago

Looks good; please copy a sample email report in a comment to confirm it's working as expected on staging.

mgalka commented 4 years ago

Example report from today. Formatting and indentation looks good.

kernelci/staging.kernelci.org baseline: 142 runs, 2 regressions (staging-20200922.1)

Regressions Summary
-------------------

platform         | arch | lab          | compiler | defconfig          | results
-----------------+------+--------------+----------+--------------------+--------
beaglebone-black | arm  | lab-baylibre | gcc-8    | multi_v7_defconfig | 2/4    

  Details:  https://staging.kernelci.org/test/job/kernelci/branch/staging.kernelci.org/kernel/staging-20200922.1/plan/baseline/

  Test:     baseline
  Tree:     kernelci
  Branch:   staging.kernelci.org
  Describe: staging-20200922.1
  URL:      https://github.com/kernelci/linux.git
  SHA:      e6454602339d0cfd0c7fce9ab7a3171e5759318c 

Test Regressions
---------------- 

platform         | arch | lab          | compiler | defconfig          | results
-----------------+------+--------------+----------+--------------------+--------
beaglebone-black | arm  | lab-baylibre | gcc-8    | multi_v7_defconfig | 2/4    

  Details:     https://staging.kernelci.org/test/plan/id/5f69b501f27a73334f6f3ea9

  Results:     2 PASS, 2 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
  Plain log:   https://storage.staging.kernelci.org/kernelci/staging.kernelci.org/staging-20200922.1/arm/multi_v7_defconfig/gcc-8/lab-baylibre/baseline-beaglebone-black.txt
  HTML log:    https://storage.staging.kernelci.org/kernelci/staging.kernelci.org/staging-20200922.1/arm/multi_v7_defconfig/gcc-8/lab-baylibre/baseline-beaglebone-black.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 

  * baseline.dmesg.alert: https://staging.kernelci.org/test/case/id/5f69b501f27a73334f6f3eae
    failing since 7 days (last pass: staging-20200914.0, first fail: staging-20200914.1)ffff
    4 lines
    2020-09-22 08:25:29.719000  kern  :alert : pgd = f38f372a
    2020-09-22 08:25:29.719000  kern  :alert : [0000001c] *pgd=8364b831, *pte=00000000, *ppte=00000000  

  * baseline.dmesg.emerg: https://staging.kernelci.org/test/case/id/5f69b501f27a73334f6f3eaf
    failing since 7 days (last pass: staging-20200914.0, first fail: staging-20200914.1)ffff
    29 lines
    2020-09-22 08:25:29.764000  kern  :emerg : Process dropbear (pid: 128, stack limit = 0x7e34231c)
    2020-09-22 08:25:29.765000  kern  :emerg : Stack: (0xc3643e30 to 0xc3644000)
    2020-09-22 08:25:29.766000  kern  :emerg : 3e20:                                     00000010 c04cbd68 c19f82bc 00000002
    2020-09-22 08:25:29.766000  kern  :emerg : 3e40: c137e618 c136d104 d8786018 d8786000 00000003 0000044f d8786018 0000003d
    2020-09-22 08:25:29.767000  kern  :emerg : 3e60: 00000000 00000000 c3643f78 c0307f4c d8786018 0000003d 00000000 d8786000
    2020-09-22 08:25:29.807000  kern  :emerg : 3e80: d8786018 c04ccbe4 00000000 00001000 beb14858 d8786028 00000000 c100edbc
    2020-09-22 08:25:29.808000  kern  :emerg : 3ea0: c1877074 c138fee4 0ccb654e db23d980 c04cc868 c363aa80 beb14858 00001000
    2020-09-22 08:25:29.809000  kern  :emerg : 3ec0: c3643f78 00000001 c0525580 c0525628 c3642000 00001000 00000000 beb14858
    2020-09-22 08:25:29.809000  kern  :emerg : 3ee0: c363aa80 c3643f78 00000001 c04a385c 00000008 00000000 00000051 c04b9134
    2020-09-22 08:25:29.810000  kern  :emerg : 3f00: ffffeee9 ffffffff 00000001 00000000 3b9ab8e9 00000000 00000001 00000000
    ... (18 line(s) more)
gctucker commented 4 years ago

There's a problem with indentation getting removed in this PR.

This is what email reports currently look like in production:

  * baseline.dmesg.alert: https://kernelci.org/test/case/id/5f69557e580734ce2cbf9dc9
      failing since 33 days (last pass: v5.8-107-gb72b3ea38c81, first fail: v5.9-rc1-4-g1f08d51cd57f)
      60 lines

    2020-09-22 01:38:00.525000  kern  :alert : BUG: Bad page state in process swapper/0  pfn:9c802

Having an indentation with the regression information and the measurements (60 lines), and a blank line before the log lines makes it a lot clearer to read.

The issue to fix here is that a 2nd regression is not aligned with the first one, as in this example:

  * baseline.dmesg.alert: https://staging.kernelci.org/test/case/id/5f607d023c44fc8731dfd2db
      failing since 0 day (last pass: staging-20200903.3, first fail: staging-20200915.0)
      4 lines

    2020-09-15 08:36:12.609000  kern  :alert : pgd = (ptrval)
    2020-09-15 08:36:12.609000  kern  :alert : [0000001c] *pgd=23a9b831, *pte=00000000, *ppte=00000000
     * baseline.dmesg.emerg: https://staging.kernelci.org/test/case/id/5f607d023c44fc8731dfd2dc
      failing since 0 day (last pass: staging-20200903.3, first fail: staging-20200915.0)
      29 lines

    2020-09-15 08:36:12.700000  kern  :emerg : Process dropbear (pid: 138, stack limit = 0x(ptrval))

It should look like this instead:

  * baseline.dmesg.alert: https://staging.kernelci.org/test/case/id/5f607d023c44fc8731dfd2db
      failing since 0 day (last pass: staging-20200903.3, first fail: staging-20200915.0)
      4 lines

    2020-09-15 08:36:12.609000  kern  :alert : pgd = (ptrval)
    2020-09-15 08:36:12.609000  kern  :alert : [0000001c] *pgd=23a9b831, *pte=00000000, *ppte=00000000

  * baseline.dmesg.emerg: https://staging.kernelci.org/test/case/id/5f607d023c44fc8731dfd2dc
      failing since 0 day (last pass: staging-20200903.3, first fail: staging-20200915.0)
      29 lines

    2020-09-15 08:36:12.700000  kern  :emerg : Process dropbear (pid: 138, stack limit = 0x(ptrval))
gctucker commented 4 years ago

Not quite there yet:

  HTML log:    https://storage.staging.kernelci.org/kernelci/staging.kernelci.org/staging-20200927.1/arm/multi_v7_defconfig/gcc-8/lab-collabora/baseline-rk3288-veyron-jaq.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/armel/baseline/rootfs.cpio.gz 

  * baseline.dmesg.alert: https://staging.kernelci.org/test/case/id/5f704ffbe79e4f2adb042eb4
    failing since 1 day (last pass: staging-20200926.0, first fail: staging-20200926.1)

    4 lines

    2020-09-27 08:40:21.401000  kern  :alert : pgd = 6e094fb0
    2020-09-27 08:40:21.402000  kern  :alert : [003bffc4] *pgd=00000000   

  * baseline.dmesg.emerg: https://staging.kernelci.org/test/case/id/5f704ffbe79e4f2adb042eb5
    failing since 1 day (last pass: staging-20200926.0, first fail: staging-20200926.1)

    42 lines

    2020-09-27 08:40:21.427
mgalka commented 4 years ago

One more formatting adjustment. Locally generated output.


Regressions Summary
-------------------

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:  https://kernelci.org/test/job/mgalka/branch/mgalka/kernel/kernelci-local-snapshot-032-2-gdf08b708456e/plan/baseline/

  Test:     baseline
  Tree:     mgalka
  Branch:   mgalka
  Describe: kernelci-local-snapshot-032-2-gdf08b708456e
  URL:      https://gitlab.collabora.com/mgalka/linux.git
  SHA:      df08b708456e4f73ba96690fa0a47722ce0a23fd 

Test Regressions
---------------- 

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:     https://kernelci.org/test/plan/id/5ea07e5a0dac971e09020c4b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig
  Compiler:    gcc-8 (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0)
  Plain log:   https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.txt
  HTML log:    https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2019.02-9-g25091c539382/x86/baseline/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/5ec4fc9a66e806ef963c70cb
    new failure (last pass: kernelci-local-snapshot-032)
    22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

  * baseline.login: https://kernelci.org/test/case/id/5ec4fc9a66e806ef963c70cb
    new failure (last pass: kernelci-local-snapshot-032)
    22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:     https://kernelci.org/test/plan/id/5f5b5c68f6ca1f70f42c1ae6

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig
  Compiler:    gcc-8 (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0)
  Plain log:   https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.txt
  HTML log:    https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/x86/baseline/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/5f5b5c68f6ca1f70f42c1ae7
    new failure (last pass: kernelci-local-snapshot-032)
    22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

  * baseline.login: https://kernelci.org/test/case/id/5f5b5c68f6ca1f70f42c1ae7
    new failure (last pass: kernelci-local-snapshot-032)
    22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)
mgalka commented 4 years ago

...and some indentation fixes


Regressions Summary
-------------------

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:  https://kernelci.org/test/job/mgalka/branch/mgalka/kernel/kernelci-local-snapshot-032-2-gdf08b708456e/plan/baseline/

  Test:     baseline
  Tree:     mgalka
  Branch:   mgalka
  Describe: kernelci-local-snapshot-032-2-gdf08b708456e
  URL:      https://gitlab.collabora.com/mgalka/linux.git
  SHA:      df08b708456e4f73ba96690fa0a47722ce0a23fd 

Test Regressions
---------------- 

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:     https://kernelci.org/test/plan/id/5ea07e5a0dac971e09020c4b

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig
  Compiler:    gcc-8 (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0)
  Plain log:   https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.txt
  HTML log:    https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2019.02-9-g25091c539382/x86/baseline/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/5ec4fc9a66e806ef963c70cb
        new failure (last pass: kernelci-local-snapshot-032)
        22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

  * baseline.login: https://kernelci.org/test/case/id/5ec4fc9a66e806ef963c70cb
        new failure (last pass: kernelci-local-snapshot-032)
        22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

platform    | arch   | lab               | compiler | defconfig        | results
------------+--------+-------------------+----------+------------------+--------
qemu_x86_64 | x86_64 | mgalka-lava-local | gcc-8    | x86_64_defconfig | 0/1    

  Details:     https://kernelci.org/test/plan/id/5f5b5c68f6ca1f70f42c1ae6

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: x86_64_defconfig
  Compiler:    gcc-8 (gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0)
  Plain log:   https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.txt
  HTML log:    https://storage.kernelci.org//mgalka/mgalka/kernelci-local-snapshot-032-2-gdf08b708456e/x86_64/x86_64_defconfig/gcc-8/mgalka-lava-local/baseline-qemu_x86_64.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/buildroot/kci-2020.05/x86/baseline/rootfs.cpio.gz 

  * baseline.login: https://kernelci.org/test/case/id/5f5b5c68f6ca1f70f42c1ae7
        new failure (last pass: kernelci-local-snapshot-032)
        22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)  

  * baseline.login: https://kernelci.org/test/case/id/5f5b5c68f6ca1f70f42c1ae7
        new failure (last pass: kernelci-local-snapshot-032)
        22 lines

    2020-05-20 09:44:50.009000  + cat dmesg.emerg dmesg.alert dmesg.crit dmesg.err dmesg.warn
    2020-05-20 09:44:50.020000  kern  :warn  : tsc: Unable to calibrate against PIT
    2020-05-20 09:44:50.040000  kern  :warn  : acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge.
    2020-05-20 09:44:50.054000  kern  :warn  : platform regulatory.0: Direct firmware load for regulatory.db failed with error -2
    2020-05-20 09:44:50.064000  kern  :warn  : urandom_read: 2 callbacks suppressed
    2020-05-20 09:44:50.068000  + set +x
    2020-05-20 09:44:50.077000  <LAVA_SIGNAL_ENDRUN 0_dmesg 490_1.3.3.1>
    2020-05-20 09:44:50.088000  <LAVA_TEST_RUNNER EXIT>
    2020-05-20 09:44:50.191000  / # #
    2020-05-20 09:44:50.298000  mkdir /lava-490 
    ... (10 line(s) more)
mgalka commented 4 years ago

E-mail report generated on staging.kernelci.org:

kernelci/staging-next sleep: 6 runs, 15 regressions (staging-next-20201001.0)

Regressions Summary
-------------------

platform          | arch  | lab           | compiler | defconfig          | results
------------------+-------+---------------+----------+--------------------+--------
rk3288-veyron-jaq | arm   | lab-collabora | gcc-8    | multi_v7_defconfig | 2/16   
rk3399-gru-kevin  | arm64 | lab-collabora | clang-11 | defconfig          | 0/1    

  Details:  https://staging.kernelci.org/test/job/kernelci/branch/staging-next/kernel/staging-next-20201001.0/plan/sleep/

  Test:     sleep
  Tree:     kernelci
  Branch:   staging-next
  Describe: staging-next-20201001.0
  URL:      https://github.com/kernelci/linux.git
  SHA:      8ad3e3a814e17b91d0f499b1edd7483765b10212 

Test Regressions
---------------- 

platform          | arch  | lab           | compiler | defconfig          | results
------------------+-------+---------------+----------+--------------------+--------
rk3288-veyron-jaq | arm   | lab-collabora | gcc-8    | multi_v7_defconfig | 2/16   

  Details:     https://staging.kernelci.org/test/plan/id/5f76095cbaa5a87901cf179e

  Results:     2 PASS, 14 FAIL, 0 SKIP
  Full config: multi_v7_defconfig
  Compiler:    gcc-8 (arm-linux-gnueabihf-gcc (Debian 8.3.0-2) 8.3.0)
  Plain log:   https://storage.staging.kernelci.org/kernelci/staging-next/staging-next-20201001.0/arm/multi_v7_defconfig/gcc-8/lab-collabora/sleep-rk3288-veyron-jaq.txt
  HTML log:    https://storage.staging.kernelci.org/kernelci/staging-next/staging-next-20201001.0/arm/multi_v7_defconfig/gcc-8/lab-collabora/sleep-rk3288-veyron-jaq.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/debian/buster/20200925.2/armhf/rootfs.cpio.gz 

  * sleep.rtcwake-mem-1: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a1
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:47:29.686000  ake: assuming RTC uses UTC ...
    2020-10-01 16:47:29.687000  rtcwake: wakeup from \"mem\" using<6>[   13.216626] Filesystems sync: 0.000 seconds
    2020-10-01 16:47:29.692000   rtc0 at Thu Oct  1 16:47:36 2020
    2020-10-01 16:47:29.699000  <6>[   13.228205] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:47:29.699000  <6>[   13.239265] OOM killer disabled.
    2020-10-01 16:47:49.718000  <6>[   13.243180] Freezing remaining freezable tasks ... 
    2020-10-01 16:47:49.730000  <3>[   33.256143] Freezing of tasks failed after 20.007 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:47:49.730000  <6>[   33.266525] Showing busy workqueues and worker pools:
    2020-10-01 16:47:49.742000  <6>[   33.272501] workqueue events: flags=0x0
    2020-10-01 16:47:49.743000  <6>[   33.277111]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

  * sleep.rtcwake-mem-2: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a2
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:47:49.959000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:47:56 2020
    2020-10-01 16:47:49.975000  <6>[   33.510022] PM: suspend entry (deep)
    2020-10-01 16:47:49.975000  <6>[   33.514410] Filesystems sync: 0.000 seconds
    2020-10-01 16:47:49.989000  <6>[   33.519771] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:47:49.989000  <6>[   33.529043] OOM killer disabled.
    2020-10-01 16:48:10.006000  <6>[   33.532966] Freezing remaining freezable tasks ... 
    2020-10-01 16:48:10.016000  <3>[   53.544669] Freezing of tasks failed after 20.005 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:48:10.016000  <6>[   53.555052] Showing busy workqueues and worker pools:
    2020-10-01 16:48:10.023000  <6>[   53.563303] workqueue events: flags=0x0
    2020-10-01 16:48:10.033000  <6>[   53.570175]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

  * sleep.rtcwake-mem-3: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a3
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:48:10.321000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:48:17 2020
    2020-10-01 16:48:10.332000  <6>[   53.873496] PM: suspend entry (deep)
    2020-10-01 16:48:10.340000  <6>[   53.880030] Filesystems sync: 0.000 seconds
    2020-10-01 16:48:10.352000  <6>[   53.887601] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:48:10.357000  <6>[   53.899057] OOM killer disabled.
    2020-10-01 16:48:30.383000  <6>[   53.905139] Freezing remaining freezable tasks ... 
    2020-10-01 16:48:30.397000  <3>[   73.924606] Freezing of tasks failed after 20.010 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:48:30.397000  <6>[   73.937141] Showing busy workqueues and worker pools:
    2020-10-01 16:48:30.407000  <6>[   73.945315] workqueue events: flags=0x0
    2020-10-01 16:48:30.415000  <6>[   73.952127]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

  * sleep.rtcwake-mem-4: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a4
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:48:30.702000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:48:37 2020
    2020-10-01 16:48:30.713000  <6>[   74.254856] PM: suspend entry (deep)
    2020-10-01 16:48:30.721000  <6>[   74.261404] Filesystems sync: 0.000 seconds
    2020-10-01 16:48:30.732000  <6>[   74.268977] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:48:30.738000  <6>[   74.280391] OOM killer disabled.
    2020-10-01 16:48:50.758000  <6>[   74.286478] Freezing remaining freezable tasks ... 
    2020-10-01 16:48:50.772000  <3>[   94.307128] Freezing of tasks failed after 20.012 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:48:50.780000  <6>[   94.319674] Showing busy workqueues and worker pools:
    2020-10-01 16:48:50.786000  <6>[   94.327848] workqueue events: flags=0x0
    2020-10-01 16:48:50.797000  <6>[   94.334666]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

  * sleep.rtcwake-mem-5: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a5
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:48:51.088000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:48:58 2020
    2020-10-01 16:48:51.098000  <6>[   94.640270] PM: suspend entry (deep)
    2020-10-01 16:48:51.105000  <6>[   94.646834] Filesystems sync: 0.000 seconds
    2020-10-01 16:48:51.573000  <3>[   95.111707] dwmmc_rockchip ff0c0000.mmc: Busy; trying anyway
    2020-10-01 16:48:52.085000  <3>[   95.620545] mmc_host mmc1: Timeout sending command (cmd 0x202000 arg 0x0 status 0x80202000)
    2020-10-01 16:48:52.095000  <6>[   95.632430] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:48:52.101000  <6>[   95.643843] OOM killer disabled.
    2020-10-01 16:49:12.114000  <6>[   95.649961] Freezing remaining freezable tasks ... 
    2020-10-01 16:49:12.127000  <3>[  115.662851] Freezing of tasks failed after 20.004 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:49:12.135000  <6>[  115.675464] Showing busy workqueues and worker pools: 
    ... (32 line(s) more)  

  * sleep.rtcwake-mem-6: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a6
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:49:12.457000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:49:19 2020
    2020-10-01 16:49:12.468000  <6>[  116.010943] PM: suspend entry (deep)
    2020-10-01 16:49:12.476000  <6>[  116.017490] Filesystems sync: 0.000 seconds
    2020-10-01 16:49:12.488000  <6>[  116.025058] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:49:12.494000  <6>[  116.036507] OOM killer disabled.
    2020-10-01 16:49:32.511000  <6>[  116.042645] Freezing remaining freezable tasks ... 
    2020-10-01 16:49:32.524000  <3>[  136.060555] Freezing of tasks failed after 20.009 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:49:32.532000  <6>[  136.073216] Showing busy workqueues and worker pools:
    2020-10-01 16:49:32.537000  <6>[  136.081506] workqueue events: flags=0x0
    2020-10-01 16:49:32.549000  <6>[  136.088414]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (30 line(s) more)  

  * sleep.rtcwake-mem-7: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a7
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:49:32.853000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:49:39 2020
    2020-10-01 16:49:32.864000  <6>[  136.407957] PM: suspend entry (deep)
    2020-10-01 16:49:32.872000  <6>[  136.414489] Filesystems sync: 0.000 seconds
    2020-10-01 16:49:32.884000  <6>[  136.422010] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:49:32.890000  <6>[  136.433400] OOM killer disabled.
    2020-10-01 16:49:52.911000  <6>[  136.439505] Freezing remaining freezable tasks ... 
    2020-10-01 16:49:52.924000  <3>[  156.460831] Freezing of tasks failed after 20.012 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:49:52.931000  <6>[  156.473405] Showing busy workqueues and worker pools:
    2020-10-01 16:49:52.939000  <6>[  156.481630] workqueue events: flags=0x0
    2020-10-01 16:49:52.948000  <6>[  156.488492]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (30 line(s) more)  

  * sleep.rtcwake-mem-8: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a8
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:49:53.253000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:50:00 2020
    2020-10-01 16:49:53.264000  <6>[  156.807871] PM: suspend entry (deep)
    2020-10-01 16:49:53.271000  <6>[  156.814389] Filesystems sync: 0.000 seconds
    2020-10-01 16:49:53.283000  <6>[  156.821926] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:49:53.289000  <6>[  156.833301] OOM killer disabled.
    2020-10-01 16:50:13.303000  <6>[  156.839386] Freezing remaining freezable tasks ... 
    2020-10-01 16:50:13.317000  <3>[  176.854638] Freezing of tasks failed after 20.006 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:50:13.324000  <6>[  176.867215] Showing busy workqueues and worker pools:
    2020-10-01 16:50:13.331000  <6>[  176.875432] workqueue events: flags=0x0
    2020-10-01 16:50:13.342000  <6>[  176.882297]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (30 line(s) more)  

  * sleep.rtcwake-mem-9: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17a9
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:50:13.646000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:50:20 2020
    2020-10-01 16:50:13.657000  <6>[  177.202349] PM: suspend entry (deep)
    2020-10-01 16:50:13.666000  <6>[  177.208907] Filesystems sync: 0.000 seconds
    2020-10-01 16:50:13.677000  <6>[  177.216448] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:50:13.683000  <6>[  177.227858] OOM killer disabled.
    2020-10-01 16:50:33.700000  <6>[  177.233950] Freezing remaining freezable tasks ... 
    2020-10-01 16:50:33.712000  <3>[  197.250534] Freezing of tasks failed after 20.008 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:50:33.720000  <6>[  197.263123] Showing busy workqueues and worker pools:
    2020-10-01 16:50:33.727000  <6>[  197.271355] workqueue events: flags=0x0
    2020-10-01 16:50:33.737000  <6>[  197.278221]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (30 line(s) more)  

  * sleep.rtcwake-mem-10: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17aa
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:50:34.046000  rtcwake: wakeup from \"mem\" using rtc0 at Thu Oct  1 16:50:40 2020
    2020-10-01 16:50:34.057000  <6>[  197.602773] PM: suspend entry (deep)
    2020-10-01 16:50:34.064000  <6>[  197.609332] Filesystems sync: 0.000 seconds
    2020-10-01 16:50:34.530000  <3>[  198.072141] dwmmc_rockchip ff0c0000.mmc: Busy; trying anyway
    2020-10-01 16:50:35.042000  <3>[  198.580980] mmc_host mmc1: Timeout sending command (cmd 0x202000 arg 0x0 status 0x80202000)
    2020-10-01 16:50:35.053000  <6>[  198.592878] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:50:35.058000  <6>[  198.604308] OOM killer disabled.
    2020-10-01 16:50:55.080000  <6>[  198.610454] Freezing remaining freezable tasks ... 
    2020-10-01 16:50:55.092000  <3>[  218.631531] Freezing of tasks failed after 20.012 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:50:55.100000  <6>[  218.644184] Showing busy workqueues and worker pools: 
    ... (32 line(s) more)  

  * sleep.rtcwake-freeze-1: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17ab
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:50:55.424000  rtcwake: wakeup from \"freeze\" using rtc0 at Thu Oct  1 16:51:02 2020
    2020-10-01 16:50:55.438000  <6>[  218.982687] PM: suspend entry (s2idle)
    2020-10-01 16:50:55.444000  <6>[  218.989432] Filesystems sync: 0.000 seconds
    2020-10-01 16:50:55.457000  <6>[  218.996973] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:50:55.463000  <6>[  219.008427] OOM killer disabled.
    2020-10-01 16:51:15.479000  <6>[  219.014568] Freezing remaining freezable tasks ... 
    2020-10-01 16:51:15.491000  <3>[  239.030980] Freezing of tasks failed after 20.008 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:51:15.500000  <6>[  239.043636] Showing busy workqueues and worker pools:
    2020-10-01 16:51:15.507000  <6>[  239.051926] workqueue events: flags=0x0
    2020-10-01 16:51:15.516000  <6>[  239.058838]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

  * sleep.rtcwake-freeze-2: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17ac
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:51:15.806000  rtcwake: wakeup from \"freeze\" using rtc0 at Thu Oct  1 16:51:22 2020
    2020-10-01 16:51:15.817000  <6>[  239.363760] PM: suspend entry (s2idle)
    2020-10-01 16:51:15.825000  <6>[  239.370521] Filesystems sync: 0.000 seconds
    2020-10-01 16:51:15.837000  <6>[  239.378115] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:51:15.843000  <6>[  239.389546] OOM killer disabled.
    2020-10-01 16:51:35.854000  <6>[  239.395636] Freezing remaining freezable tasks ... 
    2020-10-01 16:51:35.867000  <3>[  259.406918] Freezing of tasks failed after 20.003 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:51:35.874000  <6>[  259.419480] Showing busy workqueues and worker pools:
    2020-10-01 16:51:35.881000  <6>[  259.427666] workqueue events: flags=0x0
    2020-10-01 16:51:35.892000  <6>[  259.434496]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (32 line(s) more)  

  * sleep.rtcwake-freeze-3: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17ad
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:51:36.215000  rtcwake: wakeup from \"freeze\" using rtc0 at Thu Oct  1 16:51:43 2020
    2020-10-01 16:51:36.226000  <6>[  259.771889] PM: suspend entry (s2idle)
    2020-10-01 16:51:36.232000  <6>[  259.778605] Filesystems sync: 0.000 seconds
    2020-10-01 16:51:36.245000  <6>[  259.786136] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:51:36.250000  <6>[  259.797565] OOM killer disabled.
    2020-10-01 16:51:56.265000  <6>[  259.803694] Freezing remaining freezable tasks ... 
    2020-10-01 16:51:56.277000  <3>[  279.818329] Freezing of tasks failed after 20.006 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:51:56.286000  <6>[  279.830970] Showing busy workqueues and worker pools:
    2020-10-01 16:51:56.291000  <6>[  279.839245] workqueue events: flags=0x0
    2020-10-01 16:51:56.303000  <6>[  279.846117]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (30 line(s) more)  

  * sleep.rtcwake-freeze-4: https://staging.kernelci.org/test/case/id/5f76095cbaa5a87901cf17ae
        new failure (last pass: staging-next-20200930.2)

    2020-10-01 16:51:56.608000  rtcwake: wakeup from \"freeze\" using rtc0 at Thu Oct  1 16:52:03 2020
    2020-10-01 16:51:56.619000  <6>[  280.165267] PM: suspend entry (s2idle)
    2020-10-01 16:51:56.625000  <6>[  280.172035] Filesystems sync: 0.000 seconds
    2020-10-01 16:51:56.638000  <6>[  280.179568] Freezing user space processes ... (elapsed 0.001 seconds) done.
    2020-10-01 16:51:56.643000  <6>[  280.190970] OOM killer disabled.
    2020-10-01 16:52:16.654000  <6>[  280.197046] Freezing remaining freezable tasks ... 
    2020-10-01 16:52:16.666000  <3>[  300.207823] Freezing of tasks failed after 20.001 seconds (0 tasks refusing to freeze, wq_busy=1):
    2020-10-01 16:52:16.674000  <6>[  300.220415] Showing busy workqueues and worker pools:
    2020-10-01 16:52:16.680000  <6>[  300.228635] workqueue events: flags=0x0
    2020-10-01 16:52:16.692000  <6>[  300.235505]   pwq 2: cpus=1 node=0 flags=0x0 nice=0 active=17/256 refcnt=18 
    ... (28 line(s) more)  

platform          | arch  | lab           | compiler | defconfig          | results
------------------+-------+---------------+----------+--------------------+--------
rk3399-gru-kevin  | arm64 | lab-collabora | clang-11 | defconfig          | 0/1    

  Details:     https://staging.kernelci.org/test/plan/id/5f7608c03da390609dcf1802

  Results:     0 PASS, 1 FAIL, 0 SKIP
  Full config: defconfig
  Compiler:    clang-11 (Debian clang version 11.0.0-++20200827022622+522d80ab553-1~exp1~20200827123237.67)
  Plain log:   https://storage.staging.kernelci.org/kernelci/staging-next/staging-next-20201001.0/arm64/defconfig/clang-11/lab-collabora/sleep-rk3399-gru-kevin.txt
  HTML log:    https://storage.staging.kernelci.org/kernelci/staging-next/staging-next-20201001.0/arm64/defconfig/clang-11/lab-collabora/sleep-rk3399-gru-kevin.html
  Rootfs:      http://storage.kernelci.org/images/rootfs/debian/buster/20200925.2/arm64/rootfs.cpio.gz 

  * sleep.login: https://staging.kernelci.org/test/case/id/5f7608c03da390609dcf1803
        failing since 0 day (last pass: staging-next-20200930.1, first fail: staging-next-20200930.2)