search

ulfalizer / Kconfiglib

A flexible Python 2/3 Kconfig implementation and library
ISC License
450 stars 161 forks source link

Failure in testsuit, any hint? #2

Closed fpemud closed 11 years ago

fpemud commented 11 years ago

I managed to finish the testsuit of kconfiglib, test 2 and test 5 failed. I think the next step should be doing the failed test manually and inspect the difference in ".confg" and "._config". is there any better idea?

test log:

fpemud-workstation linux-test # PYTHONPATH=scripts/kconfig python scripts/kconfig/kconfigtest.py
Loading Config instances for all architectures...
Resetting all architecture Config instances prior to next test...
Test if our allnoconfig implementation generates the same .config as 'make allnoconfig', for all architectures
  sh64          OK
  c6x           OK
  arm           OK
  frv           OK
  i386          OK
  x86_64        OK
  cris          OK
  m32r          OK
  ia64          OK
  m68k          OK
  mips          OK
  s390          OK
  tile          OK
  alpha         OK
  avr32         OK
  score         OK
  sparc32       OK
  sparc64       OK
  unicore32     OK
  blackfin      OK
  mn10300       OK
  parisc        OK
  hexagon       OK
  openrisc      OK
  xtensa        OK
  microblaze    OK
  powerpc       OK

Resetting all architecture Config instances prior to next test...
Test if kconfiglib generates the same configuration as 'conf' without a .config, for each architecture
  sh64          FAIL
  c6x           FAIL
  arm           FAIL
  frv           FAIL
  i386          FAIL
  x86_64        FAIL
  cris          FAIL
  m32r          FAIL
  ia64          FAIL
  m68k          FAIL
  mips          FAIL
  s390          FAIL
  tile          FAIL
  alpha         FAIL
  avr32         FAIL
  score         FAIL
  sparc32       FAIL
  sparc64       FAIL
  unicore32     FAIL
  blackfin      FAIL
  mn10300       FAIL
  parisc        FAIL
  hexagon       FAIL
  openrisc      FAIL
  xtensa        FAIL
  microblaze    FAIL
  powerpc       FAIL

Resetting all architecture Config instances prior to next test...
Test if our allyesconfig implementation generates the same .config as 'make allyesconfig', for all architectures
  sh64          OK
  c6x           OK
  arm           OK
  frv           OK
  i386          OK
  x86_64        OK
  cris          OK
  m32r          OK
  ia64          OK
  m68k          OK
  mips          OK
  s390          OK
  tile          OK
  alpha         OK
  avr32         OK
  score         OK
  sparc32       OK
  sparc64       OK
  unicore32     OK
  blackfin      OK
  mn10300       OK
  parisc        OK
  hexagon       OK
  openrisc      OK
  xtensa        OK
  microblaze    OK
  powerpc       OK

Resetting all architecture Config instances prior to next test...
Call all public methods on all symbols, menus, choices and comments (nearly all public methods: some are hard to test like this, but are exercised by other tests) for all architectures to make sure we never crash or hang
  For sh64...
  For c6x...
  For arm...
  For frv...
  For i386...
  For x86_64...
  For cris...
  For m32r...
  For ia64...
  For m68k...
  For mips...
  For s390...
  For tile...
  For alpha...
  For avr32...
  For score...
  For sparc32...
  For sparc64...
  For unicore32...
  For blackfin...
  For mn10300...
  For parisc...
  For hexagon...
  For openrisc...
  For xtensa...
  For microblaze...
  For powerpc...

Resetting all architecture Config instances prior to next test...
Test if kconfiglib generates the same .config as conf for each architecture/defconfig pair (this takes two hours on a Core i7@2.67 GHz system)
  sh64          with arch/sh/configs/ecovec24-romimage_defconfig                  FAIL
  sh64          with arch/sh/configs/se7721_defconfig                             FAIL
  sh64          with arch/sh/configs/ecovec24_defconfig                           FAIL
...
  x86_64        with arch/sh/configs/ecovec24-romimage_defconfig                  FAIL
  x86_64        with arch/sh/configs/se7721_defconfig                             FAIL
  x86_64        with arch/sh/configs/ecovec24_defconfig                           FAIL
...
ulfalizer commented 11 years ago

$ vimdiff .config ._config

was the approach I used. Post some diffs and I might be able to help; hopefully it's something trivial :).

ulfalizer commented 11 years ago

Oh, and Kconfiglib can print the locations of the corresponding symbol definitions (see the examples), which makes it easy to locate them.

fpemud commented 11 years ago

I found the problem, I forgot to add the kconfiglibtestconfig target into scripts/kconfig/Makefile. (makefile.patch failed so i modified scripts/kconfig/Makefile manually) Now test 2 passed.

ulfalizer commented 11 years ago

Nice! What about the other tests?

fpemud commented 11 years ago

I add a exit(1) in fail() function, so the test will abort at the first failed test and I can get .config and ._config And I found one failure, I have sent you an email that contains .config and ._config.

fpemud-workstation linux-test # uname -a
Linux fpemud-workstation 3.4.9-gentoo #4 SMP PREEMPT Sun Oct 21 16:56:09 CST 2012 x86_64 Intel(R) Core(TM) i5-2500K CPU @ 3.30GHz GenuineIntel GNU/Linux

Resetting all architecture Config instances prior to next test...
Test if kconfiglib generates the same .config as conf for each architecture/defconfig pair (this takes two hours on a Core i7@2.67 GHz system)
  sh64          with arch/sh/configs/ecovec24-romimage_defconfig                  OK
  sh64          with arch/sh/configs/se7721_defconfig                             OK
  sh64          with arch/sh/configs/ecovec24_defconfig                           OK
  sh64          with arch/sh/configs/landisk_defconfig                            OK
  ...
  sh64          with arch/blackfin/configs/CM-BF561_defconfig                     OK
  sh64          with arch/blackfin/configs/DNP5370_defconfig                      OK
  sh64          with arch/blackfin/configs/CM-BF533_defconfig                     OK
  sh64          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL

fpemud-workstation linux-test # diff .config ._config 
1,4d0
< #
< # Automatically generated file; DO NOT EDIT.
< # Linux/sh64 3.4.9-gentoo Kernel Configuration
< #
805c801
< CONFIG_USB_ETH=m
---
> CONFIG_USB_ETH=y
fpemud commented 11 years ago

I'm running a full test, I want to know the number of failures first. nice to meet you on line, Ulf. i will go sleep now, leave the computer working, see you tomorrow.

ulfalizer commented 11 years ago

If you get many failures they are likely to be related. The USB-related Kconfig files do some really hairy stuff, so I'm not surprised the problem is there.

I might look into it tomorrow.

ulfalizer commented 11 years ago

Oh, and if all the errors are for nonsensical architecture/defconfig combinations, things are probably stable enough that you can safely rely on stuff working. Most of those errors are really obscure.

fpemud commented 11 years ago

There're 77 failures, all of which is a minor percentage of arch/blackfin. i'm interested in finding out what happens and fix it, but i think it's safe enough to start to write my code base on it.

below is a change I made to kconfiglib.py:

[fpemud@fpemud-workstation ~]$ diff ~/workspace/Kconfiglib/kconfiglib.py /usr/lib64/python2.7/site-packages/kconfiglib.py
2252a2253
>         "----help---"    : T_HELP,  # Hack to handle CONFIG_SFC_MCDI_MON
[fpemud@fpemud-workstation ~]$ grep FAIL kconfiglib\ testrecord
  sh64          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  sh64          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  sh64          with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  sh64          with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  c6x           with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  c6x           with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  arm           with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  arm           with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  arm           with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  arm           with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  frv           with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  frv           with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  frv           with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  frv           with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  i386          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  i386          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  i386          with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  i386          with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  x86_64        with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  x86_64        with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  x86_64        with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  x86_64        with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  cris          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  cris          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  m32r          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  m32r          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  ia64          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  ia64          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  ia64          with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  ia64          with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  m68k          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  m68k          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  mips          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  mips          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  tile          with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  tile          with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  tile          with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  tile          with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  alpha         with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  alpha         with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  alpha         with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  alpha         with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  avr32         with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  avr32         with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  score         with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  score         with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  sparc32       with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  sparc32       with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  sparc64       with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  sparc64       with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  unicore32     with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  unicore32     with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  blackfin      with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  blackfin      with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  blackfin      with arch/blackfin/configs/BF527-EZKIT_defconfig                  FAIL
  blackfin      with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  blackfin      with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  mn10300       with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  mn10300       with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  parisc        with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  parisc        with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  hexagon       with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  hexagon       with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  openrisc      with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  openrisc      with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  xtensa        with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  xtensa        with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  xtensa        with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  xtensa        with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  microblaze    with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  microblaze    with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  microblaze    with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  microblaze    with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL
  powerpc       with arch/blackfin/configs/TCM-BF537_defconfig                    FAIL
  powerpc       with arch/blackfin/configs/CM-BF537U_defconfig                    FAIL
  powerpc       with arch/blackfin/configs/BF527-EZKIT-V2_defconfig               FAIL
  powerpc       with arch/blackfin/configs/BF548-EZKIT_defconfig                  FAIL