search

crash-utility / crash

Linux kernel crash utility
https://crash-utility.github.io
836 stars 274 forks source link

crash 7.1.9/Beagle Bone Black/Yocto 2.3 out of memory? #16

Open RobertBerger opened 7 years ago

RobertBerger commented 7 years ago

Hi,

I built crash 7.1.9 for the Yocto project and make some experiments on a BeagleBone Black and it looks like it runs out of memory now and it's unusable.

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi"...

crash: vmlinux: Cannot allocate memory
      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Mon May 29 12:03:54 2017
      UPTIME: 00:05:17
LOAD AVERAGE: 1.30, 1.17, 0.56
       TASKS: 94
    NODENAME: multi-v7-ml
     RELEASE: 4.9.36-custom-student-custom-ml-debug
     VERSION: #1 SMP Fri Jul 21 17:49:10 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
         PID: 631
     COMMAND: "crash"
        TASK: d9ed1000  [THREAD_INFO: d9c66000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
crash> p jiffies
crash: fork system call failed: Cannot allocate memorycrash: cannot open pipe
crash>

Am I doing something wrong?

Is this a known issue?

Last login: Mon May 29 11:59:59 2017
root@multi-v7-ml:~# cat /proc/meminfo
MemTotal:         486028 kB
MemFree:            3564 kB
MemAvailable:     150560 kB
Buffers:             184 kB
Cached:           145820 kB
SwapCached:            0 kB
Active:           334496 kB
Inactive:         120340 kB
Active(anon):     308900 kB
Inactive(anon):       92 kB
Active(file):      25596 kB
Inactive(file):   120248 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         486028 kB
LowFree:            3564 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        308832 kB
Mapped:            16036 kB
Shmem:               164 kB
Slab:              21068 kB
SReclaimable:      11992 kB
SUnreclaim:         9076 kB
KernelStack:         760 kB
PageTables:         1300 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      243012 kB
Committed_AS:     331640 kB
VmallocTotal:     516096 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:          65536 kB
CmaFree:               0 kB
root@multi-v7-ml:~# 
root@multi-v7-ml:/tmp# cp /proc/config.gz .
root@multi-v7-ml:/tmp# gunzip config.gz 
root@multi-v7-ml:/tmp# cat config | grep DEVMEM
CONFIG_DEVMEM=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
root@multi-v7-ml:/tmp#
crash-utility commented 7 years ago

----- Original Message -----

Hi,

I built crash 7.1.9 for the Yocto project and make some experiments on a BeagleBone Black and it looks like it runs out of memory now and it's unusable.

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi"...

crash: vmlinux: Cannot allocate memory
      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Mon May 29 12:03:54 2017
      UPTIME: 00:05:17
LOAD AVERAGE: 1.30, 1.17, 0.56
       TASKS: 94
    NODENAME: multi-v7-ml
     RELEASE: 4.9.36-custom-student-custom-ml-debug
     VERSION: #1 SMP Fri Jul 21 17:49:10 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
         PID: 631
     COMMAND: "crash"
        TASK: d9ed1000  [THREAD_INFO: d9c66000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)
crash> p jiffies
crash: fork system call failed: Cannot allocate memorycrash: cannot open pipe
crash>

Am I doing something wrong?

Is this a known issue?

No, you're not doing anything wrong, but with only 512MB of physical memory, and 0 swap, I'm impressed that it even got as far as the "crash>" prompt.

Dave

Last login: Mon May 29 11:59:59 2017
root@multi-v7-ml:~# cat /proc/meminfo
MemTotal:         486028 kB
MemFree:            3564 kB
MemAvailable:     150560 kB
Buffers:             184 kB
Cached:           145820 kB
SwapCached:            0 kB
Active:           334496 kB
Inactive:         120340 kB
Active(anon):     308900 kB
Inactive(anon):       92 kB
Active(file):      25596 kB
Inactive(file):   120248 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         486028 kB
LowFree:            3564 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:        308832 kB
Mapped:            16036 kB
Shmem:               164 kB
Slab:              21068 kB
SReclaimable:      11992 kB
SUnreclaim:         9076 kB
KernelStack:         760 kB
PageTables:         1300 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      243012 kB
Committed_AS:     331640 kB
VmallocTotal:     516096 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:          65536 kB
CmaFree:               0 kB
root@multi-v7-ml:~#
root@multi-v7-ml:/tmp# cp /proc/config.gz .
root@multi-v7-ml:/tmp# gunzip config.gz
root@multi-v7-ml:/tmp# cat config | grep DEVMEM
CONFIG_DEVMEM=y
CONFIG_ARCH_HAS_DEVMEM_IS_ALLOWED=y
# CONFIG_STRICT_DEVMEM is not set
root@multi-v7-ml:/tmp#

-- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/crash-utility/crash/issues/16

RobertBerger commented 7 years ago

Hi Dave,

When I compile it on the target (the one with 512MB of physical memory) and 0 swap, as opposed to cross-compiling it with OE/Yocto it takes some time to compile;) but it runs happily and did so for quite some time - at least dating back to version 6.1.1 of crash. The strange thing is that I can not quite figure out what makes the difference since when compiled on the target the 512MB RAM does not really seem to be a problem. As soon as I'm back in the office I'll try with a board which has more RAM.

working version:

size -A -d crash
crash  :
section                  size      addr
.interp                    19     65876
.note.ABI-tag              32     65896
.note.gnu.build-id         36     65928
.gnu.hash               52796     65964
.dynsym                112848    118760
.dynstr                132280    231608
.gnu.version            14106    363888
.gnu.version_r            176    377996
.rel.dyn                  136    378172
.rel.plt                 2552    378308
.init                      12    380860
.plt                     3848    380872
.text                 3702756    384720
.fini                       8   4087476
.rodata               1324036   4087488
.ARM.exidx                  8   5411524
.eh_frame                   4   5411532
.init_array                 4   5480156
.fini_array                 4   5480160
.jcr                        4   5480164
.dynamic                  280   5480168
.got                     1296   5480448
.data                   69634   5481744
.bss                   440672   5551384
.comment                   17         0
.ARM.attributes            51         0
.debug_aranges          14200         0
.debug_info          17231425         0
.debug_abbrev          489910         0
.debug_line            980843         0
.debug_frame           373752         0
.debug_str             607368         0
.debug_loc            3694749         0
.debug_ranges          395800         0
Total                29645662

non working version:

size -A -d crash
crash  :
section                  size      addr
.interp                    19     65876
.note.ABI-tag              32     65896
.note.gnu.build-id         36     65928
.gnu.hash               44556     65964
.dynsym                112784    110520
.dynstr                132085    223304
.gnu.version            14098    355390
.gnu.version_r            144    369488
.rel.dyn                  136    369632
.rel.plt                 2504    369768
.init                      12    372272
.plt                     3776    372284
.text                 3704036    376064
.fini                       8   4080100
.rodata               1324336   4080112
.ARM.exidx                  8   5404448
.eh_frame                   4   5404456
.init_array                 4   5471972
.fini_array                 4   5471976
.jcr                        4   5471980
.dynamic                  272   5471984
.got                     1272   5472256
.data                   69626   5473528
.bss                   440664   5543160
.comment                   17         0
.ARM.attributes            51         0
.debug_aranges          14136         0
.debug_info          17231488         0
.debug_abbrev          490590         0
.debug_line           1088094         0
.debug_frame           376292         0
.debug_str             628462         0
.debug_loc            3696561         0
.debug_ranges          397408         0
Total                29773519

Of course the OE version contains those extra patches which might make the difference:

http://cgit.openembedded.org/meta-openembedded/tree/meta-oe/recipes-kernel/crash/crash

RobertBerger commented 7 years ago

I just made a test on another platform with more memory and an update version of OE/Yocto (before I used 2.3 and now it's 2.3.1) and even the bitbaked version works happily.

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi"...

      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 4
        DATE: Wed Aug 23 20:53:36 2017
      UPTIME: 00:04:44
LOAD AVERAGE: 1.13, 0.66, 0.28  
       TASKS: 128
    NODENAME: multi-v7-ml
     RELEASE: 4.9.44-custom-student-custom-ml-debug-dirty
     VERSION: #1 SMP Sun Aug 27 12:42:40 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 1 GB
         PID: 650
     COMMAND: "crash"
        TASK: ec91b000  [THREAD_INFO: edec6000]
         CPU: 1
       STATE: TASK_RUNNING (ACTIVE)

crash>
crash>
crash>
crash> p jiffies
jiffies = $1 = 4280
crash> p jiffies
jiffies = $2 = 4707
cat /proc/meminfo
MemTotal:        1003536 kB
MemFree:          678948 kB
MemAvailable:     954460 kB
Buffers:              20 kB
Cached:           277876 kB
SwapCached:            0 kB
Active:           248448 kB
Inactive:          36064 kB
Active(anon):       6696 kB
Inactive(anon):       88 kB
Active(file):     241752 kB
Inactive(file):    35976 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:        262144 kB
HighFree:         127948 kB
LowTotal:         741392 kB
LowFree:          551000 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          6500 kB
Mapped:            11840 kB
Shmem:               172 kB
Slab:              27988 kB
SReclaimable:      14028 kB
SUnreclaim:        13960 kB
KernelStack:         960 kB
PageTables:          668 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      501768 kB
Committed_AS:      21604 kB
VmallocTotal:     245760 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:          65536 kB
CmaFree:           32524 kB

I'll keep on digging

RobertBerger commented 7 years ago

With the board with 512 MB of RAM and the new setup (Yocto 2.3.1) and crash bitbaked it still does not work, similar to Yocto 2.3:

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "arm-poky-linux-gnueabi"...

crash: vmlinux: Cannot allocate memory
      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Wed Aug 23 20:15:20 2017
      UPTIME: 00:05:55
LOAD AVERAGE: 1.35, 1.28, 0.63
       TASKS: 99
    NODENAME: multi-v7-ml
     RELEASE: 4.9.44-custom-student-custom-ml-debug
     VERSION: #1 SMP Sun Aug 27 14:15:36 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
         PID: 659
     COMMAND: "crash"
        TASK: d9dc1000  [THREAD_INFO: d9a14000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)

crash> p jiffies
crash: fork system call failed: Cannot allocate memorycrash: cannot open pipe
crash> 
cat /proc/meminfo
MemTotal:         486028 kB
MemFree:          306664 kB
MemAvailable:     453000 kB
Buffers:             240 kB
Cached:           144744 kB
SwapCached:            0 kB
Active:            33264 kB
Inactive:         118324 kB
Active(anon):       6692 kB
Inactive(anon):       92 kB
Active(file):      26572 kB
Inactive(file):   118232 kB
Unevictable:           0 kB
Mlocked:               0 kB
HighTotal:             0 kB
HighFree:              0 kB
LowTotal:         486028 kB
LowFree:          306664 kB
SwapTotal:             0 kB
SwapFree:              0 kB
Dirty:                 0 kB
Writeback:             0 kB
AnonPages:          6632 kB
Mapped:            11708 kB
Shmem:               172 kB
Slab:              21528 kB
SReclaimable:      12372 kB
SUnreclaim:         9156 kB
KernelStack:         784 kB
PageTables:          668 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:      243012 kB
Committed_AS:      22880 kB
VmallocTotal:     516096 kB
VmallocUsed:           0 kB
VmallocChunk:          0 kB
CmaTotal:          65536 kB
CmaFree:           32700 kB
RobertBerger commented 7 years ago

And crash built on the target seems to work.

took some time to build it:

real    165m59.071s
user    58m26.725s
sys     34m53.630s

but:

crash 7.1.9
Copyright (C) 2002-2016  Red Hat, Inc.
Copyright (C) 2004, 2005, 2006, 2010  IBM Corporation
Copyright (C) 1999-2006  Hewlett-Packard Co
Copyright (C) 2005, 2006, 2011, 2012  Fujitsu Limited
Copyright (C) 2006, 2007  VA Linux Systems Japan K.K.
Copyright (C) 2005, 2011  NEC Corporation
Copyright (C) 1999, 2002, 2007  Silicon Graphics, Inc.
Copyright (C) 1999, 2000, 2001, 2002  Mission Critical Linux, Inc.
This program is free software, covered by the GNU General Public License,
and you are welcome to change it and/or distribute copies of it under
certain conditions.  Enter "help copying" to see the conditions.
This program has absolutely no warranty.  Enter "help warranty" for details.

GNU gdb (GDB) 7.6
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
and "show warranty" for details.
This GDB was configured as "armv7l-unknown-linux-gnueabi"...

      KERNEL: vmlinux
    DUMPFILE: /dev/mem
        CPUS: 1
        DATE: Wed Aug 23 23:37:06 2017
      UPTIME: 03:27:41
LOAD AVERAGE: 1.26, 1.17, 0.78
       TASKS: 99
    NODENAME: multi-v7-ml
     RELEASE: 4.9.44-custom-student-custom-ml-debug
     VERSION: #1 SMP Sun Aug 27 14:15:36 CEST 2017
     MACHINE: armv7l  (unknown Mhz)
      MEMORY: 510 MB
         PID: 31613
     COMMAND: "crash"
        TASK: da895000  [THREAD_INFO: cba4e000]
         CPU: 0
       STATE: TASK_RUNNING (ACTIVE)

crash>
crash> p jiffies
jiffies = $1 = 3391350
crash>