trinitronx
commented
4 years ago Just realizing that I probably should've checked JournalD log space usage and tmpfs size while it was in crashloop last time... 🤔🤦 although I do have SystemMaxUse=8M set in journald.conf to limit disk usage.
# cat /etc/systemd/journald.conf
# This file is part of systemd.
#
# systemd is free software; you can redistribute it and/or modify it
# under the terms of the GNU Lesser General Public License as published by
# the Free Software Foundation; either version 2.1 of the License, or
# (at your option) any later version.
#
# Entries in this file show the compile time defaults.
# You can change settings by editing this file.
# Defaults can be restored by simply deleting this file.
#
# See journald.conf(5) for details.
[Journal]
Storage=auto
#Compress=yes
#Seal=yes
#SplitMode=uid
#SyncIntervalSec=5m
#RateLimitIntervalSec=30s
#RateLimitBurst=1000
SystemMaxUse=8M
#SystemKeepFree=
#SystemMaxFileSize=
#SystemMaxFiles=100
RuntimeMaxUse=8M
#RuntimeKeepFree=
#RuntimeMaxFileSize=
#RuntimeMaxFiles=100
#MaxRetentionSec=
#MaxFileSec=1month
#ForwardToSyslog=yes
#ForwardToKMsg=no
#ForwardToConsole=no
#ForwardToWall=yes
#TTYPath=/dev/console
#MaxLevelStore=debug
#MaxLevelSyslog=debug
#MaxLevelKMsg=notice
#MaxLevelConsole=info
#MaxLevelWall=emerg
#LineMax=48KAfter this last reboot... disk usage looks ok and especially tmpfs ramdisk mounts are not full:
# df -h
Filesystem Size Used Avail Use% Mounted on
devtmpfs 473M 0 473M 0% /dev
/dev/mmcblk0p8 300M 208M 72M 75% /mnt/sysroot/active
/dev/disk/by-label/resin-state 19M 223K 17M 2% /mnt/state
none 300M 208M 72M 75% /
tmpfs 479M 128K 479M 1% /dev/shm
tmpfs 479M 9.9M 469M 3% /run
tmpfs 479M 0 479M 0% /sys/fs/cgroup
tmpfs 479M 0 479M 0% /tmp
/dev/mmcblk0p7 40M 18M 22M 46% /mnt/boot
tmpfs 479M 20K 479M 1% /var/volatile
/dev/mmcblk0p9 295M 2.1M 273M 1% /mnt/sysroot/inactive
/dev/mmcblk0p11 2.9G 936M 1.8G 35% /mnt/data
robertgzr
Description
Balena Engine / (moby) seems to log many messages about bytes of
blob data. Otherwise, things seem to work fine on this board (Intel Edison with BalenaOSv2.31.5+rev1.After leaving BalenaOS running for a long time
balena.servicestuck in SystemD crashloop with segfaults during startup.Steps to reproduce the issue:
balena.serviceis functional)balena.serviceis in SystemD crashloop loggingSIGSEGVsegfaultsDescribe the results you received:
balena.servicecrashed withSIGSEGV, SystemD tries to restart but keeps failing. It then just gets stuck in this crashloop indefinitely until a reboot.Describe the results you expected:
balena.serviceshould start up normally and stay running on a system with longuptimeAdditional information you deem important (e.g. issue happens only occasionally):
Note: The issue manifests as
SIGSEGVerrors inbalena-engineafter a board is left running for a long time. After a reboot, everything functions normally again. I've been able to reproduce this by leaving the board up and running for a couple days.Since this seems related to memory allocation, I tried looking at the Intel Edison board's memory info while
balena.servicewas experiencing theSIGSEGVcrashloop:Output from
top:After a reboot, while
balena.serviceis up and working:After a reboot, here is
topoutput whilebalena.serviceis working:Output of
balena-engine version:While
balena.servicewas in crashloop:After a reboot:
Output of
balena-engine info:While
balena.servicewas in crashloop:After a reboot:
Additional environment details (device type, OS, etc.):
OS:
uname -a=Linux d816afd 4.16.0-edison-standard #1 SMP Tue Mar 26 12:11:34 UTC 2019 i686 i686 i386 GNU/Linuxcat /etc/os-release=ID="balena-os" NAME="balenaOS" VERSION="2.31.5+rev1" VERSION_ID="2.31.5" PRETTY_NAME="balenaOS 2.31.5+rev1" MACHINE="edison" VARIANT="Production" VARIANT_ID="prod" RESIN_BOARD_REV="5957768" META_RESIN_REV="64859b0" SLUG="intel-edison`
JournalD logs for
balena.servicebalena-engineis in crashloop, failing with a GoLang stacktrace &SIGSEGV/ segmentation fault. Looks like it's something during startup related to some library called "boltdb". The calls fromboltdb/bolt(*Bucket).openBucket()try to allocate memory viaruntime.mallocgc()which callsruntime.heapBitsSetType(). No idea why it's crashingAfter a reboot,
blob datamessages are repeatedly loggedThe only other logs, with the
blob datamessages filtered were: