Running out of memory during initial update

sakaki- / gentoo-on-rpi-64bit

Bootable 64-bit Gentoo image for the Raspberry Pi4B, 3B & 3B+, with Linux 5.4, OpenRC, Xfce4, VC4/V3D, camera and h/w codec support, weekly-autobuild binhost

GNU General Public License v3.0

921 stars 126 forks source link

Running out of memory during initial update #59

Closed BloggerBust closed 5 years ago

BloggerBust commented 5 years ago

First of all I wanted to thank you for this contribution to the gentoo / raspberry pi communities. The time and effort that you have spent on this project is appreciated.

hardware: Raspberry Pi 3B screen: Original Kano 10.1 inch HDMI

Relevant files & configuration

Mitigations taken to reduce memory usage I shut down none-essential services to save on memory. I then connected from another machine via ssh and used emacs daemon to execute the long running install. These are the rc services I had up / down during install

Status of init scripts in runlevel "default"
  NetworkManager            [started]
  bluetooth                 [started]
  chronyd                   [started]
  consolekit                [started]
  cronie                    [started]
  cupsd                     [stopped]
  dbus                      [started]
  local                     [started]
  netmount                  [stopped]
  rpi3-bluetooth            [started]
  rpi3-ethfix               [started]
  rpi3-safecompositor       [stopped]
  rpi3-safecursor           [stopped]
  rpi3-wifi-regdom          [stopped]
  sshd                      [started]
  syslog-ng                 [started]
  xdm                       [stopped]

I checked with free and at the time I was using about 58MB of RAM.

The install I ran the install with genup -v. See the out of memory failure on pastebin. After the failure I checked with free to see how much swap I had used and how much memory was currently being used:

              total        used        free      shared  buff/cache   available
Mem:          968Mi       106Mi       485Mi       0.0Ki       377Mi       849Mi
Swap:         511Mi       407Mi       104Mi

Thoughts So obviously during install I used all RAM and then all swap. I am going to try and run again with -pipe turned off. I will report back. any ideas why I might be having so many problems running out of RAM. I have not installed any additional packages. This was just an update directly from your image and I seem to recall that you are able to run the install directly from a PI without running out of RAM. When you run the install is the PI headless, or are you running xdm?

sakaki- commented 5 years ago

@BloggerBust, thanks for reporting this. This is an issue I'm aware of, and am going to be fixing in the next release of the image (in preparation now, should ship in 2 weeks or so).

The root problem here is that the current /etc/portage/make.conf contains the following:

# for use when compiling locally
MAKEOPTS="-j5 -l4"
EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=4"
# for use with compiling with distcc only
#MAKEOPTS="-j8 -l4"
#EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=4"

This means to allow up to five parallel compilation threads per emerge, and up to five simultaneous emerges, subject to system load constraints, and is too aggressive (wrt memory usage) in many cases, given the Pi's small memory footprint (and IO capability, when it starts to hit swap).

To address this, I've added (to the profile make.defaults) more conservative settings (commit). To use these, just issue:

pi64 ~ # emaint sync --repo rpi3

Then comment out the above lines in /etc/portage/make.conf, so they read:

# for use when compiling locally
#MAKEOPTS="-j5 -l4"
#EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=4"
# for use with compiling with distcc only
#MAKEOPTS="-j8 -l4"
#EMERGE_DEFAULT_OPTS="--jobs=5 --load-average=4"

Leave the rest of the file as-is, and save it. Then retry your genup run.

hth, sakaki

BloggerBust commented 5 years ago

Thank you for your quick response. I have followed your instructions and will report back. I am going to be out of town for the next couple of days though so you won't here back from me until Monday.

Thanks again!

BloggerBust commented 5 years ago

Well, I did not end up leaving town as planned. I tried to reconnect to the pi yesterday, but was unable to. It lost connectivity with my router at some point. Since I had no xdm service running and I was unable to switch to a tty, I had to hard boot it. When I reached the logon screen there was an xconsole window with the text "Console log for pi64". It affected the dimensions of the sign on splash screen making it not render properly. When I signed in everything looked normal, but the xconsole screen was now displayed in its own window with the same message. I am just taking a look at the state of things now and will report back.

BloggerBust commented 5 years ago

Well, it wasn't clear to me what actually went wrong with the emerge. The last thing installed was libqxp based on the last thing logged in emerge.log

$ sudo tail /var/log/emerge.log                                                                                                                                                                                             
1537643204:  === (4 of 43) Compiling/Packaging (app-text/libebook-0.1.3::/usr/portage/app-text/libebook/libebook-0.1.3.ebuild)                                                                                                                 
1537643206:  >>> emerge (5 of 43) app-text/libmspub-0.1.4 to /                                                                                                                                                                                 
1537643206:  === (5 of 43) Cleaning (app-text/libmspub-0.1.4::/usr/portage/app-text/libmspub/libmspub-0.1.4.ebuild)                                                                                                                            
1537643207:  === (5 of 43) Compiling/Packaging (app-text/libmspub-0.1.4::/usr/portage/app-text/libmspub/libmspub-0.1.4.ebuild)                                                                                                                 
1537643210:  >>> emerge (6 of 43) media-libs/libvisio-0.1.6 to /                                                                                                                                                                               
1537643210:  === (6 of 43) Cleaning (media-libs/libvisio-0.1.6::/usr/portage/media-libs/libvisio/libvisio-0.1.6.ebuild)                                                                                                                        
1537643210:  === (6 of 43) Compiling/Packaging (media-libs/libvisio-0.1.6::/usr/portage/media-libs/libvisio/libvisio-0.1.6.ebuild)                                                                                                             
1537643213:  >>> emerge (7 of 43) app-text/libqxp-0.0.1 to /                                                                                                                                                                                   
1537643213:  === (7 of 43) Cleaning (app-text/libqxp-0.0.1::/usr/portage/app-text/libqxp/libqxp-0.0.1.ebuild)                                                                                                                                  
1537643214:  === (7 of 43) Compiling/Packaging (app-text/libqxp-0.0.1::/usr/portage/app-text/libqxp/libqxp-0.0.1.ebuild)

In the portage elog summary it is complaining about the fact that there is no linux kernel sources.

I also checked dmesg for error | fail

[    0.112682] dmi: Firmware registration failed.
[    0.707787] Error: Driver 'sdhost-bcm2835' is already registered, aborting...

I decided to just try updating again this time with genup -av so that I can be prompted and review what it is doing.

One thing that is not helping is that I do not have a fan on my raspberry-pi case so the cpu is overheating during the build. I get the red thermometer on the screen.

I did try a reboot to see if I still get the xconsole window open. I do. Any ideas what that might be about?

Cheers!

sakaki- commented 5 years ago

The message about kernel sources is benign - some ebuilds try to find /usr/src/linux on your machine, to test the .config settings there. The image uses a binary kernel package so this directory isn't populated, but that should only cause a warning, not stop the build.

The dmesg output messages are also benign: please see this post regarding the first (dmi); and this bug report regarding the second.

Neither should cause any major issues.

Please do let me know how your genup run goes next time.

hth, S.

BloggerBust commented 5 years ago

Thank you sakaki,

I ran genup again without xdm to save on RAM. When I ran genup -av I only had 68MiB in use, but I lost connectivity again:

Would you like to merge these packages? [Yes/No] yes
Sorry, response 'yes   ' not understood. [Yes/No] yes
>>> Verifying ebuild manifests
>>> Running pre-merge checks for sys-kernel/bcmrpi3-kernel-bis-bin-4.14.71.20180922
>>> Running pre-merge checks for sys-libs/libomp-7.0.0
 * Determining the location of the kernel source code
 * Unable to find kernel sources at /usr/src/linux
 * Please make sure that /usr/src/linux points at your running kernel,
 * (or the kernel you wish to build against).
 * Alternatively, set the KERNEL_DIR environment variable to the kernel sources location
 * Unable to calculate Linux Kernel version for build, attempting to use running version
 * Unable to check for the following kernel config options due
 * to absence of any configured kernel sources or compiled
 * config:
 *  - SCHED_PDS - PDS scheduler versions >= 0.98c < 0.98i (e.g. used in kernels >= 4.13-pf11
 * < 4.14-pf9) do not implement sched_yield() call which may result in horrible
 * performance problems with libomp. If you are using one of the specified
 * kernel versions, you may want to disable the PDS scheduler.
 * You're on your own to make sure they are set if needed.
>>> Running pre-merge checks for media-libs/mesa-18.2.0-r1
 * Ignoring USE=xa         since VIDEO_CARDS does not contain freedreno, nouveau, or vmware
 * Ignoring USE=xvmc       since VIDEO_CARDS does not contain r600 or nouveau
>>> Running pre-merge checks for sys-libs/compiler-rt-7.0.0
>>> Running pre-merge checks for sys-fs/eudev-3.2.6
 *
 * As of 2013-01-29, eudev-3.2.6 provides the new interface renaming functionality,
 * as described in the URL below:
 * https://www.freedesktop.org/wiki/Software/systemd/PredictableNetworkInterfaceNames
 *
 * This functionality is enabled BY DEFAULT because eudev has no means of synchronizing
 * between the default or user-modified choice of sys-fs/udev.  If you wish to disable
 * this new iface naming, please be sure that /etc/udev/rules.d/80-net-name-slot.rules
 * exists: touch /etc/udev/rules.d/80-net-name-slot.rules
 *
>>> Running pre-merge checks for app-office/libreoffice-6.1.1.2
 * Checking for at least 512 MiB RAM ...                                                                                                                                                                                               [ ok ]
 * Checking for at least 6 GiB disk space at "/var/tmp/portage/app-office/libreoffice-6.1.1.2/temp" ...                                                                                                                                [ ok ]
>>> Running pre-merge checks for net-misc/networkmanager-1.14.0
 * Determining the location of the kernel source code
 * Unable to find kernel sources at /usr/src/linux
 * Please make sure that /usr/src/linux points at your running kernel,
 * (or the kernel you wish to build against).
 * Alternatively, set the KERNEL_DIR environment variable to the kernel sources location
 * Was unable to determine your kernel .config
 * Please note that if CONFIG_SYSFS_DEPRECATED_V2 is set in your kernel .config, NetworkManager will not work correctly.
 * See https://bugs.gentoo.org/333639 for more info.
>>> Running pre-merge checks for net-wireless/blueman-2.0.4-r1
 * blueman-2.0.4-r1.tbz2 MD5 SHA1 size ;-) ...                           [ ok ]
>>> Emerging binary (1 of 57) sys-boot/rpi3-64bit-firmware-1.20180919::rpi3
>>> Emerging binary (2 of 57) sys-apps/less-538::gentoo
>>> Emerging (3 of 57) sys-devel/binutils-2.31.1-r1::gentoo
>>> Installing (1 of 57) sys-boot/rpi3-64bit-firmware-1.20180919::rpi3
>>> Jobs: 1 of 57 complete, 1 running               Load avg: 4.61, 4.14, 3.02packet_write_wait: Connection to 192.168.10.163 port 22: Broken pipe
dustfinger@galactica ~ $ ssh dustfinger@192.168.10.163

I think the problem might be related to the cpu overheating during the install. I am going to try rebooting and this time running genup with xdm, that way when it fails I might be able to see why.

BloggerBust commented 5 years ago

The system is still running out of RAM. I checked makeopts and it was set to run two concurrent builds with a load average of 1.

MAKEOPTS="-j2 -l1"

I then tried setting MAKEOPTS to empty string, but emerge failed again. The logs show that it is compiling multiple packages at once still.

$ emerge --info | grep -i 'makeopts'
MAKEOPTS=""

Here is the last output from emerge on my screen before the pipe broke and I lost connection to the raspberry pi

>>> Running pre-merge checks for net-wireless/blueman-2.0.4-r1
 * blueman-2.0.4-r1.tbz2 MD5 SHA1 size ;-) ...                           [ ok ]
>>> Emerging (1 of 62) virtual/cargo-1.28.0::gentoo
>>> Emerging binary (2 of 62) sys-apps/less-538::gentoo
>>> Emerging (3 of 62) sys-devel/binutils-2.31.1-r1::gentoo
>>> Installing (1 of 62) virtual/cargo-1.28.0::gentoo
>>> Jobs: 1 of 62 complete, 1 running               Load avg: 5.27, 4.32, 3.20packet_write_wait: Connection to 192.168.10.163 port 22: Broken pipe

Here you can see that when MAKEOPTS is set to empty string the system seems to default to -j5 -l4

$ sudo tail -n 20 /var/log/emerge.log
1537757078:  === regen
1537757263:  *** terminating.
1537757326: Started emerge on: Sep 23, 2018 20:48:46
1537757326:  *** emerge --oneshot --update --jobs=5 --load-average=4.0 --verbose --getbinpkg --usepkg portage
1537757355:  *** exiting successfully.
1537757355:  *** terminating.
1537759151: Started emerge on: Sep 23, 2018 21:19:10
1537759151:  *** emerge --update --backtrack=50 --deep --jobs=5 --load-average=4.0 --with-bdeps=y --reinstall=changed-use --verbose --getbinpkg --usepkg @world
1537761101:  >>> emerge (1 of 62) virtual/cargo-1.28.0 to /
1537761101:  === (1 of 62) Cleaning (virtual/cargo-1.28.0::/usr/portage/virtual/cargo/cargo-1.28.0.ebuild)
1537761102:  === (1 of 62) Compiling/Packaging (virtual/cargo-1.28.0::/usr/portage/virtual/cargo/cargo-1.28.0.ebuild)
1537761103:  >>> emerge (2 of 62) sys-apps/less-538 to /
1537761104:  === (2 of 62) Merging Binary (sys-apps/less-538::/usr/portage/packages/sys-apps/less-538.tbz2)
1537761105:  >>> emerge (3 of 62) sys-devel/binutils-2.31.1-r1 to /
1537761105:  === (3 of 62) Cleaning (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537761110:  === (3 of 62) Compiling/Packaging (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537761136:  === (1 of 62) Merging (virtual/cargo-1.28.0::/usr/portage/virtual/cargo/cargo-1.28.0.ebuild)
1537761155:  >>> AUTOCLEAN: virtual/cargo:0
1537761169:  === (1 of 62) Post-Build Cleaning (virtual/cargo-1.28.0::/usr/portage/virtual/cargo/cargo-1.28.0.ebuild)
1537761169:  ::: completed emerge (1 of 62) virtual/cargo-1.28.0 to /

I guess that is what makeopts defaults to when given a blank string. Otherwise, according to the rpi3 profile targets from your recent commit, makeopts defaults should be:

dustfinger@pi64 ~ $ grep -i 'makeopts' /usr/local/portage/rpi3/profiles/targets/rpi3/make.defaults
MAKEOPTS="-j2 -l1"

I am going to try the build again this time explicitly setting MAKEOPTS to -j1 -l1 and will report back.

sakaki- commented 5 years ago

MAKEOPTS constrains the number of parallel threads during the emerge of a given package. EMERGE_DEFAULT_OPTS constrains the number of such emerge jobs that are allowed to run simultaneously. To ensure only one thing runs at once set:

MAKEOPTS="-j1"
EMERGE_DEFAULT_OPTS="--jobs=1"

in /etc/portage/make.conf.

BloggerBust commented 5 years ago

Thank you. I ignorantly assumed that EMERGE_DEFAULT_OPTS were the defaults if MAKEOPTS were not provided. I read the make.conf and emerge man pages now for those parameters and have been enlightened :-) I am running another build now.

BloggerBust commented 5 years ago

The build is still failing, I took a step back to just observe a little. Here is the current output of emerge info. You can see that MAKEOPTS is set to make serially and not parallel. Also, EMERGE_DEFAULT_OPTS is set to single jobs. i.e.

MAKEOPTS="-j1"
EMERGE_DEFAULT_OPTS="--jobs=1"

Yet, the tail of emerge.log is showing that the emerge configuration is not being respected:

dustfinger@pi64 ~ $ sudo tail /var/log/emerge.log
1537873831:  ::: completed emerge (2 of 65) dev-db/unixODBC-2.3.7 to /
1537880453: Started emerge on: Sep 25, 2018 07:00:53
1537880453:  *** emerge --oneshot --update --jobs=5 --load-average=4.0 --verbose --getbinpkg --usepkg portage
1537880485:  *** exiting successfully.
1537880485:  *** terminating.
1537880499: Started emerge on: Sep 25, 2018 07:01:39
1537880499:  *** emerge --update --ask --backtrack=50 --deep --jobs=5 --load-average=4.0 --with-bdeps=y --reinstall=changed-use --verbose --getbinpkg --usepkg @world
1537882586:  >>> emerge (1 of 63) sys-devel/binutils-2.31.1-r1 to /
1537882586:  === (1 of 63) Cleaning (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537882590:  === (1 of 63) Compiling/Packaging (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
dustfinger@pi64 ~ $

I was curious why it is building sys-devel/binutils-2.31.1-r1, but then I realized that your repo currently only has sys-devel/binutils-2.31.1. I guess I could mask binutils for now so that it picks up your prebuilt binary, but I think it will just fail elsewhere since the actual problem is the parallel builds gobbling up all the RAM and swap.

I decided to use emaint to look for problems:

$ sudo emaint all -pc
Emaint: check binhost      100% [============================================>]
Emaint: check cleanconfmem 100% [============================================>]
Emaint: check cleanresume  100% [============================================>]

resume list 'resume' contains 63 packages

PORT_LOGDIR variable not set or PORT_LOGDIR not a directory.
See the make.conf(5) man page for PORT_LOGDIR_CLEAN usage instructions.

Emaint: check merges       100% [============================================>]
Emaint: check movebin      100% [============================================>]
Emaint: check moveinst     100% [============================================>]
Emaint: check world        100% [============================================>]

I am going to try an emerge -autDN --jobs=1 @world and see if that builds serially. I will report back. I am just going to kick off the build and get some sleep so I won't be reporting back for quite a few hours.

Cheers!

sakaki- commented 5 years ago

@BloggerBust,

OK, I think I see what is happening now, you are using the main Gentoo tree (via emerge-webrsync) and not my weekly-gated one. This will cause you to have to build large packages locally sometimes, when something revbumps in the main tree but my buildserver has not yet caught up. The isshoni.org tree has been frozen since Sep 16 as I'm about to release a new version of the image and I need things stable for testing, but it will be unlocked again shortly. Incidentally, as of the forthcoming 1.3.0 release, I'm switching to using the official Gentoo gemato repository verification (with failure quarantine).

If you are going to be building large packages locally, you'll probably want to set up a distcc server, per my notes here, run headless, and comment out the dtoverlay=vc4-fkms-v3d line in /boot/config.txt. You may also want to use the zswap transparent compressing swap cache, which is active as of release 1.3.0 (forthcoming).

BloggerBust commented 5 years ago

Well, the emerge -autDN --jobs=1 @world definitely solved the parallel builds issue:

dustfinger@pi64 ~ $ sudo tail -n 14 /var/log/emerge.log
1537933210: Started emerge on: Sep 25, 2018 21:40:09
1537933210:  *** emerge --newuse --tree --update --ask --deep --jobs=1 --getbinpkg --usepkg @world
1537934220:  >>> emerge (1 of 62) sys-devel/binutils-2.31.1-r1 to /
1537934220:  === (1 of 62) Cleaning (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537934224:  === (1 of 62) Compiling/Packaging (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537937072:  === (1 of 62) Merging (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537937093:  >>> AUTOCLEAN: sys-devel/binutils:2.31
1537937093:  === Unmerging... (sys-devel/binutils-2.31.1)
1537937115:  >>> unmerge success: sys-devel/binutils-2.31.1
1537937127:  === (1 of 62) Post-Build Cleaning (sys-devel/binutils-2.31.1-r1::/usr/portage/sys-devel/binutils/binutils-2.31.1-r1.ebuild)
1537937127:  ::: completed emerge (1 of 62) sys-devel/binutils-2.31.1-r1 to /
1537937127:  >>> emerge (2 of 62) sys-devel/gcc-8.2.0-r3 to /
1537937128:  === (2 of 62) Cleaning (sys-devel/gcc-8.2.0-r3::/usr/portage/sys-devel/gcc/gcc-8.2.0-r3.ebuild)
1537937131:  === (2 of 62) Compiling/Packaging (sys-devel/gcc-8.2.0-r3::/usr/portage/sys-devel/gcc/gcc-8.2.0-r3.ebuild)
dustfinger@pi64 ~ $

However, you can see it still died while building gcc. What happened? Is there a defect in genup or is there something wrong with my configuration? Here is some information about the gcc build itself:

I honestly don't see what happend here at all. It is as if it just stopped logging. Or froze up. No memory allocation errors. One thing to mention is that I am running this build over wifi. Maybe the network manager is crashing or something as a result of low memory or for some other reason. I could not ssh into the pi this morning so I had to pull the plug as I have shut down xdm to conserve RAM. I will try and set something up to try this build with a lan, but it is really inconvenient for me to do so. So I am not sure when I will be able to do that.

I am going to leave xdm running this time and kick off an emerge -autDN --jobs=1 sys-devel/gcc so that when it fails I can hopefully see what happend from a terminal on the pi itself without having to power it off. If you have any thoughts or ideas I would love to here them. Thanks again for your time.

BloggerBust commented 5 years ago

@BloggerBust,

OK, I think I see what is happening now, you are using the main Gentoo tree (via emerge-webrsync) and not my weekly-gated one. This will cause you to have to build large packages locally sometimes, when something revbumps in the main tree but my buildserver has not yet caught up. The isshoni.org tree has been frozen since Sep 16 as I'm about to release a new version of the image and I need things stable for testing, but it will be unlocked again shortly. Incidentally, as of the forthcoming 1.3.0 release, I'm switching to using the official Gentoo gemato repository verification (with failure quarantine).

If you are going to be building large packages locally, you'll probably want to set up a distcc server, per my notes here, run headless, and comment out the dtoverlay=vc4-fkms-v3d line in /boot/config.txt. You may also want to use the zswap transparent compressing swap cache, which is active as of release 1.3.0 (forthcoming).

Okay, I just saw your reply now. For quickness I will switch to your weekly gated tree, but I have run out of time this morning so I will have to do that tonight. Thanks for your response. You can probably disregard my previous post in this case. Except perhaps the bit about why the EMERGE_DEFAULT_OPTS="--jobs=1" was not being respected when running genup. That might still be an issue, or it could be something wrong with my configuration.

sakaki- commented 5 years ago

Oops, genup was unconditionally overriding MAKEOPTS etc. My bad, apologies ><

This has been fixed now (see above commit). Thanks for reporting! To use new genup:

pi64 ~ # emaint sync --repo sakaki-tools
pi64 ~ # emerge -av1u app-portage/genup

BloggerBust commented 5 years ago

Thank you sakaki, for all of your time and patience.

We did find a few issues, but you fixed them very quickly. I also learnt a few facts from you which was a bonus. I won't be able to do more work on this now until next week. I don't think there are any actual issues left at this point so I am going to close this github issue.

I am teaching my 10 year old daughter to program in python and am setting up the pi as a development machine for her. It used to have the kano OS on it, but they were too slow to update their distro for my liking. We started to run into issues with many essential emacs packages no longer being supported. I use gentoo on my laptop with exwm which my daughter really likes. We have two servers that are not running gentoo, but I think I might create a docker container on one of them based on gentoo and then do cross compiling from there. We just started a blog called Blogger Bust, but it has nothing on it yet. I plan on making the first post next week. I still need to setup the blog so that it can support multiple author's nicely. I wanted to give my daughter a place where she could start writing about the things she is learning and I will do the same of course. My daughter knows all about your project now and says thank you :-)

Cheers!

sakaki- commented 5 years ago

You're welcome! Hope the blog goes well. Best, S.