oresat / oresat-linux

OreSat Linux image builder and utilities.
13 stars 2 forks source link

Need to reduce boot time #34

Closed andrewgreenberg closed 1 year ago

andrewgreenberg commented 1 year ago

Because we're now running the C3 with the Octavo, it's in our best interest to reduce boot times as much as possible. In particular, this means reducing boot time on anything that comes before launching our C3 program.

This discussion seems to imply there's a lot you can do, but it's unclear to me what we do and don't need. They claim to get boot times to ~ 30 seconds, which seems like that might be OK. In particular, Robert Nelson managed to get Startup finished in 1.355s (kernel) + 27.831s (userspace) = 29.187s.

With the kernel booting in a few seconds, I wonder if it's possible to move our program up the udev (systemd?) chain of things to be running quickly while other things (like USB drivers for ethernet) load in the background.

oliver-rew commented 1 year ago

Like suggested in that thread, I trimmed a good 7 seconds on our boot time by deleting initrd? I don't really understand why we dont need initrd, I thought that was a pretty vital part of the boot process, but modern linux is weird.

Last login: Fri Mar 24 02:17:05 2023 from 10.42.1.1
debian@oresat-dev:~$ systemd-analyze
Startup finished in 14.150s (kernel) + 25.670s (userspace) = 39.820s
graphical.target reached after 25.362s in userspace
debian@oresat-dev:~$ sudo rm /boot/initrd.img-5.10.145-ti-r55
...
debian@oresat-dev:~$ systemd-analyze
Startup finished in 7.577s (kernel) + 23.578s (userspace) = 31.156s
graphical.target reached after 23.291s in userspace
ryanpdx commented 1 year ago

So, looking the logs for C3-PB, systemd tries to start the OLAF app 6 seconds after boot and the OLAF app takes an additional 29 seconds to start.

Apr 24 19:19:19 oresat-c3 kernel: Booting Linux on physical CPU 0x0
...
Apr 24 19:19:25 oresat-c3 systemd[1]: Started OreSat Linux App.
....
Apr 24 19:19:54 oresat-c3 python3[485]: 2023-04-24 19:19:54.364 | WARNING  | oresat_c3.subsystems.rtc:__init__:27 - mo>

This is weird as OLAF only about 10 seconds to start from a terminal with the limited resources of the Cortex A8.

After learning about CPUQuota and CPUWeight options in a systemd service file, which gives a daemon more resources to start with, I was able to reduce the OLAF apps start time to 16 second (still slower starting from a terminal, but a lot closer).

Apr 25 02:06:55 oresat-c3 kernel: Booting Linux on physical CPU 0x0
...
Apr 25 02:07:00 oresat-c3 systemd[1]: Started OreSat Linux App.
...
Apr 25 02:07:16 oresat-c3 python3[471]: 2023-04-25 02:07:16.958 | WARNING  | oresat_c3.subsystems.rtc:__init__:27 - mo>

The overall Linux booting to starting a systemd daemon time seems to be fine at 5-6 seconds, just need to focus on getting the OLAF app to start up faster with systemd and the limited resources of a Cortex A8.

ryanpdx commented 1 year ago

Correction: The first log message is after 16 seconds, but the app started after 10 seconds when using systemd (same time as starting from the terminal).

So we just need to get OLAF to start faster on the A8 (the OLAF daemon configs are good).

ryanpdx commented 1 year ago

Removing initrd made no difference to the time from booting linux to starting the OLAF daemon (still about 5 to 6 seconds).

ryanpdx commented 1 year ago

Made a https://github.com/oresat/oresat-olaf/issues/15 issue in the OLAF repo

ryanpdx commented 1 year ago

At the last sw meeting we decide this is okay. Closing.