Explore moving xochitl off of root partition

[Eeems]

Since we have limited space on the root partition at this point, we should explore moving it off of root. Likely as part of install we should move it to /home/root and symlink to it. The ddvk-hacks package will also need to be updated to handle the new location.

@matteodelabre @raisjn @LinusCDE thoughts?

[Etn40ff]

Is there no other way around moving stuff? Should we consider repartitioning?

In any case I would probably move libicudata first:

$ find / -type f -exec du {} \; | grep -v home | grep -v opt | sort -n | tail
3703    /usr/lib/libQt5Quick.so.5.15.1
4043    /usr/lib/libQt5Qml.so.5.15.1
4876    /usr/lib/libQt5Gui.so.5.15.1
5136    /dev/shm/swtfb.01
5267    /boot/zImage
5569    /usr/lib/libQt5Core.so.5.15.1
5682    /usr/share/misc/magic.mgc
5774    /usr/bin/xochitl
8243    /etc/udev/hwdb.bin
27380   /usr/lib/libicudata.so.66.1

[Eeems]

Since we already move xochitl around for ddvk-hacks, it seems like low hanging fruit to give a little bit of extra space. Moving other libraries is interesting and would give more space. Straight up removing unused libraries would provide even more space, but could have unintended side effects if a developer started assuming they were available.

As for repartitioning, it's possible but it introduces a much larger risk of bricking the device. I don't feel comfortable offering that as part of toltec yet due to the risk. It's also not clear what effect that would have on long term updates etc from reMarkable themselves, as they will assume a certain partition layout.

[matteodelabre]

These are interesting ideas, but moving the xochitl binary could prove to be a bit fragile and we will need to think about all corner cases and interactions. I’m not saying this isn’t doable, since as you pointed out we already do something similar with the ddvk-hacks package.

Another idea which could be more robust would be to provide our own Toltec root filesystem (without Xochitl), and to provide Xochitl as a separate package. This is no trivial task, but the main pieces are almost there, thanks to @alistair23 and @Etn40ff’s excellent work on kernel packaging, and we could investigate using OpenWrt to make a basic image.

[Eeems]

I do like the second item medium to long term. I think we have an immediate need though as the latest OS has an extreme space problem again, and this might give us a quick win to hold us over for a little bit. That and bind mounting /var/log to the home partition.

[LinusCDE]

We could also consider compressing packing binaries with upx. For some binaries the packing doesn't work or executing the binary fails. But packing a lot of things like built-in wget, vim, nano etc. might work and save some space.

I would not recommend this for xochitl, as this makes finding memory regions and patching for ddvk likely impossible. Also if the binary gets to complex for fails, it could soft-brick the device / ui.

[Eeems]

Do we know if the binaries have been stripped or not?

[LinusCDE]

Do we know if the binaries have been stripped or not?

According to file, at least vim.vim, xochitl, sync and minidump_stackwalk are stripped (those seem to be the biggest binaries).

Not sure. But xz can usually compress binaries pretty heavily (getting about a third of the original size is common) I think the main problems of upx making a binary unusable where with newer rust builds. Other binaries should be find and the project is pretty mature in general.

Not entirely sure but I think upx sometimes strips the binaries anyway.

Using the latest arm release of upx on the vim.vim and sync binary seem already pretty promising. Giving us about 1,8 MB extra space. Both binaries seem to continue working fine.

If simply compressing the biggest binaries (minidump_stackwalk, strace, sync and vim.vim) we get from a size of 4,4 MiB to 1.7 MiB => saves 2.7 MiB. With brute we get down to 1.3 MiB, but I doubt the extra time is worth the ~400 KiB saved.

Xochitl is the biggest binary and would save us 3.7 MiB alone, but then it would likely break ddvk-hacks as already mentioned.

If we wanted to use this, we could pretty easily package up the upx release as a toltec package and compress specific binaries if not compressed already as part of bootstrap or similar.

[Etn40ff]

As discussed on discord, here is a possible list of libraries that may not be used. It is the output of

diff <(find lib usr/lib | grep \\.so  | awk -F ".so" '{print "/"$1}' | sort | uniq) <(find . -executable -type f -exec /opt/x-tools/arm-remarkable-linux-gnueabihf/bin/arm-linux-gnueabihf-ldd  --root=. {} \; | grep "=>" | awk '{print $3}' | awk -F ".so" '{print $1}' | sort | uniq) | grep "<" | awk '{print $2}'

run in a docker image of ghcr.io/toltec-dev/base:v2.2 with the current working directory being the image contained in 2.9.1.236_reMarkable2.signed.

I would probably triple-check berore saying that something like libc is not used :)

/lib/ld-2.31
/lib/libBrokenLocale
/lib/libBrokenLocale-2.31
/lib/libanl
/lib/libanl-2.31
/lib/libc-2.31
/lib/libdl-2.31
/lib/libm-2.31
/lib/libnsl
/lib/libnsl-2.31
/lib/libnss_compat
/lib/libnss_compat-2.31
/lib/libnss_dns
/lib/libnss_dns-2.31
/lib/libnss_files
/lib/libnss_files-2.31
/lib/libnss_myhostname
/lib/libpthread-2.31
/lib/librt-2.31
/lib/libthread_db
/lib/libthread_db-1.0
/lib/libutil-2.31
/lib/modules/5.4.70/modules
/lib/systemd/system
/lib/systemd/system/dbus
/lib/systemd/system/dbus.target.wants/dbus
/lib/systemd/system/dropbear
/lib/systemd/system/syslog
/lib/systemd/system/systemd-initctl
/lib/systemd/system/systemd-journald
/lib/systemd/system/systemd-journald-audit
/lib/systemd/system/systemd-journald-dev-log
/lib/systemd/system/systemd-networkd
/lib/systemd/system/systemd-rfkill
/lib/systemd/system/systemd-udevd-control
/lib/systemd/system/systemd-udevd-kernel
/usr/lib/dhcpcd/dev/udev
/usr/lib/directfb-1.7-7/inputdrivers/libdirectfb_linux_input
/usr/lib/directfb-1.7-7/interfaces/ICoreR
/usr/lib/directfb-1.7-7/interfaces/IDirectFBFont/libidirectfbfont_dgiff
/usr/lib/directfb-1.7-7/interfaces/IDirectFBFont/libidirectfbfont_ft2
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_bmp
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_dfiff
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_gif
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_jpeg
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_mpeg2
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_png
/usr/lib/directfb-1.7-7/interfaces/IDirectFBImageProvider/libidirectfbimageprovider_pnm
/usr/lib/directfb-1.7-7/interfaces/IDirectFBVideoProvider/libidirectfbvideoprovider_gif
/usr/lib/directfb-1.7-7/interfaces/IDirectFBVideoProvider/libidirectfbvideoprovider_v4l
/usr/lib/directfb-1.7-7/interfaces/IDirectFBWindows/libidirectfbwindows_default
/usr/lib/directfb-1.7-7/interfaces/IWater/libiwater_default
/usr/lib/directfb-1.7-7/systems/libdirectfb_devmem
/usr/lib/directfb-1.7-7/systems/libdirectfb_dummy
/usr/lib/directfb-1.7-7/systems/libdirectfb_fbdev
/usr/lib/directfb-1.7-7/wm/libdirectfbwm_default
/usr/lib/gconv/IBM437
/usr/lib/heaptrack/libheaptrack_inject
/usr/lib/heaptrack/libheaptrack_preload
/usr/lib/libQt5Concurrent
/usr/lib/libQt5QmlWorkerScript
/usr/lib/libQt5QuickControls2
/usr/lib/libQt5QuickShapes
/usr/lib/libQt5QuickTemplates2
/usr/lib/libQt5QuickTest
/usr/lib/libQt5Test
/usr/lib/libQtWebAppLogging
/usr/lib/libQtWebAppTemplateEngine
/usr/lib/libasan
/usr/lib/libgflags_nothreads
/usr/lib/libgthread-2.0
/usr/lib/libnl/cli/cls/basic
/usr/lib/libnl/cli/cls/cgroup
/usr/lib/libnl/cli/qdisc/bfifo
/usr/lib/libnl/cli/qdisc/blackhole
/usr/lib/libnl/cli/qdisc/fq_codel
/usr/lib/libnl/cli/qdisc/hfsc
/usr/lib/libnl/cli/qdisc/htb
/usr/lib/libnl/cli/qdisc/ingress
/usr/lib/libnl/cli/qdisc/pfifo
/usr/lib/libnl/cli/qdisc/plug
/usr/lib/libprotobuf-c
/usr/lib/libubsan
/usr/lib/libunwind-arm
/usr/lib/libunwind-coredump
/usr/lib/libunwind-ptrace
/usr/lib/libunwind-setjmp
/usr/lib/plugins/generic/libqevdevkeyboardplugin
/usr/lib/plugins/generic/libqevdevmouseplugin
/usr/lib/plugins/generic/libqevdevtabletplugin
/usr/lib/plugins/generic/libqevdevtouchplugin
/usr/lib/plugins/generic/libqlibinputplugin
/usr/lib/plugins/generic/libqtuiotouchplugin
/usr/lib/plugins/iconengines/libqsvgicon
/usr/lib/plugins/imageformats/libqjpeg
/usr/lib/plugins/imageformats/libqsvg
/usr/lib/plugins/platforms/libepaper
/usr/lib/plugins/platforms/libqdirectfb
/usr/lib/plugins/platforms/libqminimal
/usr/lib/plugins/platforms/libqoffscreen
/usr/lib/plugins/platforms/libqvnc
/usr/lib/plugins/platformthemes/libqxdgdesktopportal
/usr/lib/plugins/qmltooling/libqmldbg_debugger
/usr/lib/plugins/qmltooling/libqmldbg_inspector
/usr/lib/plugins/qmltooling/libqmldbg_local
/usr/lib/plugins/qmltooling/libqmldbg_messages
/usr/lib/plugins/qmltooling/libqmldbg_native
/usr/lib/plugins/qmltooling/libqmldbg_nativedebugger
/usr/lib/plugins/qmltooling/libqmldbg_preview
/usr/lib/plugins/qmltooling/libqmldbg_profiler
/usr/lib/plugins/qmltooling/libqmldbg_quickprofiler
/usr/lib/plugins/qmltooling/libqmldbg_server
/usr/lib/plugins/qmltooling/libqmldbg_tcp
/usr/lib/qml/Qt/labs/animation/liblabsanimationplugin
/usr/lib/qml/Qt/labs/calendar/libqtlabscalendarplugin
/usr/lib/qml/Qt/labs/folderlistmodel/libqmlfolderlistmodelplugin
/usr/lib/qml/Qt/labs/platform/libqtlabsplatformplugin
/usr/lib/qml/Qt/labs/qmlmodels/liblabsmodelsplugin
/usr/lib/qml/Qt/labs/settings/libqmlsettingsplugin
/usr/lib/qml/Qt/labs/sharedimage/libsharedimageplugin
/usr/lib/qml/Qt/labs/wavefrontmesh/libqmlwavefrontmeshplugin
/usr/lib/qml/QtQml/Models.2/libmodelsplugin
/usr/lib/qml/QtQml/StateMachine/libqtqmlstatemachine
/usr/lib/qml/QtQml/WorkerScript.2/libworkerscriptplugin
/usr/lib/qml/QtQml/libqmlplugin
/usr/lib/qml/QtQuick.2/libqtquick2plugin
/usr/lib/qml/QtQuick/Controls.2/Fusion/libqtquickcontrols2fusionstyleplugin
/usr/lib/qml/QtQuick/Controls.2/Imagine/libqtquickcontrols2imaginestyleplugin
/usr/lib/qml/QtQuick/Controls.2/Material/libqtquickcontrols2materialstyleplugin
/usr/lib/qml/QtQuick/Controls.2/Universal/libqtquickcontrols2universalstyleplugin
/usr/lib/qml/QtQuick/Controls.2/libqtquickcontrols2plugin
/usr/lib/qml/QtQuick/Layouts/libqquicklayoutsplugin
/usr/lib/qml/QtQuick/Shapes/libqmlshapesplugin
/usr/lib/qml/QtQuick/Templates.2/libqtquicktemplates2plugin
/usr/lib/qml/QtQuick/Window.2/libwindowplugin
/usr/lib/qml/QtTest/libqmltestplugin
/usr/lib/qml/QtWebSockets/libdeclarative_qmlwe
/usr/lib/systemd/user
/usr/lib/systemd/user/dbus

[Eeems]

The systemd libraries are likely used as well. Especially some of them like dropbear. Some of the Qt stuff might also be used in a way that ldd might not catch, that or are used by third party code that toltec hosts. libprotobuf is used by the update engine I believe as well. libunwind would likely be used by their crash reporting.

[EntrixIII]

Considering that (if I understood it right) most (if not all) of the additional writing done onto root partition is done because of logging, I feel like @Eeems' idea of

bind mounting /var/log to the home partition

is a logical conclusion that I assume would have a much lower risk of breakage compared to moving xochitl or libraries, and would allow to buy some time to work on other solutions.

The only problem I see with this is the transition to a /var/log bind mount, and depending on future solutions, perhaps the transition away from that bind mount.

(Also, it's been about a month since the last reply... so I assume it might not be that time critical? Or maybe it's because 2.13 support wasn't that big of a deal because of the lack of rm2fb and ddvk-hacks support at the time.)