Building the toolchain and the bootloader

joaomiguelvieira commented 8 months ago

Hi! First of all, congratulations on this excellent work!

Using precompiled binaries, I could easily flash my ESP32-S3-WROOM-1 N8R16 with Linux. However, when trying to recompile the files by myself, I cannot get it working.

While compiling the toolchain (CT_PREFIX=(./builds nice ./ct-ng build), I get the following output:

[INFO ]  Checking dynamic linker symlinks
[EXTRA]    Checking dynamic linker for multilib ''
[ERROR]    collect2: error: ld returned 1 exit status
[ERROR]   
[ERROR]  >>
[ERROR]  >>  Build failed in step 'Checking dynamic linker symlinks'
[ERROR]  >>        called in step '(top-level)'
[ERROR]  >>
[ERROR]  >>  Error happened in: CT_DoExecLog[scripts/functions@377]
[ERROR]  >>        called from: CT__FixupLDSO[scripts/functions@1695]
[ERROR]  >>        called from: CT_IterateMultilibs[scripts/functions@1608]
[ERROR]  >>        called from: CT_MultilibFixupLDSO[scripts/functions@1761]
[ERROR]  >>        called from: uClibc_ng_post_cc[scripts/build/libc/uClibc-ng.sh@335]
[ERROR]  >>        called from: do_libc_post_cc[scripts/build/libc.sh@38]
[ERROR]  >>        called from: main[scripts/crosstool-NG.sh@697]
[ERROR]  >>
[ERROR]  >>  For more info on this error, look at the file: 'build.log'
[ERROR]  >>  There is a list of known issues, some with workarounds, in:
[ERROR]  >>      https://crosstool-ng.github.io/docs/known-issues/
[ERROR]  >>
[ERROR]  >> NOTE: Your configuration includes features marked EXPERIMENTAL.
[ERROR]  >> Before submitting a bug report, try to reproduce it without enabling
[ERROR]  >> any experimental features. Otherwise, you'll need to debug it
[ERROR]  >> and present an explanation why it is a bug in crosstool-NG - or
[ERROR]  >> preferably, a fix.
[ERROR]  >>
[ERROR]  >>  If you feel this is a bug in crosstool-NG, report it at:
[ERROR]  >>      https://github.com/crosstool-ng/crosstool-ng/issues/
[ERROR]  >>
[ERROR]  >>  Make sure your report includes all the information pertinent to this issue.
[ERROR]  >>  Read the bug reporting guidelines here:
[ERROR]  >>      http://crosstool-ng.github.io/support/
[ERROR]   
[ERROR]  (elapsed: 16:36.19)
[16:38] / gmake: *** [ct-ng:261: build] Error 1

Nevertheless, xtensa-esp32s3-linux-uclibcfdpic-gcc still gets produced, and I can successfully compile xipImage, rootfs.cramfs, and etc.jffs2.

However, when I try to compile a simple application using the toolchain (to execute from the board), it always fails the linking process with the following message:

~ # xtensa-esp32s3-linux-uclibcfdpic-gcc hello.c -c -o hello.o
~ # xtensa-esp32s3-linux-uclibcfdpic-gcc hello.o -o hello
/home/joaovieira/esp/rebuild_linux/build/crosstool-NG/builds/xtensa-esp32s3-linux-uclibcfdpic/lib/gcc/xtensa-esp32s3-linux-uclibcfdpic/14.0.0/../../../../xtensa-esp32s3-linux-uclibcfdpic/bin/ld: /home/joaovieira/esp/rebuild_linux/build/crosstool-NG/builds/xtensa-esp32s3-linux-uclibcfdpic/xtensa-esp32s3-linux-uclibcfdpic/sysroot/usr/lib/crt1.o: in function `_start':
(.text+0x10): undefined reference to `__self_reloc'
/home/joaovieira/esp/rebuild_linux/build/crosstool-NG/builds/xtensa-esp32s3-linux-uclibcfdpic/lib/gcc/xtensa-esp32s3-linux-uclibcfdpic/14.0.0/../../../../xtensa-esp32s3-linux-uclibcfdpic/bin/ld: BFD (crosstool-NG 1.25.0.172_c29b568) 2.40.50.20230424 assertion fail /home/joaovieira/esp/rebuild_linux/build/crosstool-NG/.build/xtensa-esp32s3-linux-uclibcfdpic/src/binutils/bfd/elf32-xtensa.c:2699
collect2: fatal error: ld terminated with signal 11 [Segmentation fault], core dumped
compilation terminated.

Could it be related to the error while compiling the toolchain?

Furthermore, I cannot generate the bootloader. After setting up the esp-idf that ships with the script, when loading the environment variables, I get the following error:

~ # ~/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf$ . export.sh
Setting IDF_PATH to '/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf'
Detecting the Python interpreter
Checking "python" ...
Python 3.10.12
"python" has been detected
Adding ESP-IDF tools to PATH...
Traceback (most recent call last):
  File "/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf/tools/idf_tools.py", line 1860, in <module>
    main(sys.argv[1:])
  File "/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf/tools/idf_tools.py", line 1856, in main
    action_func(args)
  File "/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf/tools/idf_tools.py", line 1177, in action_export
    tools_info = filter_tools_info(tools_info)
  File "/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf/tools/idf_tools.py", line 1119, in filter_tools_info
    targets = get_user_defined_targets()
  File "/home/joaovieira/esp/rebuild_linux/build/esp-hosted/esp_hosted_ng/esp/esp_driver/esp-idf/tools/idf_tools.py", line 1099, in get_user_defined_targets
    if env == idf_env_json['idfSelectedId']:
KeyError: 'idfSelectedId'

Also, when using my own esp-idf installation (which works fine for everything else), I get fatal compilation errors. Hence, no binaries are produced.

All in all, I encountered the following issues while running the rebuild-esp32s3-linux-wifi.sh script from which I am unable to recover:

Errors while testing the symlinks after building the toolchain;
Cannot compile my own applications using the toolchain (possibly related with the above issue);
Cannot build the bootloader.

In addition, it seems that the used partition table is tailored for a device with an external expansion flash of 8M. However, I am using a ESP32-S3-WROOM-1 N8R16, which as a 16M external expansion flash. It would be very useful to assign that extra space to the partition mounted on /etc, which is the only one with write permissions. Can you provide any pointers on how to do that?

Any help that you can provide to solve these issues would be greatly appreciated :)

Thank you in advance!

Kind regards, Joao Vieira

PS.: I am using Ubuntu 22.04 with GCC 11.4.0.

joaomiguelvieira commented 8 months ago

Thanks for your help! Everything works fine now!

I made a short summary of all the steps I have followed.

I hope this might help other enthusiasts!

Also, is there a way to read/write IO pins from user applications or the file system?

joaomiguelvieira commented 8 months ago

Thanks for your help!

I just realized that the changes you mentioned were made in a different branch of linux-xtensa (xtensa-6.6-rc5-esp32 instead of xtensa-6.5-esp32), so I tried to tell buildroot to fetch and compile the Linux kernel using that branch by modifying esp32s3_defconfig as follows:

@@ -14,7 +14,8 @@ BR2_ROOTFS_POST_IMAGE_SCRIPT="board/espressif/esp32s3/post-build.sh"
 BR2_LINUX_KERNEL=y
 BR2_LINUX_KERNEL_CUSTOM_GIT=y
 BR2_LINUX_KERNEL_CUSTOM_REPO_URL="https://github.com/jcmvbkbc/linux-xtensa.git"
-BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="xtensa-6.5-esp32-acm"
+BR2_LINUX_KERNEL_CUSTOM_REPO_VERSION="xtensa-6.6-rc5-esp32"

Furthermore, I have added CONFIG_GPIO_SYSFS=y to linux.config and deleted 0001-WIP-tools-perf-make-work-on-noMMU.patch (I have got an error saying that a more recent patch had already been applied).

I have also adapted the partition table and expanded the Linux partition, since the newly compiled kernel required more space than the previous one.

Everything compiles fine and I am able to obtain both the new kernel and the root filesystem.

However, I get kernel panic while booting the device:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x1064
load:0x403c9700,len:0x9e0
load:0x403cc700,len:0x291c
entry 0x403c98d8
linux ptr = 0x42120000
vectors ptr = 0x4037c000
[    0.000000] Linux version 6.6.0-rc5 (joaovieira@canberra.joaomiguelvieira.com) (xtensa-esp32s3-linux-uclibcfdpic-gcc (crosstool-NG 1.25.0.172_c29b568) 14.0.0 20230419 (experimental), GNU ld (crosstool-NG 1.25.0.172_c29b568) 2.40.50.20230424) #1 PREEMPT Tue Oct 17 11:06:3
[    0.000000] config ID: c2f0fffe:23090f1f
[    0.000000] earlycon: esp32s3uart0 at MMIO32 0x60000000 (options '115200n8,40000000')
[    0.000000] printk: bootconsole [esp32s3uart0] enabled
[    0.000000] **********************************************************
[    0.000000] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.000000] **                                                      **
[    0.000000] ** This system shows unhashed kernel memory addresses   **
[    0.000000] ** via the console, logs, and other interfaces. This    **
[    0.000000] ** might reduce the security of your system.            **
[    0.000000] **                                                      **
[    0.000000] ** If you see this message and you are not debugging    **
[    0.000000] ** the kernel, report this immediately to your system   **
[    0.000000] ** administrator!                                       **
[    0.000000] **                                                      **
[    0.000000] **   NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE   **
[    0.000000] **********************************************************
[    0.000000] Zone ranges:
[    0.000000]   Normal   [mem 0x000000003d800000-0x000000003dffffff]
[    0.000000] Movable zone start for each node
[    0.000000] Early memory node ranges
[    0.000000]   node   0: [mem 0x000000003d800000-0x000000003dffffff]
[    0.000000] Initmem setup node 0 [mem 0x000000003d800000-0x000000003dffffff]
[    0.000000] pcpu-alloc: s0 r0 d32768 u32768 alloc=1*32768
[    0.000000] pcpu-alloc: [0] 0 
[    0.000000] Kernel command line: earlycon=esp32s3uart,mmio32,0x60000000,115200n8,40000000 console=ttyS0,115200n8 debug rw root=mtd:rootfs no_hash_pointers 
[    0.000000] Dentry cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.000000] Inode-cache hash table entries: 1024 (order: 0, 4096 bytes, linear)
[    0.000000] Built 1 zonelists, mobility grouping off.  Total pages: 2032
[    0.000000] mem auto-init: stack:off, heap alloc:off, heap free:off
[    0.000000] virtual kernel memory layout:
[    0.000000]     lowmem  : 0x3d800000 - 0x3e000000  (    8 MB)
[    0.000000]     .text   : 0x42120000 - 0x4239d05c  ( 2548 kB)
[    0.000000]     .rodata : 0x4239e000 - 0x42477000  (  868 kB)
[    0.000000]     .data   : 0x3d800000 - 0x3d882da0  (  523 kB)
[    0.000000]     .init   : 0x3d882da0 - 0x3d8862c8  (   13 kB)
[    0.000000]     .bss    : 0x3d8862c8 - 0x3d8c9600  (  268 kB)
[    0.000000] Memory: 7260K/8192K available (2548K kernel code, 523K rwdata, 868K rodata, 87K init, 268K bss, 932K reserved, 0K cma-reserved)
[    0.000000] SLUB: HWalign=16, Order=0-3, MinObjects=0, CPUs=1, Nodes=1
[    0.000000] rcu: Preemptible hierarchical RCU implementation.
[    0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 10 jiffies.
[    0.000000] NR_IRQS: 33
[    0.000000] rcu: srcu_init: Setting srcu_struct sizes based on contention.
[    0.000000] Calibrating CPU frequency 160.00 MHz
[    0.000000] clocksource: ccount: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 11945377789 ns
[    0.000077] sched_clock: 32 bits at 160MHz, resolution 6ns, wraps every 13421772796ns
[    0.010415] Unable to handle kernel paging request at virtual address 900003da
[    0.010415]  pc = 4223b8be, ra = 8223c0da
[    0.011087] Oops: sig: 11 [#1] PREEMPT
[    0.011389] CPU: 0 PID: 0 Comm: swapper Not tainted 6.6.0-rc5 #1
[    0.011680] a00: 8223c0da 3d801f30 900003da 3d87394c 3d8c2c6a 00000001 3d8d1c5c 00060700
[    0.012347] a08: 90000000 3d801f10 00000000 3d87394c 00000004 3dfe3584 00000003 3d800000
[    0.012955] pc: 4223b8be, ps: 00060310, depc: 0000001c, excvaddr: 900003da
[    0.013218] lbeg: 42392a00, lend: 42392a0b lcount: 00000000, sar: 00000002
[    0.013474] Stack:
[    0.013731] > 8247a75b 3d801f90 00000000 3d86d9f8
[    0.013977]   8247d288 3d801f70 424835e4 3dfe3580
[    0.014206]   3d8c2c6c 3d8c2c60 3d8c2c54 3d804078
[    0.014421]   8247a75e 3d801f90 00000000 3d86d9f8
[    0.014642]   3d8d09c8 3d806d08 3d8d09d0 3d8c36e8
[    0.014864]   423992d2 3d801fb0 3dfec150 3d88902c
[    0.015091]   3d86d938 3d8ac754 00000000 4248a03c
[    0.015304]   00000000 3d802000 3d8c9600 3d8c9600
[    0.015510]   00000000 4247a264 3fcb4430 00000000
[    0.015733]   3d88304c 3fcb4360 3fcb4848 3fc9cb58
[    0.015946]   00000000 3d802000 3d8c9600 3d8c9600
[    0.016145]   00000001 00000000 3d889010 0000007b
[    0.016334]   00000000 00000000 00000000 00000000
[    0.016421] Call Trace:
[    0.016525] Disabling lock debugging due to kernel taint
[    0.016723] Kernel panic - not syncing: Attempted to kill the idle task!

Am I doing something wrong?

I apologize in advance if I am missing something obvious since I do not have a lot of experience on this subject :confused:

joaomiguelvieira commented 8 months ago

Thanks!

I just experimented reading and writing from/to the GPIO pins and all seems to be working :smile:

Being able to run Linux in a microcontroller enables a lot of possibilities for interesting projects!

The only thing that may be bugging in the long run is that sometimes the system runs out of memory and fails to allocate new data structures, and it may take a while until the memory used by old (dead) processes is released by the OS. This may be particularly upsetting since important processes might just start dying and there may not be an easy way to recover from this.

Here's an example of this happening after using the system for a while:

~ # lsgpio
[  919.889932] nommu: Allocation of length 131072 from process 259 (lsgpio) failed
[  919.890881] Mem-Info:
[  919.891030] active_anon:0 inactive_anon:0 isolated_anon:0
[  919.891030]  active_file:3 inactive_file:0 isolated_file:0
[  919.891030]  unevictable:43 dirty:0 writeback:0
[  919.891030]  slab_reclaimable:190 slab_unreclaimable:808
[  919.891030]  mapped:0 shmem:0 pagetables:0
[  919.891030]  sec_pagetables:0 bounce:0
[  919.891030]  kernel_misc_reclaimable:0
[  919.891030]  free:350 free_pcp:0 free_cma:0
[  919.946389] Node 0 active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:0kB unevictable:172kB isolated(anon):0kB isolated(file):0kB mapped:0kB dirty:0kB writeback:0kB shmem:0kB writeback_tmp:0kB kernel_stack:304kB pagetables:0kB sec_pagetables:0kB all_unreclo
[  919.996074] Normal free:1432kB boost:0kB min:340kB low:424kB high:508kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:12kB inactive_file:0kB unevictable:172kB writepending:0kB present:8192kB managed:7336kB mlocked:0kB bounce:0kB free_pcp:0kB locaB
[  920.035882] lowmem_reserve[]: 0 0
[  920.036780] Normal: 18*4kB (U) 24*8kB (U) 25*16kB (U) 12*32kB (U) 6*64kB (U) 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 1432kB
[  920.041116] 48 total pagecache pages
[  920.065850] 2048 pages RAM
[  920.066541] 0 pages HighMem/MovableOnly
[  920.066783] 214 pages reserved
Segmentation fault

Anyhow, I don't see a way of preventing this from happening except being extra careful with memory management. It's a microcontroller, after all :smile:

Thanks again for your help and congratulations on this impressive work!

joaomiguelvieira commented 8 months ago

Thanks for the tips!

jcmvbkbc / esp32-linux-build

Building the toolchain and the bootloader #1