search

tomeshnet / mesh-orange

A small ramdisk system running modern Debian to support mesh networking on ARM single-board computers
https://tomeshnet.github.io/mesh-orange/
GNU General Public License v2.0
39 stars 7 forks source link

RPI 3B+ not booting after adding some packages #52

Open darkdrgn2k opened 6 years ago

darkdrgn2k commented 6 years ago

When using http://github.com/benhylau/mesh-router-builder to build the mesh-workshop profile the RPI 3B+ will no longer boots and complains about EOF on the ram disk. However using the same builder without the "mesh-workshop" extras (default profile) it boots fine (just like stock mesh-orange).

It seems the only difference between the two are package that are added.

I have included the error copied form serial console durring boot, and the difference between Default and Mesh-Workshop profiles.

 100.660858] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[  100.666492] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[  100.673149] of_cfs_init
[  100.675730] of_cfs_init: OK
[  100.679199] RAMDISK: lzma image found at block 0
[  100.706675] RAMDISK: EOF while reading compressed data
[  100.706683] unexpected EOF
[  100.716473] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[  100.724084] Please append a correct "root=" boot option; here are the available partitions:
[  100.732593] 0100            4096 ram0
[  100.732598]  (driver?)
[  100.738788] 0101            4096 ram1
[  100.738792]  (driver?)
[  100.744996] 0102            4096 ram2
[  100.745001]  (driver?)
[  100.751190] 0103            4096 ram3
[  100.751195]  (driver?)
[  100.757401] 0104            4096 ram4
[  100.757406]  (driver?)
[  100.763611] 0105            4096 ram5
[  100.763615]  (driver?)
[  100.768089] mmc1: new high speed SDIO card at address 0001
[  100.775376] 0106            4096 ram6
[  100.775381]  (driver?)
[  100.781587] 0107            4096 ram7
[  100.781592]  (driver?)
[  100.787780] 0108            4096 ram8
[  100.787784]  (driver?)
[  100.791254] usb 1-1: new high-speed USB device number 2 using dwc_otg
[  100.791370] Indeed it is in host mode hprt0 = 00001101
[  100.805737] 0109            4096 ram9
[  100.805742]  (driver?)
[  100.811948] 010a            4096 ram10
[  100.811952]  (driver?)
[  100.818229] 010b            4096 ram11
[  100.818233]  (driver?)
[  100.824527] 010c            4096 ram12
[  100.824532]  (driver?)
[  100.830808] 010d            4096 ram13
[  100.830812]  (driver?)
[  100.837113] 010e            4096 ram14
[  100.837117]  (driver?)
[  100.843410] 010f            4096 ram15
[  100.843418]  (driver?)
[  100.849702] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[  100.858091] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.14.34-v7+ #1110
[  100.864799] Hardware name: BCM2835
[  100.868275] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  100.876142] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
[  100.883570] [<807840a4>] (dump_stack) from [<8011dc34>] (panic+0xf0/0x274)
[  100.890555] [<8011dc34>] (panic) from [<80b014f8>] (mount_block_root+0x1e8/0x2b8)
[  100.898154] [<80b014f8>] (mount_block_root) from [<80b017e8>] (mount_root+0x12c/0x134)
[  100.906193] [<80b017e8>] (mount_root) from [<80b01990>] (prepare_namespace+0x1a0/0x1e8)
[  100.914320] [<80b01990>] (prepare_namespace) from [<80b010ac>] (kernel_init_freeable+0x2a8/0x2bc

Difference in build

vagrant@stretch:~/mesh-orange-images/mesh-orange/mesh-workshop/debian-config$ diff debian.stretch.multistrap ../../default/debian-config/debian.stretch.multistrap 
2c2
< bootstrap=Stretch Docker Local
---
> bootstrap=Stretch Local
17d16
< # TODO get selected mesh router from node-profiles
19,27d17
< 
< [Docker]
< # TODO add keyring
< source=https://download.docker.com/linux/debian
< suite=stretch
< omitdebsrc=true
< components=nightly
< 
< packages= docker-ce
vagrant@stretch:~/mesh-orange-images/mesh-orange/mesh-workshop/debian-config$ diff packages.txt  ../../default/debian-config/packages.txt 
5,14d4
< # common ssl certificates
< ca-certificates
< 
< # mdns
< avahi-daemon
< libnss-mdns
< 
< # networking tools
< curl
< iperf3
16,18d5
< netcat
< 
< # text editor
hamishcoleman commented 6 years ago

This looks like a classic case of the built image being too big for the RAM available.

Can you paste the entire boot log? The initrd unpacking and out-of-memory errors occur much earlier in the boot log.

What are the sizes of the built .img files in all your tested cases? (also, if possible, what are the sizes of the initrd files in all cases too)

I really dont know what all these other repos contain or are doing. Do you have a minimal testcase patch to the mesh-orange repo that replicates this issue?

darkdrgn2k commented 6 years ago

full boot log

[    0.000000] Booting Linux on physical CPU 0x0
[    0.000000] Linux version 4.14.34-v7+ (dc4@dc4-XPS13-9333) (gcc version 4.9.3 (crosstool-NG crosstool-ng-1.22.0-88-g8460611)) #1110 SMP Mon Apr 16 15:18:51 BST 2018
[    0.000000] CPU: ARMv7 Processor [410fd034] revision 4 (ARMv7), cr=10c5383d
[    0.000000] CPU: div instructions available: patching division code
[    0.000000] CPU: PIPT / VIPT nonaliasing data cache, VIPT aliasing instruction cache
[    0.000000] OF: fdt: Machine model: Raspberry Pi 3 Model B Plus Rev 1.3
[    0.000000] Memory policy: Data cache writealloc
[    0.000000] cma: Reserved 8 MiB at 0x3dc00000
[    0.000000] random: fast init done
[    0.000000] percpu: Embedded 17 pages/cpu @be59f000 s38720 r8192 d22720 u69632
[    0.000000] Built 1 zonelists, mobility grouping on.  Total pages: 253242
[    0.000000] Kernel command line: 8250.nr_uarts=1 bcm2708_fb.fbwidth=656 bcm2708_fb.fbheight=416 bcm2708_fb.fbdepth=16 bcm2708_fb.fbswap=1 vc_mem.mem_base=0x3f000000 vc_mem.mem_size=0x3f600000  dwc_otg.lpm_enable=0 console=ttyS0,115200 console=ttyS0,115200
[    0.000000] PID hash table entries: 4096 (order: 2, 16384 bytes)
[    0.000000] Dentry cache hash table entries: 131072 (order: 7, 524288 bytes)
[    0.000000] Inode-cache hash table entries: 65536 (order: 6, 262144 bytes)
[    0.000000] Memory: 881252K/1021952K available (7168K kernel code, 575K rwdata, 2072K rodata, 1024K init, 706K bss, 132508K reserved, 8192K cma-reserved)
[    0.000000] Virtual kernel memory layout:
[    0.000000]     vector  : 0xffff0000 - 0xffff1000   (   4 kB)
[    0.000000]     fixmap  : 0xffc00000 - 0xfff00000   (3072 kB)
[    0.000000]     vmalloc : 0xbe800000 - 0xff800000   (1040 MB)
[    0.000000]     lowmem  : 0x80000000 - 0xbe600000   ( 998 MB)
[    0.000000]     modules : 0x7f000000 - 0x80000000   (  16 MB)
[    0.000000]       .text : 0x80008000 - 0x80800000   (8160 kB)
[    0.000000]       .init : 0x80b00000 - 0x80c00000   (1024 kB)
[    0.000000]       .data : 0x80c00000 - 0x80c8fd4c   ( 576 kB)
[    0.000000]        .bss : 0x80c96f4c - 0x80d478b4   ( 707 kB)
[    0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1
[    0.000000] ftrace: allocating 25231 entries in 74 pages
[    0.000000] Hierarchical RCU implementation.
[    0.000000] NR_IRQS: 16, nr_irqs: 16, preallocated irqs: 16
[    0.000000] arch_timer: cp15 timer(s) running at 19.20MHz (phys).
[    0.000000] clocksource: arch_sys_counter: mask: 0xffffffffffffff max_cycles: 0x46d987e47, max_idle_ns: 440795202767 ns
[    0.000007] sched_clock: 56 bits at 19MHz, resolution 52ns, wraps every 4398046511078ns
[    0.000023] Switching to timer-based delay loop, resolution 52ns
[    0.000278] Console: colour dummy device 80x30
[    0.000314] Calibrating delay loop (skipped), value calculated using timer frequency.. 38.40 BogoMIPS (lpj=192000)
[    0.000333] pid_max: default: 32768 minimum: 301
[    0.000649] Mount-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.000668] Mountpoint-cache hash table entries: 2048 (order: 1, 8192 bytes)
[    0.001678] Disabling memory control group subsystem
[    0.001760] CPU: Testing write buffer coherency: ok
[    0.002162] CPU0: thread -1, cpu 0, socket 0, mpidr 80000000
[    0.002541] Setting up static identity map for 0x100000 - 0x10003c
[    0.002664] Hierarchical SRCU implementation.
[    0.003303] smp: Bringing up secondary CPUs ...
[    0.003988] CPU1: thread -1, cpu 1, socket 0, mpidr 80000001
[    0.004732] CPU2: thread -1, cpu 2, socket 0, mpidr 80000002
[    0.005468] CPU3: thread -1, cpu 3, socket 0, mpidr 80000003
[    0.005572] smp: Brought up 1 node, 4 CPUs
[    0.005597] SMP: Total of 4 processors activated (153.60 BogoMIPS).
[    0.005607] CPU: All CPU(s) started in HYP mode.
[    0.005615] CPU: Virtualization extensions available.
[    0.006392] devtmpfs: initialized
[    0.016506] VFP support v0.3: implementor 41 architecture 3 part 40 variant 3 rev 4
[    0.016738] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604462750000 ns
[    0.016761] futex hash table entries: 1024 (order: 4, 65536 bytes)
[    0.017329] pinctrl core: initialized pinctrl subsystem
[    0.018070] NET: Registered protocol family 16
[    0.020519] DMA: preallocated 1024 KiB pool for atomic coherent allocations
[    0.025044] hw-breakpoint: found 5 (+1 reserved) breakpoint and 4 watchpoint registers.
[    0.025056] hw-breakpoint: maximum watchpoint size is 8 bytes.
[    0.025261] Serial: AMBA PL011 UART driver
[    0.026880] bcm2835-mbox 3f00b880.mailbox: mailbox enabled
[    0.027343] uart-pl011 3f201000.serial: could not find pctldev for node /soc/gpio@7e200000/uart0_pins, deferring probe
[    0.058072] bcm2835-dma 3f007000.dma: DMA legacy API manager at be813000, dmachans=0x1
[    0.059472] SCSI subsystem initialized
[    0.059677] usbcore: registered new interface driver usbfs
[    0.059733] usbcore: registered new interface driver hub
[    0.059820] usbcore: registered new device driver usb
[    0.070080] raspberrypi-firmware soc:firmware: Attached to firmware from 2018-04-16 18:18
[    0.071217] clocksource: Switched to clocksource arch_sys_counter
[    0.148575] VFS: Disk quotas dquot_6.6.0
[    0.148666] VFS: Dquot-cache hash table entries: 1024 (order 0, 4096 bytes)
[    0.148856] FS-Cache: Loaded
[    0.149059] CacheFiles: Loaded
[    0.157876] NET: Registered protocol family 2
[    0.158602] TCP established hash table entries: 8192 (order: 3, 32768 bytes)
[    0.158717] TCP bind hash table entries: 8192 (order: 4, 65536 bytes)
[    0.158907] TCP: Hash tables configured (established 8192 bind 8192)
[    0.159040] UDP hash table entries: 512 (order: 2, 16384 bytes)
[    0.159090] UDP-Lite hash table entries: 512 (order: 2, 16384 bytes)
[    0.159326] NET: Registered protocol family 1
[    0.159772] RPC: Registered named UNIX socket transport module.
[    0.159783] RPC: Registered udp transport module.
[    0.159792] RPC: Registered tcp transport module.
[    0.159801] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.160020] Trying to unpack rootfs image as initramfs...
[   98.764075] rootfs image is not initramfs (write error); looks like an initrd
[   98.764198] /initrd.image: incomplete write (4096 != 112347770)
[   98.894058] Freeing initrd memory: 109716K
[   98.894958] hw perfevents: enabled with armv7_cortex_a7 PMU driver, 7 counters available
[   98.897812] workingset: timestamp_bits=14 max_order=18 bucket_order=4
[   98.905739] FS-Cache: Netfs 'nfs' registered for caching
[   98.906344] NFS: Registering the id_resolver key type
[   98.906379] Key type id_resolver registered
[   98.906389] Key type id_legacy registered
[   98.906409] nfs4filelayout_init: NFSv4 File Layout Driver Registering...
[   98.908281] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 251)
[   98.908410] io scheduler noop registered
[   98.908422] io scheduler deadline registered
[   98.908691] io scheduler cfq registered (default)
[   98.908703] io scheduler mq-deadline registered
[   98.908713] io scheduler kyber registered
[   98.911059] BCM2708FB: allocated DMA memory fdd00000
[   98.911091] BCM2708FB: allocated DMA channel 0 @ be813000
[   98.916038] Console: switching to colour frame buffer device 82x26
[   98.919468] Serial: 8250/16550 driver, 1 ports, IRQ sharing enabled
[   98.920752] bcm2835-rng 3f104000.rng: hwrng registered
[   98.920883] vc-mem: phys_addr:0x00000000 mem_base=0x3f000000 mem_size:0x3f600000(1014 MiB)
[   98.921354] vc-sm: Videocore shared memory driver
[   98.921634] gpiomem-bcm2835 3f200000.gpiomem: Initialised: Registers at 0x3f200000
[   98.931052] brd: module loaded
[   98.939578] loop: module loaded
[   98.939598] Loading iSCSI transport class v2.0-870.
[   98.940239] libphy: Fixed MDIO Bus: probed
[   98.940326] usbcore: registered new interface driver lan78xx
[   98.940385] usbcore: registered new interface driver smsc95xx
[   98.940404] dwc_otg: version 3.00a 10-AUG-2012 (platform bus)
[   99.168473] Core Release: 2.80a
[   99.168488] Setting default values for core params
[   99.168522] Finished setting default values for core params
[   99.368776] Using Buffer DMA mode
[   99.368788] Periodic Transfer Interrupt Enhancement - disabled
[   99.368797] Multiprocessor Interrupt Enhancement - disabled
[   99.368807] OTG VER PARAM: 0, OTG VER FLAG: 0
[   99.368820] Dedicated Tx FIFOs mode
[   99.369151] WARN::dwc_otg_hcd_init:1046: FIQ DMA bounce buffers: virt = 0xbdd14000 dma = 0xfdd14000 len=9024
[   99.369181] FIQ FSM acceleration enabled for :
[   99.369181] Non-periodic Split Transactions
[   99.369181] Periodic Split Transactions
[   99.369181] High-Speed Isochronous Endpoints
[   99.369181] Interrupt/Control Split Transaction hack enabled
[   99.369257] WARN::hcd_init_fiq:459: FIQ on core 1 at 0x805e6a40
[   99.369273] WARN::hcd_init_fiq:460: FIQ ASM at 0x805e6da8 length 36
[   99.369289] WARN::hcd_init_fiq:486: MPHI regs_base at 0xc007e000
[   99.369349] dwc_otg 3f980000.usb: DWC OTG Controller
[   99.369385] dwc_otg 3f980000.usb: new USB bus registered, assigned bus number 1
[   99.369420] dwc_otg 3f980000.usb: irq 62, io mem 0x00000000
[   99.369471] Init: Port Power? op_state=1
[   99.369480] Init: Power Port (0)
[   99.369684] usb usb1: New USB device found, idVendor=1d6b, idProduct=0002
[   99.369700] usb usb1: New USB device strings: Mfr=3, Product=2, SerialNumber=1
[   99.369713] usb usb1: Product: DWC OTG Controller
[   99.369725] usb usb1: Manufacturer: Linux 4.14.34-v7+ dwc_otg_hcd
[   99.369737] usb usb1: SerialNumber: 3f980000.usb
[   99.370326] hub 1-0:1.0: USB hub found
[   99.370368] hub 1-0:1.0: 1 port detected
[   99.371133] usbcore: registered new interface driver usb-storage
[   99.371333] mousedev: PS/2 mouse device common for all mice
[   99.371404] IR NEC protocol handler initialized
[   99.371414] IR RC5(x/sz) protocol handler initialized
[   99.371423] IR RC6 protocol handler initialized
[   99.371431] IR JVC protocol handler initialized
[   99.371440] IR Sony protocol handler initialized
[   99.371449] IR SANYO protocol handler initialized
[   99.371457] IR Sharp protocol handler initialized
[   99.371466] IR MCE Keyboard/mouse protocol handler initialized
[   99.371475] IR XMP protocol handler initialized
[   99.372118] bcm2835-wdt 3f100000.watchdog: Broadcom BCM2835 watchdog timer
[   99.372367] bcm2835-cpufreq: min=600000 max=1400000
[   99.372713] sdhci: Secure Digital Host Controller Interface driver
[   99.372723] sdhci: Copyright(c) Pierre Ossman
[   99.373065] mmc-bcm2835 3f300000.mmc: could not get clk, deferring probe
[   99.373375] sdhost-bcm2835 3f202000.mmc: could not get clk, deferring probe
[   99.373472] sdhci-pltfm: SDHCI platform and OF driver helper
[   99.374859] ledtrig-cpu: registered to indicate activity on CPUs
[   99.375002] hidraw: raw HID events driver (C) Jiri Kosina
[   99.375152] usbcore: registered new interface driver usbhid
[   99.375162] usbhid: USB HID core driver
[   99.375733] vchiq: vchiq_init_state: slot_zero = bdd80000, is_master = 0
[   99.377010] [vc_sm_connected_init]: start
[   99.380450] vc_vchi_sm_init: failed to open VCHI service (-1)
[   99.380458] [vc_sm_connected_init]: failed to initialize shared memory service
[   99.380476] [vc_sm_connected_init]: end - returning -1
[   99.381049] Initializing XFRM netlink socket
[   99.381076] NET: Registered protocol family 17
[   99.381175] Key type dns_resolver registered
[   99.381471] Registering SWP/SWPB emulation handler
[   99.382045] registered taskstats version 1
[   99.387847] uart-pl011 3f201000.serial: cts_event_workaround enabled
[   99.387924] 3f201000.serial: ttyAMA0 at MMIO 0x3f201000 (irq = 87, base_baud = 0) is a PL011 rev2
[   99.389519] console [ttyS0] disabled
[   99.389553] 3f215040.serial: ttyS0 at MMIO 0x0 (irq = 166, base_baud = 31250000) is a 16550
[  100.457480] console [ttyS0] enabled
[  100.461970] mmc-bcm2835 3f300000.mmc: mmc_debug:0 mmc_debug2:0
[  100.467891] mmc-bcm2835 3f300000.mmc: DMA channel allocated
[  100.531853] sdhost: log_buf @ bdd13000 (fdd13000)
[  100.571318] Indeed it is in host mode hprt0 = 00021501
[  100.637206] mmc1: queuing unknown CIS tuple 0x80 (2 bytes)
[  100.644381] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[  100.651544] mmc1: queuing unknown CIS tuple 0x80 (3 bytes)
[  100.659935] mmc1: queuing unknown CIS tuple 0x80 (7 bytes)
[  100.671246] mmc0: sdhost-bcm2835 loaded - DMA enabled (>1)
[  100.677896] of_cfs_init
[  100.680476] of_cfs_init: OK
[  100.683983] RAMDISK: lzma image found at block 0
[  100.694446] RAMDISK: EOF while reading compressed data
[  100.694453] unexpected EOF
[  100.723031] VFS: Cannot open root device "(null)" or unknown-block(0,0): error -6
[  100.730624] Please append a correct "root=" boot option; here are the available partitions:
[  100.739128] 0100            4096 ram0
[  100.739133]  (driver?)
[  100.745348] 0101            4096 ram1
[  100.745353]  (driver?)
[  100.751556] 0102            4096 ram2
[  100.751560]  (driver?)
[  100.757749] 0103            4096 ram3
[  100.757753]  (driver?)
[  100.763956] 0104            4096 ram4
[  100.763961]  (driver?)
[  100.770149] 0105            4096 ram5
[  100.770153]  (driver?)
[  100.776355] 0106            4096 ram6
[  100.776360]  (driver?)
[  100.780763] mmc1: new high speed SDIO card at address 0001
[  100.788121] 0107            4096 ram7
[  100.788125]  (driver?)
[  100.791261] usb 1-1: new high-speed USB device number 2 using dwc_otg
[  100.791378] Indeed it is in host mode hprt0 = 00001101
[  100.806075] 0108            4096 ram8
[  100.806080]  (driver?)
[  100.812285] 0109            4096 ram9
[  100.812290]  (driver?)
[  100.818479] 010a            4096 ram10
[  100.818484]  (driver?)
[  100.824777] 010b            4096 ram11
[  100.824782]  (driver?)
[  100.831057] 010c            4096 ram12
[  100.831061]  (driver?)
[  100.837351] 010d            4096 ram13
[  100.837356]  (driver?)
[  100.843646] 010e            4096 ram14
[  100.843651]  (driver?)
[  100.849927] 010f            4096 ram15
[  100.849932]  (driver?)
[  100.856236] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[  100.864626] CPU: 2 PID: 1 Comm: swapper/0 Not tainted 4.14.34-v7+ #1110
[  100.871334] Hardware name: BCM2835
[  100.874811] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  100.882680] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
[  100.890107] [<807840a4>] (dump_stack) from [<8011dc34>] (panic+0xf0/0x274)
[  100.890725] mmc0: host does not support reading read-only switch, assuming write-enable
[  100.894973] mmc0: new high speed SDHC card at address e624
[  100.895684] mmcblk0: mmc0:e624 SL08G 7.40 GiB
[  100.915176] [<8011dc34>] (panic) from [<80b014f8>] (mount_block_root+0x1e8/0x2b8)
[  100.922775] [<80b014f8>] (mount_block_root) from [<80b017e8>] (mount_root+0x12c/0x134)
[  100.930814] [<80b017e8>] (mount_root) from [<80b01990>] (prepare_namespace+0x1a0/0x1e8)
[  100.938941] [<80b01990>] (prepare_namespace) from [<80b010ac>] (kernel_init_freeable+0x2a8/0x2bc)
[  100.947953] [<80b010ac>] (kernel_init_freeable) from [<807991a0>] (kernel_init+0x18/0x124)
[  100.956348] [<807991a0>] (kernel_init) from [<8010810c>] (ret_from_fork+0x14/0x28)
[  100.964039] CPU0: stopping
[  100.966790] CPU: 0 PID: 0 Comm: swapper/0 Not tainted 4.14.34-v7+ #1110
[  100.973499] Hardware name: BCM2835
[  100.976963] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  100.984828] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
[  100.992254] [<807840a4>] (dump_stack) from [<8010e458>] (handle_IPI+0x2bc/0x33c)
[  100.999767] [<8010e458>] (handle_IPI) from [<801014d8>] (bcm2836_arm_irqchip_handle_irq+0x7c/0xac)
[  101.008866] [<801014d8>] (bcm2836_arm_irqchip_handle_irq) from [<8079fcbc>] (__irq_svc+0x5c/0x7c)
[  101.017868] Exception stack(0x80c01ef0 to 0x80c01f38)
[  101.022995] 1ee0:                                     00000000 00010980 3da19000 00000000
[  101.031297] 1f00: 80c00000 80c03dcc 80c03d68 80c88172 00000001 80b60a30 be5ffb00 80c01f4c
[  101.039597] 1f20: 80c04174 80c01f40 80108a4c 80108a50 60000013 ffffffff
[  101.046318] [<8079fcbc>] (__irq_svc) from [<80108a50>] (arch_cpu_idle+0x34/0x4c)
[  101.053831] [<80108a50>] (arch_cpu_idle) from [<8079f434>] (default_idle_call+0x34/0x48)
[  101.062050] [<8079f434>] (default_idle_call) from [<801611cc>] (do_idle+0xd8/0x150)
[  101.069826] [<801611cc>] (do_idle) from [<801614e0>] (cpu_startup_entry+0x28/0x2c)
[  101.077513] [<801614e0>] (cpu_startup_entry) from [<80799184>] (rest_init+0xbc/0xc0)
[  101.085375] [<80799184>] (rest_init) from [<80b00df8>] (start_kernel+0x3d4/0x3e0)
[  101.092974] [<80b00df8>] (start_kernel) from [<0000807c>] (0x807c)
[  101.099245] CPU1: stopping
[  101.101994] CPU: 1 PID: 0 Comm: swapper/1 Not tainted 4.14.34-v7+ #1110
[  101.108703] Hardware name: BCM2835
[  101.112162] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  101.120027] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
[  101.127452] [<807840a4>] (dump_stack) from [<8010e458>] (handle_IPI+0x2bc/0x33c)
[  101.134963] [<8010e458>] (handle_IPI) from [<801014d8>] (bcm2836_arm_irqchip_handle_irq+0x7c/0xac)
[  101.144062] [<801014d8>] (bcm2836_arm_irqchip_handle_irq) from [<8079fcbc>] (__irq_svc+0x5c/0x7c)
[  101.153063] Exception stack(0xbcd25f38 to 0xbcd25f80)
[  101.158186] 5f20:                                                       00000000 000088f4
[  101.166488] 5f40: 3da2a000 00000000 bcd24000 80c03dcc 80c03d68 80c88172 00000001 410fd034
[  101.174790] 5f60: 00000000 bcd25f94 80c04174 bcd25f88 80108a4c 80108a50 60000013 ffffffff
[  101.183096] [<8079fcbc>] (__irq_svc) from [<80108a50>] (arch_cpu_idle+0x34/0x4c)
[  101.190609] [<80108a50>] (arch_cpu_idle) from [<8079f434>] (default_idle_call+0x34/0x48)
[  101.198826] [<8079f434>] (default_idle_call) from [<801611cc>] (do_idle+0xd8/0x150)
[  101.206601] [<801611cc>] (do_idle) from [<801614e0>] (cpu_startup_entry+0x28/0x2c)
[  101.214288] [<801614e0>] (cpu_startup_entry) from [<8010df50>] (secondary_start_kernel+0x130/0x13c)
[  101.223472] [<8010df50>] (secondary_start_kernel) from [<0010196c>] (0x10196c)
[  101.230800] CPU3: stopping
[  101.233551] CPU: 3 PID: 0 Comm: swapper/3 Not tainted 4.14.34-v7+ #1110
[  101.240260] Hardware name: BCM2835
[  101.243720] [<8010ffd8>] (unwind_backtrace) from [<8010c240>] (show_stack+0x20/0x24)
[  101.251585] [<8010c240>] (show_stack) from [<807840a4>] (dump_stack+0xd4/0x118)
[  101.259009] [<807840a4>] (dump_stack) from [<8010e458>] (handle_IPI+0x2bc/0x33c)
[  101.266520] [<8010e458>] (handle_IPI) from [<801014d8>] (bcm2836_arm_irqchip_handle_irq+0x7c/0xac)
[  101.275619] [<801014d8>] (bcm2836_arm_irqchip_handle_irq) from [<8079fcbc>] (__irq_svc+0x5c/0x7c)
[  101.284620] Exception stack(0xbcd29f38 to 0xbcd29f80)
[  101.289744] 9f20:                                                       00000000 000035f0
[  101.298047] 9f40: 3da4c000 00000000 bcd28000 80c03dcc 80c03d68 80c88172 00000001 410fd034
[  101.306349] 9f60: 00000000 bcd29f94 80c04174 bcd29f88 80108a4c 80108a50 60000013 ffffffff
[  101.314655] [<8079fcbc>] (__irq_svc) from [<80108a50>] (arch_cpu_idle+0x34/0x4c)
[  101.322168] [<80108a50>] (arch_cpu_idle) from [<8079f434>] (default_idle_call+0x34/0x48)
[  101.330384] [<8079f434>] (default_idle_call) from [<801611cc>] (do_idle+0xd8/0x150)
[  101.338159] [<801611cc>] (do_idle) from [<801614e0>] (cpu_startup_entry+0x28/0x2c)
[  101.345846] [<801614e0>] (cpu_startup_entry) from [<8010df50>] (secondary_start_kernel+0x130/0x13c)
[  101.355028] [<8010df50>] (secondary_start_kernel) from [<0010196c>] (0x10196c)
[  101.362368] ---[ end Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

ls of sd card

total 115540
drwxr-xr-x   3 user user     16384 Dec 31  1969 .
drwxr-xr-x 107 user user      4096 Jun 17 08:17 ..
-rw-r--r--   1 user user     23044 Jun 15 19:54 bcm2709-rpi-2-b.dtb
-rw-r--r--   1 user user     24240 Jun 15 19:54 bcm2710-rpi-3-b.dtb
-rw-r--r--   1 user user     24503 Jun 15 19:54 bcm2710-rpi-3-b-plus.dtb
-rw-r--r--   1 user user     22952 Jun 15 19:54 bcm2710-rpi-cm3.dtb
-rw-r--r--   1 user user     52064 Jun 15 19:54 bootcode.bin
-rw-r--r--   1 user user        65 Jun 15 19:54 cmdline.txt
drwxr-xr-x   2 user user     16384 Jun 15 19:54 conf.d
-rw-r--r--   1 user user        66 Jun 15 19:54 config.txt
-rw-r--r--   1 user user     18693 Jun 15 19:54 COPYING.linux
-rw-r--r--   1 user user      2599 Jun 15 19:54 fixup_cd.dat
-rw-r--r--   1 user user 112347770 Jun 15 19:54 initrd
-rw-r--r--   1 user user   4922144 Jun 15 19:54 kernel7.img
-rw-r--r--   1 user user      1494 Jun 15 19:54 LICENCE.broadcom
-rw-r--r--   1 user user    673444 Jun 15 19:54 start_cd.elf
hamishcoleman commented 6 years ago

That log pretty much confirms the theory that the image is too large - these log lines:

[    0.160020] Trying to unpack rootfs image as initramfs...
[   98.764075] rootfs image is not initramfs (write error); looks like an initrd
[   98.764198] /initrd.image: incomplete write (4096 != 112347770)

I think can occur for other reasons, but I have only seen them when there is too much to fit in RAM.

What are the sizes of the uncompressed debian images for all your tested builds? ("ls -l debian/build/debian.stretch.armhf.cpio" should show this)

darkdrgn2k commented 6 years ago 
vagrant@stretch:/tmp/mesh-orange$ ls -l debian/build/debian.stretch.armhf.cpio 
-rw-r--r-- 1 vagrant vagrant 307975680 Jun 16 04:10 debian/build/debian.stretch.armhf.cpio

The raspberry Pi 3b+ has 1 gig of ram We ran these on the RPI3B boards for the workshop. I currently do not have a working 3B board to test with (@benhylau does it boot on the RPI3B?)

benhylau commented 6 years ago

On the Raspberry Pi 3B

The default profile image that works:

vagrant@stretch:~$ sudo mount -o offset=$((512*2048)) -t vfat /vagrant/output/mesh-orange-images/raspberrypi3-0.8_default.img /mnt
vagrant@stretch:~$ ls -lh /mnt
total 73M
-rwxr-xr-x 1 root root  23K Jun 17 18:56 bcm2709-rpi-2-b.dtb
-rwxr-xr-x 1 root root  24K Jun 17 18:56 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root  24K Jun 17 18:56 bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root  23K Jun 17 18:56 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root  51K Jun 17 18:56 bootcode.bin
-rwxr-xr-x 1 root root   65 Jun 17 18:56 cmdline.txt
drwxr-xr-x 2 root root  16K Jun 17 18:56 conf.d
-rwxr-xr-x 1 root root   66 Jun 17 18:56 config.txt
-rwxr-xr-x 1 root root  19K Jun 17 18:56 COPYING.linux
-rwxr-xr-x 1 root root 2.6K Jun 17 18:56 fixup_cd.dat
-rwxr-xr-x 1 root root  68M Jun 17 18:56 initrd
-rwxr-xr-x 1 root root 4.7M Jun 17 18:56 kernel7.img
-rwxr-xr-x 1 root root 1.5K Jun 17 18:56 LICENCE.broadcom
-rwxr-xr-x 1 root root 658K Jun 17 18:56 start_cd.elf

The mesh-workshop profile image that fails:

vagrant@stretch:~$ sudo mount -o offset=$((512*2048)) -t vfat /vagrant/output/mesh-orange-images/raspberrypi3-0.8_mesh-workshop.img /mnt
vagrant@stretch:~$ ls -lh /mnt
total 114M
-rwxr-xr-x 1 root root  23K Jun 13 05:26 bcm2709-rpi-2-b.dtb
-rwxr-xr-x 1 root root  24K Jun 13 05:26 bcm2710-rpi-3-b.dtb
-rwxr-xr-x 1 root root  24K Jun 13 05:26 bcm2710-rpi-3-b-plus.dtb
-rwxr-xr-x 1 root root  23K Jun 13 05:26 bcm2710-rpi-cm3.dtb
-rwxr-xr-x 1 root root  51K Jun 13 05:26 bootcode.bin
-rwxr-xr-x 1 root root   65 Jun 13 05:26 cmdline.txt
drwxr-xr-x 2 root root  16K Jun 13 05:26 conf.d
-rwxr-xr-x 1 root root   66 Jun 13 05:26 config.txt
-rwxr-xr-x 1 root root  19K Jun 13 05:26 COPYING.linux
-rwxr-xr-x 1 root root 2.6K Jun 13 05:26 fixup_cd.dat
-rwxr-xr-x 1 root root 109M Jun 13 05:26 initrd
-rwxr-xr-x 1 root root 4.7M Jun 13 05:26 kernel7.img
-rwxr-xr-x 1 root root 1.5K Jun 13 05:26 LICENCE.broadcom
-rwxr-xr-x 1 root root 658K Jun 13 05:26 start_cd.elf

mesh-workshop.log

darkdrgn2k commented 6 years ago

Looks like its a size issue

any idea why since there seems to be more ram then needed?

@benhylau any packages we can take out to see if it will work

benhylau commented 6 years ago

Want to add that the previous version that worked (with both build profiles) have the following initrd sizes:

default -> 60M Apr 26 05:52 initrd mesh-workshop -> 98M Apr 26 06:48 initrd

So it is the additional 11M (from 98M to 109M), probably from ff3e59dc916b550b49c0531ff6198d6eac26f7fd and 1b762ae886af758d2137dce2c798de2a90edacf6. @hamishcoleman do these numbers look significant in any way?

hamishcoleman commented 6 years ago

Firstly, I've led you astray by asking for the image size or the initrd size - they are going to be largely secondary. The important size is the uncompressed sizes of the files going into the initrd. This is the debian/build/debian.stretch.armhf.cpio file. (There are also the kernel modules and firmware cpio files, but they will not be changing sizes when you add/remove packages)

So, adding up all the numbers: modules-armhf.cpio = 31M (size from a default build I just did) firmware-raspbian.cpio = 29M (size from a default build I just did) debian.stretch.armhf.cpio = 307M (size reported by darkdrgn2k above)

total = ~370M

From the boot log:

[ 0.000000] Memory: 881252K/1021952K available (7168K kernel code, 575K rwdata, 2072K rodata, 1024K init, 706K bss, 132508K reserved, 8192K cma-reserved)

Telling us that there is ~880M of ram and thus ~440M is available for the initramfs. So, there /should/ be enough ram to unpack the image, but remember that my target size for the initramfs was "under 100M", so this is already three times the expected size. There could easily be another expansion factor somewhere that consumes the 70Meg of space from the above calculations (EG: are there lots of small files? does initramfs have a minimum sector size? does the slack space therefore add up?)

darkdrgn2k commented 6 years ago

Sounds like to me like we have a few options

Also i THINK there is a way to set the amount of ram the GPU takes in the pi. Ill try to look through the code (raspi-config) and see if it maybe is just a boot time argument.

hamishcoleman commented 6 years ago

Before you go reading the raspi-config source, have a look at boards/raspberrypi2/config.txt - the line is "gpu_mem=16" (ref https://www.raspberrypi.org/documentation/configuration/config-txt/)

benhylau commented 6 years ago

removing packages sounds like a win

I need all of them in the workshop image, so that's not really an option unfortunately.

reducing the base image is a simple process that has not been done for a long time now... but the gains will be far outweighed by the increase from adding all these new packages

Agree.

darkdrgn2k commented 6 years ago

Confirmed this is a lack of space issue. Stripped out most of the extra packages and finally got it to boot with a 24mb "free space"

Possibly two sets of kernel modules installed?

/lib/modules/4.14.34+/
/lib/modules/4.14.34-v7+/
hamishcoleman commented 6 years ago

The raspberry pi modules package does include modules for both the rpi hardware kernel types, however removing one of them will only buy you 50Meg - the amount that these packages increase the size of things is four times that.

I've said it before, but I will say it again now, I dont think that installing all these packages is a good idea. Perhaps you should install all these packages ontop of a different base system? All the troubles you have been getting stem from the scaling up of something that had a target size of under 120Meg to be three times that size.