Failing to build `system_ext.img` - `system_ext` is too big

[petefoth]

Building with WITH_GMS=true fails to build system_ext.img

• The max image size for filesystem files is 408571904 bytes (389 MB), out of a total partition size of 408571904 bytes (389 MB)
• du -h out/target/product/sunfish/system_ext/ gives 399M

Error building `lemonadep`

``` [ 98% 181644/184025] Target system_ext fs image: out/target/product/lemonadep/system_ext.img (priority: 8) FAILED: out/target/product/lemonadep/system_ext.img /bin/bash -c "(mkdir -p out/target/product/lemonadep/system_ext ) && (mkdir -p out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates && rm -rf out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_fs_type=ext4\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_reserved_size=8388609\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_disable_sparse=true\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_selinux_fc=out/target/product/lemonadep/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"building_system_ext_image=true\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"ext_mkuserimg=mkuserimg_mke2fs\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"fs_type=ext4\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"extfs_sparse_flag=-s\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"erofs_sparse_flag=-s\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"squashfs_sparse_flag=-s\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"f2fs_sparse_flag=-S\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"erofs_default_compressor=\"lz4hc,9\"\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_avbtool=avbtool\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_system_ext_hashtree_enable=true\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_system_ext_add_hashtree_footer_args=--prop com.android.build.system_ext.os_version:14 --prop com.android.build.system_ext.fingerprint:\$(cat out/target/product/lemonadep/build_fingerprint.txt) --prop com.android.build.system_ext.security_patch:2024-04-05\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"root_dir=out/target/product/lemonadep/root\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"use_dynamic_partition_size=true\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"skip_fsck=true\" >> out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (sort -o out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (PATH=out/host/linux-x86/bin/:system/extras/ext4_utils/:\$PATH out/host/linux-x86/bin/build_image out/target/product/lemonadep/system_ext out/target/product/lemonadep/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt out/target/product/lemonadep/system_ext.img out/target/product/lemonadep/system ) && (true )" 2024-04-24 23:25:07 - build_image.py - ERROR : Failed to build out/target/product/lemonadep/system_ext.img from out/target/product/lemonadep/system_ext Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/srv/src/LINEAGE_21_0/out/host/linux-x86/bin/build_image/__main__.py", line 12, in File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "build_image.py", line 1038, in File "build_image.py", line 1033, in main File "build_image.py", line 937, in BuildImageOrVBMeta File "build_image.py", line 581, in BuildImage File "build_image.py", line 447, in BuildImageMkfs File "common.py", line 328, in RunAndCheckOutput common.ExternalError: Failed to run command '['mkuserimg_mke2fs', 'out/target/product/lemonadep/system_ext', 'out/target/product/lemonadep/system_ext.img', 'ext4', 'system_ext', '406761472', '-j', '0', '-D', 'out/target/product/lemonadep/system', '-L', 'system_ext', '-i', '3666', '-M', '0', '-U', '12be0001-b2c4-5242-85c9-2e3f32645118', '-S', '5959b069-7333-5a49-9458-c4fa53731533', '--inode_size', '256', 'out/target/product/lemonadep/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin']' (exit code 4): 23:25:06 mkuserimg_mke2fs.py INFO: Env: {'MKE2FS_CONFIG': '/srv/src/LINEAGE_21_0/out/soong/.temp/tmp2zat1x3t'} 23:25:06 mkuserimg_mke2fs.py INFO: Running: /srv/src/LINEAGE_21_0/out/host/linux-x86/bin/mke2fs -O ^has_journal -L system_ext -N 3666 -I 256 -M /system_ext -m 0 -U 12be0001-b2c4-5242-85c9-2e3f32645118 -E hash_seed=5959b069-7333-5a49-9458-c4fa53731533 -t ext4 -b 4096 out/target/product/lemonadep/system_ext.img 99307 23:25:07 mkuserimg_mke2fs.py INFO: Env: {} 23:25:07 mkuserimg_mke2fs.py INFO: Running: /srv/src/LINEAGE_21_0/out/host/linux-x86/bin/e2fsdroid -e -p out/target/product/lemonadep/system -S out/target/product/lemonadep/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin -f out/target/product/lemonadep/system_ext -a /system_ext out/target/product/lemonadep/system_ext.img 23:25:07 mkuserimg_mke2fs.py ERROR: Failed to run e2fsdroid_cmd: __populate_fs: Could not allocate block in ext2 filesystem while writing file "spup.vim" e2fsdroid: Could not allocate block in ext2 filesystem while populating file system mke2fs 1.46.6 (1-Feb-2023) Creating filesystem with 99307 4k blocks and 3712 inodes Filesystem UUID: 12be0001-b2c4-5242-85c9-2e3f32645118 Superblock backups stored on blocks: 32768, 98304 Allocating group tables: 0/4 done Writing inode tables: 0/4 done Writing superblocks and filesystem accounting information: 0/4 done __populate_fs: Could not allocate block in ext2 filesystem while writing file "spup.vim" e2fsdroid: Could not allocate block in ext2 filesystem while populating file system Out of space? Out of inodes? The tree size of out/target/product/lemonadep/system_ext is 398370816 bytes (379 MB), with reserved space of 8388609 bytes (8 MB). The max image size for filesystem files is 406761472 bytes (387 MB), out of a total partition size of 406761472 bytes (387 MB). [ 98% 181645/184025] compile out/host/linux-x86/bin/go/androidmk-lib/pkg/android/soong/androidmk/androidmk.a (priority: 8) ```

Same error occurs building sunfish. I tried changing value of BOARD_SYSTEM_EXTIMAGE_PARTITION_RESERVED_SIZE from 8388609 to 4194305 in file vendor/lineage/config/BoardConfigReservedSize.mk, but still failed

Error building `sunfish`

``` [ 79% 319/401] Target system_ext fs image: out/target/product/sunfish/system_ext.img FAILED: out/target/product/sunfish/system_ext.img /bin/bash -c "(mkdir -p out/target/product/sunfish/system_ext ) && (mkdir -p out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates && rm -rf out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_fs_type=ext4\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_reserved_size=4194305\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_disable_sparse=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"system_ext_selinux_fc=out/target/product/sunfish/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"building_system_ext_image=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"ext_mkuserimg=mkuserimg_mke2fs\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"fs_type=ext4\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"extfs_sparse_flag=-s\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"erofs_sparse_flag=-s\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"squashfs_sparse_flag=-s\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"f2fs_sparse_flag=-S\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_avbtool=avbtool\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_system_ext_hashtree_enable=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"avb_system_ext_add_hashtree_footer_args=--hash_algorithm sha256 --prop com.android.build.system_ext.os_version:14 --prop com.android.build.system_ext.fingerprint:\$(cat out/target/product/sunfish/build_fingerprint.txt) --prop com.android.build.system_ext.security_patch:2024-04-05\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"recovery_as_boot=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"root_dir=out/target/product/sunfish/root\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"use_dynamic_partition_size=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (echo \"skip_fsck=true\" >> out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (sort -o out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt ) && (PATH=out/host/linux-x86/bin/:system/extras/ext4_utils/:\$PATH out/host/linux-x86/bin/build_image out/target/product/sunfish/system_ext out/target/product/sunfish/obj/PACKAGING/system_ext_intermediates/system_ext_image_info.txt out/target/product/sunfish/system_ext.img out/target/product/sunfish/system ) && (true )" 2024-04-25 19:18:47 - build_image.py - ERROR : Failed to build out/target/product/sunfish/system_ext.img from out/target/product/sunfish/system_ext Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/srv/src/LINEAGE_21_0/out/host/linux-x86/bin/build_image/__main__.py", line 12, in File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "build_image.py", line 1038, in File "build_image.py", line 1033, in main File "build_image.py", line 937, in BuildImageOrVBMeta File "build_image.py", line 581, in BuildImage File "build_image.py", line 447, in BuildImageMkfs File "common.py", line 328, in RunAndCheckOutput common.ExternalError: Failed to run command '['mkuserimg_mke2fs', 'out/target/product/sunfish/system_ext', 'out/target/product/sunfish/system_ext.img', 'ext4', 'system_ext', '412766208', '-j', '0', '-D', 'out/target/product/sunfish/system', '-L', 'system_ext', '-i', '3628', '-M', '0', '-U', '3f03299a-4175-53fd-9041-1f6ac7a6973b', '-S', 'eca82a74-417a-5468-9d9e-86beef6b387a', '--inode_size', '256', 'out/target/product/sunfish/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin']' (exit code 4): 19:18:33 mkuserimg_mke2fs.py INFO: Env: {'MKE2FS_CONFIG': '/srv/src/LINEAGE_21_0/out/soong/.temp/tmpbql7ara7'} 19:18:33 mkuserimg_mke2fs.py INFO: Running: /srv/src/LINEAGE_21_0/out/host/linux-x86/bin/mke2fs -O ^has_journal -L system_ext -N 3628 -I 256 -M /system_ext -m 0 -U 3f03299a-4175-53fd-9041-1f6ac7a6973b -E hash_seed=eca82a74-417a-5468-9d9e-86beef6b387a -t ext4 -b 4096 out/target/product/sunfish/system_ext.img 100773 19:18:45 mkuserimg_mke2fs.py INFO: Env: {} 19:18:45 mkuserimg_mke2fs.py INFO: Running: /srv/src/LINEAGE_21_0/out/host/linux-x86/bin/e2fsdroid -e -p out/target/product/sunfish/system -S out/target/product/sunfish/obj/ETC/file_contexts.bin_intermediates/file_contexts.bin -f out/target/product/sunfish/system_ext -a /system_ext out/target/product/sunfish/system_ext.img 19:18:47 mkuserimg_mke2fs.py ERROR: Failed to run e2fsdroid_cmd: __populate_fs: Could not allocate block in ext2 filesystem while writing file "exim.vim" e2fsdroid: Could not allocate block in ext2 filesystem while populating file system mke2fs 1.46.6 (1-Feb-2023) Creating filesystem with 100773 4k blocks and 3648 inodes Filesystem UUID: 3f03299a-4175-53fd-9041-1f6ac7a6973b Superblock backups stored on blocks: 32768, 98304 Allocating group tables: done Writing inode tables: done Writing superblocks and filesystem accounting information: done __populate_fs: Could not allocate block in ext2 filesystem while writing file "exim.vim" e2fsdroid: Could not allocate block in ext2 filesystem while populating file system Out of space? Out of inodes? The tree size of out/target/product/sunfish/system_ext is 408568832 bytes (389 MB), with reserved space of 4194305 bytes (4 MB). The max image size for filesystem files is 412766208 bytes (393 MB), out of a total partition size of 412766208 bytes (393 MB). ``` ```

Originally posted by @gwstorm in https://github.com/lineageos4microg/docker-lineage-cicd/issues/615#issuecomment-2076478803

[petefoth]

• Try removing vim `: No, that fails with

  build/make/core/main.mk:1376: warning:  device/google/sunfish/lineage_sunfish.mk includes non-existent modules in PRODUCT_PACKAGES
  Offending entries:
  vim

[petefoth]

The problem:

• in file vendor/lineage/config/BoardConfigReservedSize.mk, if 'WITH_GMSistrue, variableBOARD_SYSTEM_EXTIMAGE_PARTITION_RESERVED_SIZEis set to8388609`
• if we comment that line out, the error goes away, and the build succeeds. My guess is that this extra space reserved in the system_ext partition is enough to take the image size over the limit
• permanent solutions
  1. Create and maintain a patch to that file, and apply it during the build
  2. explicitly set BOARD_SYSTEM_EXTIMAGE_PARTITION_RESERVED_SIZE to 0 in the build script. (The file uses the ?= make operator to set the variable only if it is not already set - see here for more information)

I built successfully with by removing the line. I'll try 2 and see if it works (actually I'll set it to 1 - I tried 0 earlier and it didn't work). Nope it failed with

Out of space? Out of inodes? The tree size of out/target/product/sunfish/system_ext is 408568832 bytes (389 MB), with reserved space of 1 bytes (0 MB).
The max image size for filesystem files is 408571904 bytes (389 MB), out of a total partition size of 408571904 bytes (389 MB).

[petefoth]

The value of 8MB reserved size for builds with WITH_GMS=true was set in this commit: https://github.com/LineageOS/android_vendor_lineage/commit/3940fb424045c829ab2f40a3759e9710c04ca5a5 . It was reverted a few hours ago in this commit: https://github.com/LineageOS/android_vendor_lineage/commit/a27417bb8126bdb461e1778619441b155ba4962e

I have tried building without the workaround and the build succeeded.

Closing this issue