e2mkdir 'No free space in the directory' failure

[Gateworks]

I build firmware images for embdedded systems and I have a root filesystem generation process that creates an ext4 filesystem using mkfs.ext4 (mkfs.ext4 -q -F -O ^metadata_csum rootfs.ext4) and populating it by mounting it and installing files. This is done on a system with mke2fs v1.46.5.

In a follow-on process I use e2tools e2cp/e2mkdir to add files to it and I'm hitting a failure where e2mkdir errors out with 'No free space in the directory' yet as far as I can tell there is plenty of space.

I've built e2tools from the latest git source linking with e2fsprogs from its latest git source to try to better understand the failure. Before I got too deep into the details of the ext filesystem format I figured I would ask for some help here: e2mkdir fails with: No free space in the directory (EXT2_ET_DIR_NO_SPACE) ext2fs_link() fails with EXT2_ET_DIR_NO_SPACE ext2fs_dir_iterate2() does not set ls.done flag

After original creation of the ext4 I see the following details: $ e2freefrag rootfs.ext4 Device: rootfs.ext4 Blocksize: 4096 bytes Total blocks: 655360 Free blocks: 170921 (26.1%)

Min. free extent: 4 KB Max. free extent: 516064 KB Avg. free extent: 11992 KB Num. free extent: 57

HISTOGRAM OF FREE EXTENT SIZES: Extent Size Range : Free extents Free Blocks Percent 4K... 8K- : 6 6 0.00% 8K... 16K- : 7 16 0.01% 16K... 32K- : 8 37 0.02% 32K... 64K- : 16 158 0.09% 64K... 128K- : 1 30 0.02% 256K... 512K- : 4 413 0.24% 512K... 1024K- : 2 362 0.21% 1M... 2M- : 10 4293 2.51% 8M... 16M- : 1 3822 2.24% 128M... 256M- : 1 32768 19.17% 256M... 512M- : 1 129016 75.48%

$ tune2fs -l rootfs.ext4 tune2fs 1.46.5 (30-Dec-2021) Filesystem volume name: rootfs Last mounted on: /tmp/tmp.TKm2wiMNtR Filesystem UUID: 82ddb049-9636-4b26-b9cc-a0628ebfcfeb Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 163840 Block count: 655360 Reserved block count: 32768 Overhead clusters: 28590 Free blocks: 170921 Free inodes: 130337 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 319 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Thu Feb 15 14:48:14 2024 Last mount time: Thu Feb 15 14:48:14 2024 Last write time: Thu Feb 15 14:48:20 2024 Mount count: 1 Maximum mount count: -1 Last checked: Thu Feb 15 14:48:14 2024 Check interval: 0 () Lifetime writes: 1817 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: b9544c01-e03f-40da-b209-435591957e26 Journal backup: inode blocks

After failed e2mkdir: $ e2freefrag rootfs.ext4 Device: rootfs.ext4 Blocksize: 4096 bytes Total blocks: 655360 Free blocks: 109063 (16.6%)

Min. free extent: 436252 KB Max. free extent: 436252 KB Avg. free extent: 436252 KB Num. free extent: 1

HISTOGRAM OF FREE EXTENT SIZES: Extent Size Range : Free extents Free Blocks Percent 256M... 512M- : 1 109063 100.00%

$ tune2fs -l rootfs.ext4 tune2fs 1.46.5 (30-Dec-2021) Filesystem volume name: rootfs Last mounted on: /tmp/tmp.TKm2wiMNtR Filesystem UUID: 82ddb049-9636-4b26-b9cc-a0628ebfcfeb Filesystem magic number: 0xEF53 Filesystem revision #: 1 (dynamic) Filesystem features: has_journal ext_attr resize_inode dir_index filetype extent 64bit flex_bg sparse_super large_file huge_file dir_nlink extra_isize Filesystem flags: signed_directory_hash Default mount options: user_xattr acl Filesystem state: clean Errors behavior: Continue Filesystem OS type: Linux Inode count: 163840 Block count: 655360 Reserved block count: 32768 Overhead clusters: 28590 Free blocks: 109063 Free inodes: 124748 First block: 0 Block size: 4096 Fragment size: 4096 Group descriptor size: 64 Reserved GDT blocks: 319 Blocks per group: 32768 Fragments per group: 32768 Inodes per group: 8192 Inode blocks per group: 512 Flex block group size: 16 Filesystem created: Thu Feb 15 14:48:14 2024 Last mount time: Thu Feb 15 14:48:14 2024 Last write time: Fri Feb 16 09:51:53 2024 Mount count: 1 Maximum mount count: -1 Last checked: Thu Feb 15 14:48:14 2024 Check interval: 0 () Lifetime writes: 2971 MB Reserved blocks uid: 0 (user root) Reserved blocks gid: 0 (group root) First inode: 11 Inode size: 256 Required extra isize: 32 Desired extra isize: 32 Journal inode: 8 Default directory hash: half_md4 Directory Hash Seed: b9544c01-e03f-40da-b209-435591957e26 Journal backup: inode blocks

As you can see there are plenty of free blocks available but it looks like there is something funny about the 'extents' and I'm not really clear what that means. Am I running out of inodes? What could be going wrong here?

[tytso]

This is a bug in e2tools. The ext2fs library was originally designed for use by tools like e2fsck, so things like ext2fs_link() are a very low level library function. If ext2fs_link() fails with the error EXT2_ET_DIR_NO_SPACE the calling program needs to call ext2fs_expand_dir() and then retry the call to ext2fs_link(). For example, please see:

https://github.com/tytso/e2fsprogs/blob/260dfea450e387cbd2c8de79a7c2eeacc26f74e9/misc/fuse2fs.c#L972

[Gateworks]

Sorry it has taken me so long to get back to this.

The call-stack here is: e2tools:main_e2mkdir() -> e2tools:create_dir() -> e2tools:create_subdir() -> ext2fsprgs:ext2fs_mkdir() and its failing the ext2fs_link in that function: https://github.com/tytso/e2fsprogs/blob/master/lib/ext2fs/mkdir.c#L168

So based on your explanation shouldn't the ext2fs_expand_dir be placed in e2fsprogs:ext2fs_mkdir?

diff --git a/lib/ext2fs/mkdir.c b/lib/ext2fs/mkdir.c
index 437c8ffc..666bf700 100644
--- a/lib/ext2fs/mkdir.c
+++ b/lib/ext2fs/mkdir.c
@@ -166,6 +166,12 @@ errcode_t ext2fs_mkdir(ext2_filsys fs, ext2_ino_t parent, ext2_ino_t inum,
                if (retval != EXT2_ET_FILE_NOT_FOUND)
                        goto cleanup;
                retval = ext2fs_link(fs, parent, name, ino, EXT2_FT_DIR);
+               if (retval == EXT2_ET_DIR_NO_SPACE) {
+                       retval = ext2fs_expand_dir(fs, parent);
+                       if (retval)
+                               goto cleanup;
+                       retval = ext2fs_link(fs, parent, name, ino, EXT2_FT_DIR);
+               }
                if (retval)
                        goto cleanup;
        }

[tytso]

The ext2fs library has the convention that it is the caller which is responsible for calling ext2fs_expand_dir(). The reason for that is that (for example) e2fsck might need to wrap ext2fs_expnd_dir() with a function like e2fsck_expand_dir() which also manages its own internal bookkeeping (since the allocation bitmaps haven't been validated yet, so e2fsck has to track what blocks and inodes are actually allocated so that in pass 5 it can compare it with what the on-disk allocation bitmaps contains). This is why I said above that the bug is in e2tools, as the caller of ext2fs_mkdir(), should call ext2fs_exand_dir() if necessary.

Now, it's true that ext2fs_mkdir() isn't called by e2fsck, so in theory, we could make this change. But (a) e2tools might be linked against other versions of libext2fs, where as if you make the change to e2tools, it will work with new and older version of the library, and (b) I want to keep consistency with the other ext2fs functions where we can't just change the semantics where it is the caller's responsible to expand the directory.

I could imagine adding new functions, ext2fs_mkdir2(), ext2fs_symlink2(), etc., which take a new function paramater, flags, so we could pass in a new flag, DIRENT_FLAG_AUTOEXPAND, which would automatically expand the directory if needed. This would be needed if we wanted to make ext2fs{mkdir,link,symlink}() / ext2fs_expand_dir() more efficient (by avoiding a N*3 overhead for very large directories), and if we want to add support for htree/indexed directories (right now if we add a directory entry we have to clear the indexed flag and depend on e2fsck to convert the directory back to be indexed). This would primarily needed if we wanted to make fuse2fs more efficient, since for the mke2fs -d use case, it's good enough to populate file system and then run e2fsck -fD to index all of the directories.

But even if we added these two new functions (we need to create new functions instead of changing function prototypes to avoid breaking existing binaries depending on the libext2fs shared library), e2tools would have to be changed to use the new functions, e2tools would only be usable with the newer libext2fs libraries. So that's not the easist way to fix your problem; the simplest way to fix this is to just edit/change e2tools.

In general, fuse2fs is only used by Mac users with MacFuse, or for users who don't have root access, but where the system administrator allows unprivileged FUSE mounts. (Perhaps because if the user picks up a USB thumb drive containig a maliciously corrupt file system prepared by a national state intelligence agency, if they use fuse2fs, they will only screw them selves, but if the file system is mounted by the kernel, a zero-day kerne bug could result in a complete system compromise.) So we haven't had anyone sufficiently motivated to make fuse2fs more performant since for Linux users, if they really care about performance, they'll just mount the file system and use the ext4 kernel driver.. And that's why fuse2fs and libext2fs don't have support for htree directories or support for journalled access....

[Gateworks]

Ok - understood. I think I was mislead because in some places in e2fsprogs ext2fs_mkdir and ext2fs_link are protected with this but it looks like those calls are in 'apps' and not in 'lib'.

it looks like someone already issued a PR for e2tools which unfortunately has gone stale https://github.com/e2tools/e2tools/pull/30