Some filesystems don't support direct I/O

[JoanFM]

Hello @penberg ,

I was exploring the repo,

and I am having an issue starting the limbo CLI with the testing DB.

python testing/gen-database.py
cargo run database.db

Gives:

Error: Invalid argument (os error 22)

[JoanFM]

Just debugging, I removed the O_DIRECT flag but then I get the error:

Error: file is not a database

[penberg]

Curious, what does uname -a say? Is this old enough Linux that doesn't support io_uring with O_DIRECT... 🤔

[JoanFM]

Linux *** 5.4.0-150-generic #167~18.04.1-Ubuntu SMP Wed May 24 00:51:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

Also, if is useful, the testing gen-database script finished successfully with no exceptions. I double checked.

[penberg]

@JoanFM Can you revert the O_DIRECT change and run strace -f target/debug/limbo database.db on pristine sources to see what system call returns -22 (which is EINVAL signaling that io_uring setup or some syscall parameters are wrong).

[JoanFM]

Here it is

openat(AT_FDCWD, "database.db", O_RDWR|O_DIRECT|O_CLOEXEC) = -1 EINVAL (Invalid argument)
munmap(0x7fd7a16eb000, 4928)            = 0
munmap(0x7fd7a16ed000, 8192)            = 0
close(3)                                = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "Invalid argument", 16Invalid argument)        = 16
write(2, " (os error ", 11 (os error )             = 11
write(2, "22", 222)                       = 2
write(2, ")", 1))                        = 1
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7fd7a16ef000, 12288)           = 0
madvise(0x2a868050000, 33226752, MADV_DONTNEED) = 0
exit_group(1)                           = ?
+++ exited with 1 ++

[penberg]

@JoanFM Thanks, what filesystem do you have? Can you also trace the failing system call when you drop O_DIRECT?

[JoanFM]

Hey @penberg ,

Funny finding:

My fs is ecryptfs and then if I place in a non encrypted part of my fs I do not get the OSError 22, but get the File is not a database.

The strace when bypassing it is:

mmap(0x5abe8000000, 1073741824, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS|MAP_NORESERVE, -1, 0) = 0x5abe8000000
ioctl(2, TCGETS, {B38400 opost isig icanon echo ...}) = 0
syscall_0x1a9(0x80, 0x7ffd35b1aa00, 0x7ffd35b1abac, 0, 0, 0x3) = 0x3
mmap(NULL, 8192, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0x10000000) = 0x7ff4f5e81000
mmap(NULL, 4928, PROT_READ|PROT_WRITE, MAP_SHARED|MAP_POPULATE, 3, 0) = 0x7ff4f5e7f000
openat(AT_FDCWD, "database.db", O_RDWR|O_CLOEXEC) = 4
syscall_0x1aa(0x3, 0x1, 0x1, 0x1, 0, 0x80) = 0x1
getrandom("\x53\x29\x55\x94\xeb\xba\x6f\x92\x67\x7b\x4c\x01\x6d\xf3\x78\x69", 16, 0x4 /* GRND_??? */) = 16
close(4)                                = 0
munmap(0x7ff4f5e7f000, 4928)            = 0
munmap(0x7ff4f5e81000, 8192)            = 0
close(3)                                = 0
write(2, "Error: ", 7Error: )                  = 7
write(2, "file is not a database", 22file is not a database)  = 22
write(2, "\n", 1
)                       = 1
sigaltstack({ss_sp=NULL, ss_flags=SS_DISABLE, ss_size=8192}, NULL) = 0
munmap(0x7ff4f5e83000, 12288)           = 0
madvise(0x5abe8050000, 33226752, MADV_DONTNEED) = 0
exit_group(1)                           = ?
+++ exited with 1 +++

[JoanFM]

So I assume this ecryptfs does not support I/O Direct I guess this is intended that u do not want to skip some things for security.

But in the non-ecrypt side I get a not a database error

[penberg]

Indeed looks like encryptfs does not support direct I/O so we need to probe the filesystem before using it. The "not a database" error needs more digging why that happens.

[penberg]

Fixed by https://github.com/penberg/limbo/pull/166