Can't mount encrypted partition when LTO is enabled

[hadess]

Originally reported as https://bugzilla.redhat.com/show_bug.cgi?id=1950208

Using the current git HEAD version (ee71aa5c87c0831c1ae17048951fe9cd7579c3db)

$ sudo apfs-fuse /dev/sdd2 /mnt -d 31
Device /dev/sdd2 opened. Size is 16000000000
Mounting xid different from NXSB at 0 (xid = 36). xid = 36
Mounting xid 36
omap: oid=1027 xid=36 flags=0 size=0 paddr=1027
omap: oid=1028 xid=36 flags=0 size=0 paddr=1028
starting LoadKeybag @ 20003
Initialization of KeyManager failed.
Unable to load container.

This seems to be caused by Fedora enabling LTO by default in its builds: https://fedoraproject.org/wiki/LTOByDefault

Building with LTO generate quite a few new warnings compared to without LTO, including a fair bunch of signedness issues, which are probably symptoms of places that LTO will eventually modify and thus break.

• Build without LTO
• Build with LTO

[sgan81]

Stupid aliasing rules ... anyway, it should be fixed now, at the expense of encryption speed ...

[sgan81]

So, after a few more tests, I found the following:

• The original code works if -flto is specified alongside with -fno-strict-alias
• LTO with clang works fine

Apparently, the reason was that I had done some pointer casting in aes.Encrypt and aes.Decrypt (void to uint32_t ) in order to speed up things. GCC was of the opinion though that aes.Encrypt/Decrypt did not write anything, and that the uint32_t did not alias the void , and therefore the whole call to aes.Decrypt could be optimized out ...

So I don't really know what's better ... having somewhat slower code, or using -fno-strict-alias when compiling with GCC and LTO ...

[RJVB]

On Wednesday January 04 2023 18:42:33 Simon Gander wrote:

So I don't really know what's better ... having somewhat slower code, or using -fno-strict-alias when compiling with GCC and LTO ...

I'd say the latter because that's a condition you can check for and handle at configure time. And the flag need only be passed when compiling that single file.

FWIW, doesn't -fno-strict-alias reenable us to do the things we could do in C, back when we were still allowed to know better than the compiler and write optimised code?

[bbetter173]

@sgan81 - I've tried to build with clang but get the same result (On Asahi Fedora Remix):

$ env | grep -E "CC|CXX"
CXX=/usr/bin/clang++
CC=/usr/bin/clang

$ mkdir build; cd build; cmake ..
CMake Deprecation Warning at CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

-- The C compiler identification is Clang 18.1.6
-- The CXX compiler identification is Clang 18.1.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /usr/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/clang++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Configuring done (0.6s)
-- Generating done (0.0s)
-- Build files have been written to: /home/apfs-fuse/build

$ make
...
[ 86%] Building CXX object CMakeFiles/apfs-dump.dir/ApfsDump/Apfs.cpp.o
[ 88%] Linking CXX executable apfs-dump
[ 88%] Built target apfs-dump
[ 90%] Building CXX object CMakeFiles/apfs-dump-quick.dir/ApfsDumpQuick/ApfsDumpQuick.cpp.o
[ 92%] Linking CXX executable apfs-dump-quick
[ 92%] Built target apfs-dump-quick
[ 94%] Building CXX object CMakeFiles/apfs-fuse.dir/apfsfuse/ApfsFuse.cpp.o
/home/bbettridge/Development/apfs-fuse/apfsfuse/ApfsFuse.cpp:676:38: warning: unused parameter 'data' [-Wunused-parameter]
  676 | static int apfs_parse_fuse_opt(void *data, const char *arg, int key, struct fuse_args* outargs)
      |                                      ^
/home/bbettridge/Development/apfs-fuse/apfsfuse/ApfsFuse.cpp:676:88: warning: unused parameter 'outargs' [-Wunused-parameter]
  676 | static int apfs_parse_fuse_opt(void *data, const char *arg, int key, struct fuse_args* outargs)
      |                                                                                        ^
2 warnings generated.
[ 96%] Linking CXX executable apfs-fuse
[ 96%] Built target apfs-fuse
[ 98%] Building CXX object CMakeFiles/apfsutil.dir/ApfsUtil/ApfsUtil.cpp.o
[100%] Linking CXX executable apfsutil
[100%] Built target apfsutil

$ sudo ./apfs-fuse /dev/nvme0n1p2 /mnt -d 31
Device /dev/nvme0n1p2 opened. Size is 350001037312
Mounting xid different from NXSB at 0 (xid = 5898584). xid = 5898584
Mounting xid 5898584
omap: oid=4204049 xid=5898584 flags=0 size=4096 paddr=85434423
omap: oid=4204048 xid=5898584 flags=0 size=4096 paddr=85434422
starting LoadKeybag @ 183cde
Initialization of KeyManager failed.
Unable to load container.

I'm not sure how to build it with g++ using cmake and the flags referenced earlier. Would you be able to provide an example? I can include full build logs if you think they would help, but nothing stands out to me as an obvious issue. Thanks!

[sgan81]

In case you tried to mount an encrypted internal drive of a mac with T2 chip, that does not work.