Support for FreeBSD-arm64

[sec]

Hi, Not to spam #14537 (as it's already quite big) :) As full native build of either v6 and v7 is possible under FreeBSD-amd64, I went ahead and tried to make the same process work for ARM64.

After getting configure to success, I've tried:

1. Using Linux arm64 host to cross compile to FreeBSD (with custom rootfs created with libs, etc.) As it was used before, to cross compile from Linux-amd64 to Freebsd-x64, this fail at very early stage with:

   [  1%] Built target System.Globalization.Native-Static
   Scanning dependencies of target coreclrpal
   [  1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/dec/state.c.o
   [  1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
   [  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
   In file included from /root/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
   /root/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
   #include <type_traits>
        ^~~~~~~~~~~~~
   [  1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/root/runtime/src/native/external/brotli/enc/backward_references.c.o
   [  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
   1 error generated.
   [  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_time.c.o
   make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
   make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2

Command I've used ROOTFS_DIR=/home/ubuntu/arm64 ./src/coreclr/build-runtime.sh -arm64 -debug clang14 -cross -os FreeBSD. This does look like something missing in configure step, as those headers are present on rootfs_dir and host system:

root@instance-20220624-1221:/# find . -name type_traits
./home/ubuntu/arm64/usr/include/c++/v1/tr1/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/experimental/type_traits
./home/ubuntu/arm64/usr/include/c++/v1/type_traits

./usr/include/c++/11/tr2/type_traits
./usr/include/c++/11/tr1/type_traits
./usr/include/c++/11/experimental/type_traits
./usr/include/c++/11/type_traits

./root/runtime/src/coreclr/inc/clr_std/type_traits

so I guess something during configure stage should be added to look in those directories?

2. Using FreeBSD arm64 host directly to compile Trying with both -cross and without, it fails with the same error:

   
   [  4%] Building C object libs-native/System.Security.Cryptography.Native/CMakeFiles/objlib.dir/pal_evp_pkey_ecdh.c.o
   In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:450:1: error: unknown type name 'fpsimd_context'
   fpsimd_context* GetNativeSigSimdContext(native_context_t *mc)
   ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:9: error: unknown type name 'fpsimd_context'
       fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
       ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:47: error: unknown type name 'fpsimd_context'
       fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
                                             ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:456:82: error: no member named '__reserved' in '__mcontext'
       fpsimd_context* fp = reinterpret_cast<fpsimd_context *>(&mc->uc_mcontext.__reserved[size]);
                                                                ~~~~~~~~~~~~~~~ ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:458:30: error: use of undeclared identifier 'FPSIMD_MAGIC'
       if(fp->head.magic == FPSIMD_MAGIC)
                            ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:460:46: error: use of undeclared identifier 'fpsimd_context'
           _ASSERTE(fp->head.size >= sizeof(fpsimd_context));
                                            ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:461:69: error: no member named '__reserved' in '__mcontext'
           _ASSERTE(size + fp->head.size <= sizeof(mc->uc_mcontext.__reserved));
                                                   ~~~~~~~~~~~~~~~ ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/dbgmsg.h:362:35: note: expanded from macro '_ASSERTE'
   #define _ASSERTE(expr) do { if (!(expr)) { ASSERT("Expression: " #expr "\n"); } } while(0)
                                 ^~~~
   In file included from /usr/home/sec/runtime/src/coreclr/pal/src/debug/debug.cpp:32:
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:28: error: use of undeclared identifier 'fpsimd_context'
   } while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
                          ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:472:70: error: no member named '__reserved' in '__mcontext'
   } while (size + sizeof(fpsimd_context) <= sizeof(mc->uc_mcontext.__reserved));
                                                    ~~~~~~~~~~~~~~~ ^
   /usr/home/sec/runtime/src/coreclr/pal/src/include/pal/context.h:480:7: error: unknown type name 'fpsimd_context'
   const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
     ^
   10 errors generated.
   [  4%] Linking C static library libSystem.Native.a
   --- pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o ---
   *** [pal/src/CMakeFiles/coreclrpal.dir/debug/debug.cpp.o] Error code 1

make[2]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug *** [all] Error code 6

make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug 1 error

make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug

Command used was the same as with ad. 1. This looks more complicated, as it have something to do with libunwind being used? No idea how to get over that :)

Including patches I've come up with.

If anyone expierenced with cross build could comment on that, to even confirm it's possible in the current state to make that work or would it require far more work?

If someone is interested in helping/testing/making this work I can provide access to either FreeBSD or Linux ARM64 box that I use.

[runtime_freebsd_arm64.txt](https://github.com/dotnet/runtime/files/8993111/runtime_freebsd_arm64.txt)

[Thefrank]

OSX/BSD builds use the target system's libunwind: from src/coreclr/pal/src/CMakeLists.txt

if(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)
     # On OSX and *BSD, we use the libunwind that's part of the OS
     set(CLR_CMAKE_USE_SYSTEM_LIBUNWIND 1)
endif(CLR_CMAKE_TARGET_OSX OR CLR_CMAKE_TARGET_FREEBSD)

which makes it add only include_directories(SYSTEM $ENV{ROOTFS_DIR}/usr/local/include)

[am11]

Yup, I think we just need to update the list of compatible context registers for freebsd-arm64 to make it work.

[sec]

If that would only make sense to me I would maybe help more, still learing on the fly. I found #40149 and doing quick change:

diff --git a/src/coreclr/pal/src/include/pal/context.h b/src/coreclr/pal/src/include/pal/context.h
index fec5139fd5e..2cc4941d653 100644
--- a/src/coreclr/pal/src/include/pal/context.h
+++ b/src/coreclr/pal/src/include/pal/context.h
@@ -407,7 +407,7 @@ inline void *FPREG_Xstate_Ymmh(const ucontext_t *uc)

 #if defined(HOST_ARM64)

-#ifndef TARGET_OSX
+#if !defined(TARGET_OSX) && !defined(TARGET_FREEBSD)

 #define MCREG_X0(mc)      ((mc).regs[0])
 #define MCREG_X1(mc)      ((mc).regs[1])
@@ -482,7 +482,7 @@ const fpsimd_context* GetConstNativeSigSimdContext(const native_context_t *mc)
     return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
 }

-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX

 #define MCREG_X0(mc)      ((mc)->__ss.__x[0])
 #define MCREG_X1(mc)      ((mc)->__ss.__x[1])

native compile fail with:

In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:177:2: error: unsupported architecture
#error unsupported architecture
 ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:183:5: error: use of undeclared identifier 'ASSIGN_UNWIND_REGS'
    ASSIGN_UNWIND_REGS
    ^
2 errors generated.
--- pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o ---
*** [pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o] Error code 1

Is this the context registers that needs to be updated (looked at the code and there's no ARM64 for any of the targets)?

[janvorli]

@sec, what are the detected features (the HAVE_xxx) for the FreeBSD arm64 build? When you do a clean build, it lists them to the console. From those, I should be able to tell where to update the context registers extraction.

[sec]

@janvorli There you go:

-- Performing Test COMPILER_SUPPORTS_F_STACK_PROTECTOR_STRONG - Success
-- Performing Test COMPILER_SUPPORTS_W_IMPLICIT_FALLTHROUGH - Success
-- Performing Test HAVE_LTO - Failed
-- Performing Test COMPILER_SUPPORTS_F_ALIGNED_NEW - Success
-- Performing Test HAVE_DIRENT_D_TYPE - Success
-- Performing Test C_SUPPORTS_WUNGUARDED_AVAILABILITY - Success
-- Performing Test HAVE_IN_PKTINFO - Failed
-- Performing Test HAVE_IP_MREQN - Success
-- Performing Test HAVE_STATFS_VFS - Failed
-- Performing Test HAVE_STATFS_MOUNT - Success
-- Performing Test HAVE_FLOCK64 - Failed
-- Performing Test HAVE_IFADDRS - Success
-- Performing Test HAVE_UTSNAME_DOMAINNAME - Failed
-- Performing Test HAVE_STAT_BIRTHTIME - Success
-- Performing Test HAVE_STAT_FLAGS - Success
-- Performing Test HAVE_STAT_TIMESPEC - Success
-- Performing Test HAVE_STAT_TIM - Success
-- Performing Test HAVE_STAT_NSEC - Success
-- Performing Test HAVE_DIRENT_NAME_LEN - Success
-- Performing Test HAVE_STATFS_FSTYPENAME - Success
-- Performing Test HAVE_STATVFS_FSTYPENAME - Failed
-- Performing Test HAVE_GNU_STRERROR_R - Failed
-- Performing Test HAVE_READDIR_R - Success
-- Performing Test KEVENT_HAS_VOID_UDATA - Success
-- Performing Test HAVE_FDS_BITS - Success
-- Performing Test HAVE_PRIVATE_FDS_BITS - Success
-- Performing Test HAVE_SENDFILE_4 - Failed
-- Performing Test HAVE_SENDFILE_6 - Failed
-- Performing Test HAVE_SENDFILE_7 - Success
-- Performing Test HAVE_GETNAMEINFO_SIGNED_FLAGS - Success
-- Performing Test HAVE_SHM_OPEN_THAT_WORKS_WELL_ENOUGH_WITH_MMAP - Success
-- Performing Test HAVE_CLOCK_MONOTONIC - Success
-- Performing Test HAVE_CLOCK_REALTIME - Success
-- Performing Test BIND_ADDRLEN_UNSIGNED - Success
-- Performing Test IPV6MR_INTERFACE_UNSIGNED - Success
-- Performing Test INOTIFY_RM_WATCH_WD_UNSIGNED - Failed
-- Performing Test HAVE_MKSTEMPS - Success
-- Performing Test HAVE_MKSTEMP - Success
-- Performing Test HAVE_NETINET_TCP_VAR_H - Success
-- Performing Test HAVE_NETINET_UDP_VAR_H - Success
-- Performing Test HAVE_NETINET_IP_VAR_H - Success
-- Performing Test HAVE_NETINET_ICMP_VAR_H - Success
-- Performing Test HAVE_TCP_H_TCPSTATE_ENUM - Failed
-- Performing Test HAVE_GETDOMAINNAME_SIZET - Failed
-- Performing Test HAVE_IN_EXCL_UNLINK - Success
-- Performing Test HAVE_TCP_H_TCP_KEEPALIVE - Failed
-- Performing Test HAVE_BUILTIN_MUL_OVERFLOW - Success
-- Performing Test HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS - Success
-- Performing Test HAVE_VM_FLAGS_SUPERPAGE_SIZE_ANY - Failed
-- Performing Test HAVE_MAP_HUGETLB - Failed
-- Performing Test HAVE_LTTNG_TRACEPOINT_H - Failed
-- Performing Test HAVE_CPUSET_T - Success
-- Performing Test HAVE_TM_GMTOFF - Success
-- Performing Test HAVE_GREGSET_T - Failed
-- Performing Test HAVE___GREGSET_T - Failed
-- Performing Test HAVE_FPSTATE_GLIBC_RESERVED1 - Failed
-- Performing Test HAVE_SYSINFO_WITH_MEM_UNIT - Failed
-- Performing Test HAVE_FPREGS_WITH_CW - Failed
-- Performing Test REALPATH_SUPPORTS_NONEXISTENT_FILES - Failed
-- Performing Test SSCANF_SUPPORT_ll - Success
-- Performing Test SSCANF_CANNOT_HANDLE_MISSING_EXPONENT - Failed
-- Performing Test HAVE_LARGE_SNPRINTF_SUPPORT - Success
-- Performing Test HAVE_BROKEN_FIFO_SELECT - Failed
-- Performing Test HAVE_BROKEN_FIFO_KEVENT - Failed
-- Performing Test HAVE_SCHED_GET_PRIORITY - Success
-- Performing Test HAVE_WORKING_GETTIMEOFDAY - Success
-- Performing Test HAVE_WORKING_CLOCK_GETTIME - Success
-- Performing Test HAVE_CLOCK_MONOTONIC_COARSE - Success
-- Performing Test HAVE_CLOCK_THREAD_CPUTIME - Success
-- Performing Test MMAP_ANON_IGNORES_PROTECTION - Failed
-- Performing Test ONE_SHARED_MAPPING_PER_FILEREGION_PER_PROCESS - Failed
-- Performing Test PTHREAD_CREATE_MODIFIES_ERRNO - Failed
-- Performing Test SEM_INIT_MODIFIES_ERRNO - Failed
-- Performing Test HAVE_PROCFS_CTL - Failed
-- Performing Test HAVE_PROCFS_MAPS - Failed
-- Performing Test HAVE_PROCFS_STAT - Failed
-- Performing Test HAVE_PROCFS_STATUS - Success
-- Performing Test HAVE_COMPATIBLE_ACOS - Success
-- Performing Test HAVE_COMPATIBLE_ASIN - Success
-- Performing Test HAVE_COMPATIBLE_POW - Success
-- Performing Test HAVE_VALID_NEGATIVE_INF_POW - Success
-- Performing Test HAVE_VALID_POSITIVE_INF_POW - Success
-- Performing Test HAVE_COMPATIBLE_ATAN2 - Success
-- Performing Test HAVE_COMPATIBLE_EXP - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGB0 - Failed
-- Performing Test HAVE_COMPATIBLE_ILOGBNAN - Success
-- Performing Test HAVE_COMPATIBLE_LOG - Success
-- Performing Test HAVE_COMPATIBLE_LOG10 - Success
-- Performing Test UNGETC_NOT_RETURN_EOF - Failed
-- Performing Test HAS_POSIX_SEMAPHORES - Success
-- Performing Test GETPWUID_R_SETS_ERRNO - Failed
-- Performing Test FILE_OPS_CHECK_FERROR_OF_PREVIOUS_CALL - Failed
-- Performing Test UNWIND_CONTEXT_IS_UCONTEXT_T - Success
-- Performing Test HAVE_XSWDEV - Success
-- Performing Test HAVE_XSW_USAGE - Failed
-- Performing Test HAVE_PUBLIC_XSTATE_STRUCT - Failed
-- Performing Test HAVE_PR_SET_PTRACER - Failed
-- Performing Test HAVE_FULLY_FEATURED_PTHREAD_MUTEXES - Success
-- Performing Test HAVE_PTHREAD_THREADID_NP - Failed
-- Performing Test HAVE_PTHREAD_GETTHREADID_NP - Success

edit: also attached full log from build attempt. build_log.txt

[janvorli]

It seems to me that your last change was correct. But you also need to add #elif (defined(HOST_UNIX) && defined(HOST_ARM64)) before the #else with #error that fires. In that #elif, you'd add:

    ASSIGN_REG(Pc) \
    ASSIGN_REG(Sp) \
    ASSIGN_REG(Fp) \
    ASSIGN_REG(Lr) \
    ASSIGN_REG(X19) \
    ASSIGN_REG(X20) \
    ASSIGN_REG(X21) \
    ASSIGN_REG(X22) \
    ASSIGN_REG(X23) \
    ASSIGN_REG(X24) \
    ASSIGN_REG(X25) \
    ASSIGN_REG(X26) \
    ASSIGN_REG(X27) \
    ASSIGN_REG(X28)

I hope I got it right.

[sec]

@janvorli Tried that, it failed with bunch of errors for every register:

In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:190:2: error: use of undeclared identifier 'MCREG_X21'
        ASSIGN_UNWIND_REGS
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:178:2: note: expanded from macro 'ASSIGN_UNWIND_REGS'
        ASSIGN_REG(X21) \
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:189:25: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(unwContext->uc_mcontext) = winContext->reg;
                        ^
<scratch space>:95:1: note: expanded from here
MCREG_X21
^

Leaving this empty, will go on but then fail again with similar errors. If I'm looking at this correctly, I think it's missing some #def for HOST_ARM64 and TARGET_FREEBSD inside src/coreclr/pal/src/include/pal/context.h with those registers?

And also there's again error: unknown type name 'fpsimd_context' - which FreeBSD doesn't have, but if I add || defined(TARGET_FREEBSD) next to TARGET_OSX those errors seems to go away (don't know if that will work in the end though, as I going blind in here. Also the undefined registers errors are there, so my bet is that fixing those first would be real fix).

I wodner, if there's libunwind used (which state FreeBSD is supported under x64 and arm64) - shouldn't this be used (if I assume this code here is about that).

As I saw @sdmaclea did the OSX part of changes - maybe some hints? :)

Just to make sure, this is native build under ARM64 FreeBSD host.

[janvorli]

Ah, I haven't noticed that your change around line 490 was like this:

-#else // TARGET_OSX
+#elif defined(TARGET_OSX) // TARGET_OSX

That way, you have eliminated both definitions of the MCREG_xxx for arm64. What does the ucontext_t struct look like on FreeBSD arm64? We'll either need to define the MCREG_xxx for the FreeBSD separately or reuse the one for Linux / OSX if it happens to be the same.

[janvorli]

Ah, found it in FreeBSD sources. So it is different and we will need to have another #elif TARGET_FREEBSD there. It seems that the definitions should look like:

#define MCREG_X21(mc) (mc.mc_gpregs.gp_x[21])
#define MCREG_X22(mc) (mc.mc_gpregs.gp_x[22])
#define MCREG_X23(mc) (mc.mc_gpregs.gp_x[23])
#define MCREG_X24(mc) (mc.mc_gpregs.gp_x[24])
#define MCREG_X25(mc) (mc.mc_gpregs.gp_x[25])
#define MCREG_X26(mc) (mc.mc_gpregs.gp_x[26])
#define MCREG_X27(mc) (mc.mc_gpregs.gp_x[27])
#define MCREG_X28(mc) (mc.mc_gpregs.gp_x[28])
#define MCREG_Fp(mc)      (mc.mc_gpregs.gp_fp)
#define MCREG_Lr(mc)      (mc.mc_gpregs.gp_lr)
#define MCREG_Sp(mc)      (mc.mc_gpregs.gp_sp)
#define MCREG_Pc(mc)      (mc.mc_gpregs.gp_pc)

As for the floating point access we will also need an extra code here. Take the following with a grain of salt, it might not be exactly right or might need extra change at another place too:

inline
struct fpregs* GetNativeSigSimdContext(native_context_t *mc)
{
    return &(mc->mc_fpregs);
}

inline
const struct fpregs* GetConstNativeSigSimdContext(const native_context_t *mc)
{
    return GetNativeSigSimdContext(const_cast<native_context_t*>(mc));
}

[janvorli]

Btw, this is based on https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/arm64/include/ucontext.h#L53

[sec]

@janvorli Thanks, I was looking at the same files and was about to post :) Can't find Pc and Fp in ucontext in structs btw.

Managed to get past some errors, but now a lot of:

[  6%] Built target eventpipe_objs
Consolidate compiler generated dependencies of target mdruntimerw_wks_obj
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: use of undeclared identifier 'BSDREG_Cpsr'
        ASSIGN_ALL_REGS
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
        ASSIGN_CONTROL_REGS \
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:137:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
        ASSIGN_REG(Cpsr)    \
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:333:62: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(registers.uc_mcontext) = BSDREG_##reg(ptrace_registers);
                                                             ^
<scratch space>:248:1: note: expanded from here
BSDREG_Cpsr
^
[  8%] Built target ceefgen_obj
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: use of undeclared identifier 'BSDREG_Fp'
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
        ASSIGN_CONTROL_REGS \
        ^
/usr/home/sec/runtime/src/coreclr/pal/src/thread/context.cpp:138:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
        ASSIGN_REG(Fp)      \
        ^
...

I haven't saw any existing BSDREG_X for defined (even for OSX)

I'm starting to think does this road make sense. Shouldn't this reuse existing ARM64 code and/or use machine specific libunwind for the rest?

[janvorli]

We cannot reuse the code, because the structure of the ucontext_t is different from all other platforms, using different names of data fields for the registers etc. That's why we are adding new ones.

[janvorli]

Can you please share the log from the clean build after you've applied my previous fix? It is not clear to me how come we are compiling code using the BSDREG_xxx stuff when the HAVE_BSD_REGS_T doesn't seem to be defined.

[sec]

build_log.txt freebsd_arm64_patch.txt

@janvorli Here's build output and git diff of everything I have tweaked :)

edit: accoring to CMakeOutput.txt, BSD_REG check passed:

Run Build Command(s):/usr/local/bin/gmake -f Makefile cmTC_c2fef/fast && /usr/local/bin/gmake -f CMakeFiles/cmTC_c2fef.dir/build.make CMakeFiles/cmTC_c2fef.dir/build gmake[1]: Entering directory '/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CMakeTmp' Building C object CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o /usr/local/bin/clang14 -D_FILE_OFFSET_BITS=64 -I/usr/local/include -fPIE -MD -MT CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o -MF CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o.d -o CMakeFiles/cmTC_c2fef.dir/BSD_REGS _T.c.o -c /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CheckTypeSize/BSD_REGS_T.c Linking C executable cmTC_c2fef /usr/local/bin/cmake -E cmake_link_script CMakeFiles/cmTC_c2fef.dir/link.txt --verbose=1 /usr/local/bin/clang14 -fuse-ld=lld -pie -Wl,--build-id=sha1 CMakeFiles/cmTC_c2fef.dir/BSD_REGS_T.c.o -o cmTC_c2fef gmake[1]: Leaving directory '/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/CMakeFiles/CMakeTmp' CMakeError.log CMakeOutput.log

edit 2: added CMakeFiles output, maybe it will help also

[janvorli]

Ah, I've forgotten that the check_type_size("struct reg" BSD_REGS_T) doesn't log anything. In that case, we need to add definitions for BSDREG_xxx stuff. The HAVE_BSD_REGS_T branch in the source assumes it is either x64 or x86. So in context.h, look for the #if HAVE_BSD_REGS_T and then #ifdef HOST_64BIT under that. Inside that #ifdef, we need to put the current stuff under #ifdef HOST_AMD64 and add #elif defined(HOST_ARM64) branch where we put the following:

#define BSDREG_X0(reg) BSD_REGS_STYLE(reg,X[0],x[0])
.
.
.
#define BSDREG_X29(reg) BSD_REGS_STYLE(reg,X[29],x[29])
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,Fp,fp)
#define BSDREG_Sp(reg) BSD_REGS_STYLE(reg,Sp,sp)
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
#define BSDREG_Lr(reg) BSD_REGS_STYLE(reg,Lr,lr)
#define BSDREG_Cpsr(reg) BSD_REGS_STYLE(reg,Spsr,spsr)

And then in src/coreclr/pal/src/configure.cmake, find elseif(CLR_CMAKE_TARGET_FREEBSD). In that branch, the set(BSD_REGS_STYLE "((reg).r_##rr)") has to be different for amd64 and arm64. For arm64, it needs to be set(BSD_REGS_STYLE "((reg).rr)"), because the naming convention of the registers is different there. So it should be:

if (CLR_CMAKE_HOST_ARCH_AMD64)
  set(BSD_REGS_STYLE "((reg).r_##rr)")
elseif(CLR_CMAKE_HOST_ARCH_ARM64)
  set(BSD_REGS_STYLE "((reg).rr)")
else()
  message(FATAL_ERROR "Unknown FreeBSD architecture")
endif()

[sec]

Thanks, will try that - any idea what about missing MCREG_Pc and MCREG_Fp - can't see those inside ucontext struct (or the name is not close to that) ?

[janvorli]

Ah, sorry, the Fp is X29, so it should be

#define MCREG_Fp(mc)      (mc.mc_gpregs.gp_x[29])

And the MCREG_Pc - hmm, that's strange. I can see that there is gp_lr and gp_elr. I think that maybe the gp_elr is in fact the Pc.

Also related to the x29 above, for the most recent change I've suggested, this is wrong:

#define BSDREG_X29(reg) BSD_REGS_STYLE(reg,X[29],x[29])
#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,Fp,fp)

It should be just

#define BSDREG_Fp(reg) BSD_REGS_STYLE(reg,X[29],x[29])

[Thefrank]

With everything so far: freebsd_arm_patch.txt

/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:339:9: error: no member named 'pc' in 'reg'
        ASSIGN_ALL_REGS
        ^~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:286:9: note: expanded from macro 'ASSIGN_ALL_REGS'
        ASSIGN_CONTROL_REGS \
        ^~~~~~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:141:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
        ASSIGN_REG(Pc)
        ^~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:333:62: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) MCREG_##reg(registers.uc_mcontext) = BSDREG_##reg(ptrace_registers);
                                                             ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<scratch space>:310:1: note: expanded from here
BSDREG_Pc
^
/home/frank/runtime/src/coreclr/pal/src/include/pal/context.h:1020:46: note: expanded from macro 'BSDREG_Pc'
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
                       ~~~~~~~~~~~~~~~~~~~~~~^~~
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/pal/src/config.h:158:44: note: expanded from macro 'BSD_REGS_STYLE'
#define BSD_REGS_STYLE(reg, RR, rr) ((reg).rr)
                                     ~~~~~ ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:491:13: error: no member named 'pc' in 'reg'
            ASSIGN_CONTROL_REGS
            ^~~~~~~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:141:9: note: expanded from macro 'ASSIGN_CONTROL_REGS'
        ASSIGN_REG(Pc)
        ^~~~~~~~~~~~~~
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:483:25: note: expanded from macro 'ASSIGN_REG'
#define ASSIGN_REG(reg) BSDREG_##reg(ptrace_registers) = lpContext->reg;
                        ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
<scratch space>:385:1: note: expanded from here
BSDREG_Pc
^
/home/frank/runtime/src/coreclr/pal/src/include/pal/context.h:1020:46: note: expanded from macro 'BSDREG_Pc'
#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Pc,pc)
                       ~~~~~~~~~~~~~~~~~~~~~~^~~
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/pal/src/config.h:158:44: note: expanded from macro 'BSD_REGS_STYLE'
#define BSD_REGS_STYLE(reg, RR, rr) ((reg).rr)
                                     ~~~~~ ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:600:9: error: unknown type name 'fpsimd_context'
        fpsimd_context* fp = GetNativeSigSimdContext(native);
        ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:744:15: error: unknown type name 'fpsimd_context'
        const fpsimd_context* fp = GetConstNativeSigSimdContext(native);
              ^
4 errors generated.

(Note: using Ubuntu 18.04 with FreeBSD AARCH64 crossroot)

[janvorli]

Ok, sounds like again elr vs pc. So can you please make this change?:

#define BSDREG_Pc(reg) BSD_REGS_STYLE(reg,Elr,elr)

Last thing is the floating point context. In context.cpp at line 743, we'll need to add before the #else:

#elif TARGET_FREEBSD
        const struct fpregs* fp = GetConstNativeSigSimdContext(native);
        if (fp)
        {
            lpContext->Fpsr = fp->fp_sr;
            lpContext->Fpcr = fp->fp_cr;
            for (int i = 0; i < 32; i++)
            {
                lpContext->V[i] = *(NEON128*) &fp->q[i];
            }
        }

Similarly at line 599, we need to add before the #else:

#elif TARGET_FREEBSD
        struct fpregs* fp = GetNativeSigSimdContext(native);
        if (fp)
        {
            fp->fp_sr = lpContext->Fpsr;
            fp->fp_cr = lpContext->Fpcr;
            for (int i = 0; i < 32; i++)
            {
                *(NEON128*) &fp->q[i] = lpContext->V[i];
            }
        }

[Thefrank]

Including above code:

/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:607:34: error: no member named 'q' in 'fpregs'
                *(NEON128*) &fp->q[i] = lpContext->V[i];
                             ~~  ^
/home/frank/runtime/src/coreclr/pal/src/thread/context.cpp:762:52: error: no member named 'q' in 'fpregs'
                lpContext->V[i] = *(NEON128*) &fp->q[i];
                                               ~~  ^
2 errors generated.

Not sure why its throwing that. usr/include/machine/ucontext.h on the crossrootfs has that member and seen (https://github.com/freebsd/freebsd-src/blob/373ffc62c158e52cde86a5b934ab4a51307f9f2e/sys/arm64/include/ucontext.h#L45)

[sec]

@janvorli Big thanks for help/pointers. I was able to compile coreclr part (there were some typos to fix):

Repo successfully built.
Product binaries are available at /usr/home/sec/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug

Also I've pushed those changes here https://github.com/dotnet/runtime/compare/main...sec:runtime:freebsd.arm64 Any way to test if all those register parts are working correctly (aka how to run the tests for the part that just got compiled) ?

@Thefrank I also took few changes from your patch regarding RID's - thanks :) Would you be able to try to cross compile under Linux? I've tried, but on my setup (Linux ARM64) it fails with:

Scanning dependencies of target coreclrpal
[  1%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o
[  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_tcpstate.c.o
In file included from /home/ubuntu/runtime/src/coreclr/pal/src/cruntime/file.cpp:21:
/home/ubuntu/runtime/src/coreclr/pal/src/include/pal/palinternal.h:158:10: fatal error: 'type_traits' file not found
#include <type_traits>
         ^~~~~~~~~~~~~
[  1%] Building C object libs-native/System.IO.Compression.Native/CMakeFiles/System.IO.Compression.Native-Static.dir/home/ubuntu/runtime/src/native/external/brotli/enc/backward_references.c.o
[  1%] Building C object libs-native/System.Native/CMakeFiles/System.Native-Static.dir/pal_threading.c.o
1 error generated.
make[2]: *** [pal/src/CMakeFiles/coreclrpal.dir/build.make:76: pal/src/CMakeFiles/coreclrpal.dir/cruntime/file.cpp.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:2020: pal/src/CMakeFiles/coreclrpal.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....

Also if you'll take my branch, those can also be needed - don't know if those are needed or it's just my setup:

diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index d5dfc13504b..478029410b7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -133,7 +133,11 @@ elseif(FREEBSD)
     set(CMAKE_CXX_COMPILER_TARGET ${triple})
     set(CMAKE_ASM_COMPILER_TARGET ${triple})
     set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
-    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+    if(CLR_CMAKE_HOST_OS STREQUAL FreeBSD)
+      set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+    else()
+      set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -stdlib=libc++ -Qunused-arguments")
+    endif()
     set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld")
     set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -fuse-ld=lld")
 elseif(ILLUMOS)
diff --git a/eng/native/configuretools.cmake b/eng/native/configuretools.cmake
index 6697524c659..b44bcc3417c 100644
--- a/eng/native/configuretools.cmake
+++ b/eng/native/configuretools.cmake
@@ -54,7 +54,11 @@ if(NOT WIN32 AND NOT CLR_CMAKE_TARGET_BROWSER)
       set(TOOLSET_PREFIX ${ANDROID_TOOLCHAIN_PREFIX})
     elseif(CMAKE_CROSSCOMPILING AND NOT DEFINED CLR_CROSS_COMPONENTS_BUILD AND
         CMAKE_SYSTEM_PROCESSOR MATCHES "^(armv8l|armv7l|armv6l|aarch64|arm|s390x|ppc64le)$")
-      set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+      if(FREEBSD)
+        set(TOOLSET_PREFIX "llvm-")
+      else()
+        set(TOOLSET_PREFIX "${TOOLCHAIN}-")
+      endif()
     else()
       set(TOOLSET_PREFIX "")
     endif()

What should be next steps to get full SDK - as under FreeBSD ARM64 I don't have bootstrapped one, I think I will move to AMD64 FreeBSD and try there to cross compile?

edit: @Thefrank for the error, it's fp_q and also sorry I wrongly tagged @wfurt :)

edit 2: cross build under FreeBSD is going well... does this mean I will have working SDK on FreeBSD-arm64? 🤞

[janvorli]

@sec I was happy to help. As for testing whether these changes are correct, running coreclr tests is the best way. These code paths are exercised by exception handling. On the target, you can build all coreclr tests (in checked flavor, which means mostly optimized code, but including asserts in PAL / runtime) using

src/tests/build.sh checked -priority1

(Please note that you have to have the runtime clr+libs part built there before building the tests.)

And then run them using

src/tests/run.sh checked

[sec]

Hm, not that easy @janvorli :) While doing crosscompile under FreeBSD-amd64, using command ROOTFS_DIR=/home/sec/arm64_rootfs eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false -subset Clr.Native+Host.Native it fails after soem time:

  Repo successfully built.
  Product binaries are available at /usr/home/sec/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug
  Executing "/usr/home/sec/runtime/src/coreclr/build-runtime.sh" -arm64 -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -debug -ci -os FreeBSD -hostarch x64 -component crosscomponents
  Commencing CoreCLR Repo build
  __DistroRid: freebsd-arm64
  Setting up directories for build
  Checking prerequisites...
  Commencing build of "  crosscomponents " target in "CoreCLR component" for FreeBSD.arm64.Debug in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
  Invoking "/usr/home/sec/runtime/eng/native/gen-buildsys.sh" "/usr/home/sec/runtime/src/coreclr" "/usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64" x64 clang Debug ""  -DCLR_CMAKE_TARGET_ARCH=arm64 -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCLR_CROSS_COMPONENTS_BUILD=1
  /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64 /usr/home/sec/runtime/src/coreclr
  Not searching for unused variables given on the command line.
  -- The C compiler identification is Clang 10.0.1
  -- The CXX compiler identification is Clang 10.0.1

...

  [  2%] Linking CXX static library libgcinfo_universal_arm64.a
  In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh.cpp:407:
  In file included from /usr/home/sec/runtime/src/coreclr/pal/src/exception/seh-unwind.cpp:34:
  /home/sec/arm64_rootfs/usr/local/include/libunwind.h:25:11: fatal error: 'libunwind-x86_64.h' file not found
  # include "libunwind-x86_64.h"
            ^~~~~~~~~~~~~~~~~~~~
  [  2%] Built target gcinfo_universal_arm64
  [  2%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/file/directory.cpp.o
  1 error generated.
  --- pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o ---
  *** [pal/src/CMakeFiles/coreclrpal.dir/exception/seh.cpp.o] Error code 1

  make[3]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
  --- CMakeFiles/crosscomponents.dir/rule ---
  *** [CMakeFiles/crosscomponents.dir/rule] Error code 6

  make[1]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
  1 error

  make[1]: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
  *** [crosscomponents] Error code 2

  make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64

  make: stopped in /usr/home/sec/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/x64
  1 error
  /usr/home/sec/runtime/src/coreclr

What's going on in here? @Thefrank how's your Linux crossbuild?

[janvorli]

The cross components that are failing to build are compiled for x64, so they should be built on the host without the rootfs usage. It seems like there is some problem causing that the rootfs is still being used.

[Thefrank]

So for my Ubuntu 18.04 AMD64 -> FreeBSD AARCH64 an old bug has presented itself: [edit: ROOTFS_DIR=/home/frank/freebsdarm/ ./src/coreclr/build-runtime.sh -arm64 --clang11 -cross -os FreeBSD]

/home/frank/runtime/src/coreclr/debug/daccess/daccess.cpp:6996:38: error: invalid suffix 'x' on integer constant
    *dacTableAddress = baseAddress + DAC_TABLE_RVA;
                                     ^
/home/frank/runtime/artifacts/obj/coreclr/FreeBSD.arm64.Debug/inc/dactablerva.h:1:24: note: expanded from macro 'DAC_TABLE_RVA'
#define DAC_TABLE_RVA 0x          c06bb8
                       ^
1 error generated.

Previously under net6, this could be worked around in eng/native/gen-buildsys.sh

diff --git a/eng/native/gen-buildsys.sh b/eng/native/gen-buildsys.sh
index bf04c26f2b1..7d8793dd92c 100755
--- a/eng/native/gen-buildsys.sh
+++ b/eng/native/gen-buildsys.sh
@@ -97,7 +97,7 @@ if [[ "$build_arch" == "wasm" ]]; then
     cmake_command="emcmake $cmake_command"
 fi

-cmake_args_to_cache="$scan_build\n$SCAN_BUILD_COMMAND\n$generator\n$__UnprocessedCMakeArgs"
+cmake_args_to_cache=$(printf "$scan_build${SCAN_BUILD_COMMAND:+ }${SCAN_BUILD_COMMAND} $generator $__UnprocessedCMakeArgs")
 cmake_args_cache_file="$2/cmake_cmd_line.txt"
 if [[ -z "$__ConfigureOnly" ]]; then
     if [[ -e "$cmake_args_cache_file" ]]; then

This whole codeblock was removed in https://github.com/dotnet/runtime/commit/d22aceff4b4b83672cfaaab92d9977445213f9f5 so this hacky-workaround won't work.

[janvorli]

The contents of the dactablerva.h is generated by this script: https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/tools/gen-dactable-rva.sh Sounds like this generates the extra spaces, so we can fix it there: https://github.com/dotnet/runtime/blob/879f201ceff095ff59a1f6d2373cee871d7aa102/src/coreclr/pal/tools/gen-dactable-rva.sh#L18

[sec]

@janvorli About that crosscomponents compile issue... Looking at the command "/usr/home/sec/runtime/src/coreclr/build-runtime.sh" -arm64 -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -debug -ci -os FreeBSD -hostarch x64 -component crosscomponents - there's no -cross so it shouldn't use cross fs, but it's using, so looks like it's env thing - anyway - if I run the command directly it fail with something more harder to fix (hope not):

Consolidate compiler generated dependencies of target cee_wks_obj
[ 21%] Building ASM object vm/wks/CMakeFiles/cee_wks_core.dir/__/arm64/asmhelpers.S.o
[ 21%] Building CXX object vm/wks/CMakeFiles/cee_wks_obj.dir/__/codeman.cpp.o
[ 21%] Building CXX object pal/src/CMakeFiles/coreclrpal.dir/map/map.cpp.o
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:9:5: error: unknown use of instruction mnemonic without a size suffix
    mov x0, lr
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:10:5: error: unknown use of instruction mnemonic without a size suffix
    ret lr
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:15:5: error: unknown use of instruction mnemonic without a size suffix
    mov x0, sp
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:16:5: error: unknown use of instruction mnemonic without a size suffix
    ret lr
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:21:5: error: invalid instruction mnemonic 'mrs'
    mrs x0, dczid_el0
    ^~~
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:22:5: error: unknown use of instruction mnemonic without a size suffix
    and x0, x0, 31
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:23:5: error: unknown use of instruction mnemonic without a size suffix
    ret lr
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:32:13: error: unknown token in expression
    mov w1, #0
            ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:33:16: error: expected ']' in brackets expression
    str w1, [x0, #0xc0]
               ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:35:16: error: expected ']' in brackets expression
    str lr, [x0, #((0xc0 +8)+8)]
               ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:38:5: error: unknown use of instruction mnemonic without a size suffix
    mov x1, sp
    ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:39:16: error: expected ']' in brackets expression
    str x1, [x0, #(0xc0 +8)]
               ^
/usr/home/sec/runtime/src/coreclr/vm/arm64/asmhelpers.S:42:17: error: unknown token in expression
    add x1, x0, #0x0

... a lot more of those errors ...

edit: running ./src/coreclr/build-runtime.sh -arm64 -hostarch arm64 -component crosscomponents -cmakeargs -DCLR_CROSS_COMPONENTS_BUILD=1 on FreeBSD-arm64 fail with

Consolidate compiler generated dependencies of target coreclr
[ 94%] Linking CXX shared library libcoreclr.so
ld.lld: error: unable to find library -lSystem.Globalization.Native-Static
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)
--- dlls/mscoree/coreclr/libcoreclr.so ---
*** [dlls/mscoree/coreclr/libcoreclr.so] Error code 1

Albo native libs compile fine under ARM64 host, but executing ./src/coreclr/build-runtime.sh fail with:

[ 94%] Linking CXX shared library libcoreclr.so
ld.lld: error: unable to find library -lSystem.Globalization.Native-Static
[ 94%] Building CXX object jit/CMakeFiles/clrjit_win_x86_arm64.dir/typeinfo.cpp.o
clang-14: error: linker command failed with exit code 1 (use -v to see invocation)

and

sec@amper:~/runtime % ls artifacts/bin/native/FreeBSD-arm64-Debug/
./                                                     libSystem.IO.Compression.Native.so.dbg                 libSystem.Net.Security.Native.a
../                                                    libSystem.IO.Ports.Native.a                            libSystem.Net.Security.Native.so*
libSystem.Globalization.Native.a                       libSystem.IO.Ports.Native.so*                          libSystem.Net.Security.Native.so.dbg
libSystem.Globalization.Native.so*                     libSystem.IO.Ports.Native.so.dbg                       libSystem.Security.Cryptography.Native.OpenSsl.a
libSystem.Globalization.Native.so.dbg                  libSystem.Native.a                                     libSystem.Security.Cryptography.Native.OpenSsl.so*
libSystem.IO.Compression.Native.a                      libSystem.Native.so*                                   libSystem.Security.Cryptography.Native.OpenSsl.so.dbg
libSystem.IO.Compression.Native.so*                    libSystem.Native.so.dbg

edit: trying to build those components under Linux is working fine - maybe this is something wrong with cmakefiles def's ? :/

[sec]

Tried to use Linux for crosscompile

• RVA is caused I think when you have bash version <5.1 - dirty hack was to remove 016 from the format string in script
• eng/common/cross/toolchain.cmake need a fix when setting FreeBSD triple x86_64 -> aarch64

but even then it fails, at mono stage, so that's the far most progress:

  [31/351] Building C object mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o
  FAILED: mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o
  /usr/bin/clang-10 --target=aarch64-unknown-freebsd12 --sysroot=/home/linux/arm64/ -DHAVE_CONFIG_H -I/home/linux/runtime/artifacts/obj -I/home/linux/runtime/src/native -Imono/mini -Imono/mini/../.. -Imono/mini/../../mono/eglib -I/home/linux/runtime/src/mono/mono/mini/../.. -I/home/linux/runtime/src/mono/mono/mini/.. -I/home/linux/runtime/src/mono/mono/mini/../eglib -I/home/linux/runtime/src/mono/mono/mini/../sgen -I/home/linux/runtime/src/mono/mono/mini/../eventpipe -Imono/mini/eventpipe -I/home/linux/runtime/src/native/public/. -std=gnu99 -fno-strict-aliasing -fwrapv -Wall -Wunused -Wmissing-declarations -Wpointer-arith -Wno-cast-qual -Wwrite-strings -Wno-switch -Wno-switch-enum -Wno-unused-value -Wno-attributes -Wno-format-zero-length -Wno-unused-function -Qunused-arguments -Wno-tautological-compare -Wno-parentheses-equality -Wno-self-assign -Wno-return-stack-address -Wno-constant-logical-operand -Wno-zero-length-array -Wno-asm-operand-widths -Wmissing-prototypes -Wstrict-prototypes -Wnested-externs -Werror=return-type -Werror=implicit-function-declaration -Werror=incompatible-pointer-types -g   -g -fPIC -fvisibility=hidden -ffp-contract=off -MD -MT mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o -MF mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o.d -o mono/mini/CMakeFiles/debugger-objects.dir/__/component/debugger-agent.c.o   -c /home/linux/runtime/src/mono/mono/component/debugger-agent.c
  <inline asm>:1:6: error: invalid token in expression
          int $03
              ^
  <inline asm>:1:6: error: invalid operand
          int $03
              ^
  2 errors generated.
  ninja: build stopped: subcommand failed.
/home/linux/runtime/src/mono/mono.proj(541,5): error MSB3073: The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.
##vso[task.logissue type=error;sourcepath=/home/linux/runtime/src/mono/mono.proj;linenumber=541;columnnumber=5;code=MSB3073;](NETCORE_ENGINEERING_TELEMETRY=Build) The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.

Build FAILED.

/home/linux/runtime/src/mono/mono.proj(541,5): error MSB3073: The command "TARGET_BUILD_ARCH=arm64 PKG_CONFIG_PATH=/home/linux/arm64//usr/lib/aarch64-linux-gnu/pkgconfig TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug" exited with code 1.
    0 Warning(s)
    1 Error(s)

Commands I used:

• ROOTFS_DIR=/home/linux/arm64/ ./eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false -subset Clr.Native+Host.Native - compile fine
• ROOTFS_DIR=/home/linux/arm64/ ./eng/build.sh -ci -c Debug -cross -os FreeBSD -a arm64 /p:IsEligibleForNgenOptimization=false /p:AppHostSourcePath=/home/linux/runtime/artifacts/obj/freebsd-arm64.Debug/apphost/standalone/apphost - failed

[Thefrank]

• RVA is caused I think when you have bash version <5.1 - dirty hack was to remove 016 from the format string in script

  frank@buildbot:~/runtime$ bash --version
  GNU bash, version 5.0.17(1)-release (x86_64-pc-linux-gnu)

  O.K. /sigh/ removing that deleting the old artifacts/obj/coreclr/FreeBSD.arm64.Debug/inc/dactablerva.h and rerunning finished! the new file contained #define DAC_TABLE_RVA 0xc06bb8

  Repo successfully built.
  Product binaries are available at /home/frank/runtime/artifacts/bin/coreclr/FreeBSD.arm64.Debug

  It also makes it past the cross components but runs into the same issue for mono.

Which is interesting because...FreeBSD doesn't have pkg-config as a part of the base OS. It's in ports (as pkgconf). That path looks like it gets added from the mono.proj

[Thefrank]

TOP EDIT: almost forgot to give credit to @janvorli for all the help It's built!

Build succeeded.
    0 Warning(s)
    0 Error(s)

Time Elapsed 00:43:38.40

I just have no way to test it as...I don't have any ARM64 hardware sitting around. Posting patch + binaries shortly

EDIT1: freebsd_arm_patch.txt

EDIT2: https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/V7.0-ARM64-EXPERIMENTAL

EDIT3: build command ROOTFS_DIR=/home/frank/freebsd13arm/ ./build.sh -a arm64 --clang11 -cross -os FreeBSD

EDIT4: Tomorrow ill look into ASPNET and Installer, both of which should be much easier than runtime

[sec]

@Thefrank @janvorli

I was wrong about RVa issue - I have same version of bash and it format diffrently - but made a fix for that - https://github.com/sec/runtime/commit/1a6d28850f973de8a0713e785996f656389b27f8

Also all the other fixes (with some not needed removed) are integrated into my branch - would it be feasible to make PR for that at this stage @wfurt ?

Thanks to last mono fix - I was also able to crosscompile Linux x64 -> FreeBSD arm64

Linux arm64 -> FreeBSD arm64 - also fine, with small patch (edit: looks it's no longer need, maybe becuase clang11 was used, but I will leave it here for future reference anyway):

diff --git a/eng/common/cross/toolchain.cmake b/eng/common/cross/toolchain.cmake
index d5dfc13504b..478029410b7 100644
--- a/eng/common/cross/toolchain.cmake
+++ b/eng/common/cross/toolchain.cmake
@@ -133,7 +133,11 @@ elseif(FREEBSD)
     set(CMAKE_CXX_COMPILER_TARGET ${triple})
     set(CMAKE_ASM_COMPILER_TARGET ${triple})
     set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
-    set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+    if(CLR_CMAKE_HOST_OS STREQUAL FreeBSD)
+      set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld")
+    else()
+      set(CMAKE_EXE_LINKER_FLAGS "${CMAKE_EXE_LINKER_FLAGS} -fuse-ld=lld -stdlib=libc++ -Qunused-arguments")
+    endif()
     set(CMAKE_SHARED_LINKER_FLAGS "${CMAKE_SHARED_LINKER_FLAGS} -fuse-ld=lld")
     set(CMAKE_MODULE_LINKER_FLAGS "${CMAKE_MODULE_LINKER_FLAGS} -fuse-ld=lld")
 elseif(ILLUMOS)

FreeBSD x64 -> FreeBSD arm64 - two issues:

• on first run when it try to build crosscomponents - looks like crossrootfs is used
• on manual re-run of the command used to build those - there are some errors in /tmp/asmhelpers-8f0fb3.s (more above)

FreeBSD arm64:

• ./src/coreclr/build-runtime.sh - ok
• ./src/native/libs/build-native.sh - ok
• don't know if building mono is possible without working runtime first (found https://github.com/dotnet/runtime/blob/main/docs/workflow/building/libraries/cross-building.md#build-corefx-for-a-new-architecture but it not enough information to me)

NOW, testing runtime under FreeBSD arm64 host...

[sec@amper ~]$ ~/dotnet7/dotnet --info

Host:
  Version:      7.0.0-dev
  Architecture: arm64
  Commit:       N/A

.NET SDKs installed:
  No SDKs were found.

.NET runtimes installed:
  Microsoft.NETCore.App 7.0.0-dev [/usr/home/sec/dotnet7/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Tried to make simple console app, restore, build, publish, copy to arm64 host, adjust json to use 7.0.0-dev and

[sec@amper ~/runtime/console_app]$ ~/dotnet7/dotnet bin/Debug/net6.0/publish/console_app.dll
Failed to create CoreCLR, HRESULT: 0x80004005

Attaching COREHOST trace file - hope it's something trivial to fix :) host.txt

[wfurt]

yes, I think so @sec. main branch for 7.0 is still generally open (closing soon). I'm on the road so my capacity is small for next 2-3 weeks.

[Thefrank]

@sec if run in jail, does the jail have allow_mlock on? Does the jail/OS have a defined and writable TMP folder?

[sec]

Not running in jail and yes, there's a writeable /tmp folder.

[sec]

@janvorli any idea on how to debug this problem?

[Thefrank]

truss work under ARM64? Might give that a shot

[sec]

@Thefrank tried that already, nothing special, but attaching sample log truss_output.txt

[Thefrank]

attempt at full SDK. Still no way to test it:https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/7.0.100-preview.7.22353.99-freebsd-arm64

[sec]

@Thefrank nice - tried SDK, same result (one thing I notice is no runtime info about FreeBSD host etc., but that's why it fail right?):

sec@amper:~/sdk7 % ./dotnet --info
Failed to create CoreCLR, HRESULT: 0x80004005

Host:
  Version:      7.0.0-preview.6.22329.5
  Architecture: arm64
  Commit:       c9140d621a

.NET SDKs installed:
  7.0.100-preview.7.22353.99 [/usr/home/sec/sdk7/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.0-preview.7.22351.1 [/usr/home/sec/sdk7/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.0-preview.6.22329.5 [/usr/home/sec/sdk7/shared/Microsoft.NETCore.App]

Other architectures found:
  None

Environment variables:
  Not set

global.json file:
  Not found

Learn more:
  https://aka.ms/dotnet/info

Download .NET:
  https://aka.ms/dotnet/download

Output from SDK7 under x64 host:

.NET SDK:
 Version:   7.0.100-preview.5.22307.18
 Commit:    bd8b088037

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  13
 OS Platform: FreeBSD
 RID:         freebsd.13-x64
 Base Path:   /usr/home/sec/dotnet7/sdk/7.0.100-preview.5.22307.18/

global.json file:
  Not found

Host:
  Version:      7.0.0-preview.5.22301.12
  Architecture: x64
  Commit:       425fedc0fb

.NET SDKs installed:
  7.0.100-preview.5.22307.18 [/usr/home/sec/dotnet7/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.0-preview.5.22303.8 [/usr/home/sec/dotnet7/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.0-preview.5.22301.12 [/usr/home/sec/dotnet7/shared/Microsoft.NETCore.App]

@Thefrank could you share the steps/any patches that you used to make this - maybe we could use this to automate x64 builds and publish them on https://github.com/dotnet/runtimelab ?

Does anyone have any expierence in troubleshoting root cuase of this CLR fail - maybe something from https://github.com/dotnet/diagnostics will be to good use?

[Thefrank]

@Thefrank nice - tried SDK, same result (one thing I notice is no runtime info about FreeBSD host etc., but that's why it fail right?):

From what was posted it does not appear to be making it very far. Now WHY that is happening is beyond me

@Thefrank could you share the steps/any patches that you used to make this - maybe we could use this to automate x64 builds and publish them on https://github.com/dotnet/runtimelab ?

Runtime builds from head with only a quick sed patch to make it use a prebuilt SDK and that is how I do nightlies+tests for FreeBSD-x64

Building a WHOLE SDK requires a bit more tweaks but is rather easy from TAG.

However, if you want to build from a random commit, well, that takes a bit more work. For that I usually work backwards; examine what BUILDIDs dotnet/installer wants for dotnet/runtime and dotnet/aspnetcore, reverse them into YYYYMMDD.# format and plug them into the builds first for runtime and second for aspnetcore.

EDIT: Does lldb provide any insights? Is it even making it to managed code?

[sec]

@Thefrank I was rather asking, how to you build e. g. installer repo in cross mode, as I didn't saw any swithces just for that (and --runtime-id freebsd-x64 /p:OSName=freebsd prepare SDK with correct names, but Linux inside) - but that's offtopic :)

As for lldb - not an export on this one, but doesn't look like it's getting into managed code - tried to use corerun with bogus parameters and it's throwing the same error before even checking if given file exists.

edit: I may be wrong, but maybe it's connected with fact, that cross build from FreeBSD-x64 host fail on cross components stage with some assembler compiler error (but is fine when doing the same from Linux host)?

[sec]

Had fun with lldb, learning at 2am is great :) The code looks like it's failing when loaded/loading System.Private.CoreLib and then going this route:

* thread #1, name = 'corerun', stop reason = breakpoint 7.1
    frame #0: 0x0000000041544ce8 libcoreclr.so`Assembly::Init(this=0x0000000040c11650, pamTracker=0x0000ffffffffccf0, pLoaderAllocator=0x0000000000000000) at assembly.cpp:196:53
   193          // manifest modules of dynamic assemblies are always transient
   194          m_pModule = ReflectionModule::Create(this, pPEAssembly, pamTracker, REFEMIT_MANIFEST_MODULE_NAME);
   195      else
-> 196          m_pModule = Module::Create(this, mdFileNil, pPEAssembly, pamTracker);
   197
   198      InterlockedIncrement((LONG*)&g_cAssemblies);
   199

after some steps

Process 25614 stopped
* thread #1, name = 'corerun', stop reason = step over
    frame #0: 0x000000004156461c libcoreclr.so`Module::Initialize(this=0x0000000042224000, pamTracker=0x0000ffffffffccf0, szName=0x0000000000000000) at ceeload.cpp:443:63
   440
   441  #ifdef FEATURE_READYTORUN
   442      m_pNativeImage = NULL;
-> 443      if ((m_pReadyToRunInfo = ReadyToRunInfo::Initialize(this, pamTracker)) != NULL)
   444      {
   445          m_pNativeImage = m_pReadyToRunInfo->GetNativeImage();
   446          if (m_pNativeImage != NULL)
(lldb)
BEGIN: coreclr_initialize failed - Error: 0x80004005

time for bed...

edit: I think final fail is here:

Process 26961 stopped
* thread #1, name = 'corerun', stop reason = step over
    frame #0: 0x000000004172268c libcoreclr.so`ReadyToRunInfo::Initialize(pModule=0x0000000042254000, pamTracker=0x0000ffffffffccf0) at readytoruninfo.cpp:561:9
   558      if (!pLayout->IsNativeMachineFormat())
   559      {
   560          // For CoreCLR, be strict about disallowing machine mismatches.
-> 561          COMPlusThrowHR(COR_E_BADIMAGEFORMAT);
   562      }

Is this something wrong with cross compiled runtime/sdk ?

It's not libcoreclr.so fault for sure

sec@amper:~/runtime % file artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so
artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so: ELF 64-bit LSB shared object, ARM aarch64, version 1 (FreeBSD), dynamically linked, for FreeBSD 13.1, BuildID[sha1]=3d30f6f528036a298d342a15af8b3b1e47af6a09, stripped
sec@amper:~/runtime % ldd artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so
artifacts/bin/coreclr/FreeBSD.arm64.Debug/libcoreclr.so:
        libthr.so.3 => /lib/libthr.so.3 (0x4195e000)
        librt.so.1 => /usr/lib/librt.so.1 (0x419ba000)
        libunwind-aarch64.so.8 => /usr/local/lib/libunwind-aarch64.so.8 (0x419ef000)
        libunwind.so.8 => /usr/local/lib/libunwind.so.8 (0x41a51000)
        libc++.so.1 => /usr/lib/libc++.so.1 (0x41aae000)
        libcxxrt.so.1 => /lib/libcxxrt.so.1 (0x41ba9000)
        libm.so.5 => /lib/libm.so.5 (0x41bf5000)
        libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x41c63000)
        libc.so.7 => /lib/libc.so.7 (0x401e5000)
        liblzma.so.5 => /usr/lib/liblzma.so.5 (0x41ccc000)
        libz.so.6 => /lib/libz.so.6 (0x41d21000)
        libmd.so.6 => /lib/libmd.so.6 (0x41d68000)

So it's this one that's faulty?

sec@amper:~/runtime % file artifacts/bin/coreclr/FreeBSD.arm64.Debug/System.Private.CoreLib.dll
artifacts/bin/coreclr/FreeBSD.arm64.Debug/System.Private.CoreLib.dll: PE32+ executable (DLL) (console) Mono/.Net assembly, for MS Windows

When doing cross build Linux -> FreeBSD arm64, is this ok on Linux host - shouldn't this be "freebsd-arm64" by any chance?

./artifacts/bin/coreclr/FreeBSD.arm64.Debug/crossgen2/linux-arm64/System.Private.CoreLib.dll
./artifacts/bin/coreclr/FreeBSD.arm64.Debug/crossgen2/linux-arm64/publish/System.Private.CoreLib.dll

Checking Microsoft.NETCore.App.Crossgen2.freebsd-arm64.7.0.0-dev.nupkg there are Linux binaries inside:

sec@amper:~/newb/tools % file crossgen2
crossgen2: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=725302cc1678f04d8935457644ebd9deeb09f8ab, stripped

sec@amper:~/newb/tools % file dotnet
dotnet: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=eb185bf4cb8984561b845c57ea43484c3ed2b40a, stripped

sec@amper:~/newb/tools % file corerun
corerun: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 13.0 (1300139), FreeBSD-style, BuildID[sha1]=da7462690d3094983f41df3114505547a61835c8, with debug_info, not stripped

[sec]

@Thefrank @janvorli @jkotas Maybe this is silly, but managed to get runtime/SDK to work (at some point at least). What I did was

• took Crossgen2 package built for FreeBSD-x64 (maybe any will work)
• took System.Private.CoreLib.dll from cross compiled arm64 runtime/SDK
• run tools/crossgen2 System.Private.CoreLib.dll --targetarch arm64 --out System.Private.CoreLib.dll.2 on it
• replaced the original with new one

And

sec@amper:~/sdk7 % ./dotnet --info
.NET SDK:
 Version:   7.0.100-preview.7.22353.99
 Commit:    be72bcc21b

Runtime Environment:
 OS Name:     FreeBSD
 OS Version:  13
 OS Platform: FreeBSD
 RID:         freebsd.13-arm64
 Base Path:   /usr/home/sec/sdk7/sdk/7.0.100-preview.7.22353.99/

Host:
  Version:      7.0.0-preview.6.22329.5
  Architecture: arm64
  Commit:       c9140d621a

.NET SDKs installed:
  7.0.100-preview.7.22353.99 [/usr/home/sec/sdk7/sdk]

.NET runtimes installed:
  Microsoft.AspNetCore.App 7.0.0-preview.7.22351.1 [/usr/home/sec/sdk7/shared/Microsoft.AspNetCore.App]
  Microsoft.NETCore.App 7.0.0-preview.6.22329.5 [/usr/home/sec/sdk7/shared/Microsoft.NETCore.App]

Run simple console app published from Windows host:

sec@amper:~/dotnet7 % ./dotnet ~/runtime/console_app/bin/Debug/net6.0/publish/console_app.dll
Hello, World!

But:

sec@amper:~/temp % ~/sdk7/dotnet new console
Segmentation fault (core dumped)

More info:

sec@amper:~/temp % lldb14 ~/sdk7/dotnet new console
(lldb) target create "/home/sec/sdk7/dotnet"
Current executable set to '/home/sec/sdk7/dotnet' (aarch64).
(lldb) settings set -- target.run-args  "new" "console"
(lldb) r
Process 27236 launched: '/home/sec/sdk7/dotnet' (aarch64)
Process 27236 stopped
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
    frame #0: 0x0000000041859f6c libunwind.so.8`___lldb_unnamed_symbol150 + 32
libunwind.so.8`___lldb_unnamed_symbol150:
->  0x41859f6c <+32>: ldr    x8, [x1]
    0x41859f70 <+36>: mov    w0, wzr
    0x41859f74 <+40>: str    x8, [x2]
    0x41859f78 <+44>: ldr    x19, [sp, #0x20]
(lldb) bt
* thread #1, name = 'dotnet', stop reason = signal SIGSEGV: invalid address (fault address: 0x0)
  * frame #0: 0x0000000041859f6c libunwind.so.8`___lldb_unnamed_symbol150 + 32
    frame #1: 0x000000004185af70 libunwind.so.8`_ULaarch64_step + 568
    frame #2: 0x000000004178e8ec libcoreclr.so`___lldb_unnamed_symbol14794 + 256
    frame #3: 0x00000000415b29ec libcoreclr.so`___lldb_unnamed_symbol7901 + 256
    frame #4: 0x00000000413f75a0 libcoreclr.so`___lldb_unnamed_symbol2153 + 136
    frame #5: 0x00000000413f7508 libcoreclr.so`___lldb_unnamed_symbol2152 + 28
    frame #6: 0x00000000415b8400 libcoreclr.so`___lldb_unnamed_symbol8017 + 84
    frame #7: 0x00000000415b7db0 libcoreclr.so`___lldb_unnamed_symbol8013 + 752
    frame #8: 0x00000000415b7198 libcoreclr.so`___lldb_unnamed_symbol8011 + 1108
    frame #9: 0x00000000415bb444 libcoreclr.so`___lldb_unnamed_symbol8050 + 336
    frame #10: 0x00000000415bb71c libcoreclr.so`___lldb_unnamed_symbol8051 + 68
    frame #11: 0x0000000041518964 libcoreclr.so`___lldb_unnamed_symbol6091 + 540

@Thefrank makes sense to build SDK in Debug to see where/why it's SIGSEGV or this is easy to fix now? :)

[Thefrank]

Crossgen2 packages, AFAIK, are no longer generated for FreeBSD as that ability got removed somewhere after net7p3. This is not ideal as those packages are needed for native builds (restore requires it)

Crossbuilt net7p5 for FreeBSD x64 with the package added back in looks like it generates incorrectly too:

Shipping/Microsoft.NETCore.App.Crossgen2.freebsd-x64.7.0.0-preview.5.22301.12/tools # ls | xargs file
coreclr_delegates.h:                               C source, ASCII text
corerun:                                           ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.3, FreeBSD-style, BuildID[sha1]=ea37edd26732ab0d86c7eff56bfd2dc4b97d7de5, with debug_info, not stripped
crossgen2:                                         ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, interpreter /libexec/ld-elf.so.1, for FreeBSD 12.3, FreeBSD-style, BuildID[sha1]=f2abf1f880cce1b944515bb44ccf8778bd1e0d4d, stripped
dotnet:                                            ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=342e6dc49a1ef7d664ecf66554bd91f12377deb8, stripped
hostfxr.h:                                         C source, ASCII text
libclrjit_universal_arm_x64.so:                    ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=5371055af15bb23ddab55f6d56ea05e68c4ec014, stripped
libclrjit_universal_arm64_x64.so:                  ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=df1c176a5aedfe127ee1097723e5a9dd1ca0c9d6, stripped
libclrjit_unix_x64_x64.so:                         ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=c0d6411b4d9dc86718543182049e390bebcc75c0, stripped
libclrjit_win_x64_x64.so:                          ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=c8498128119ad328a63433b0e113bf35a28b67b8, stripped
libclrjit_win_x86_x64.so:                          ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e1ef1e6c1d44175ebe3f2dd1ed7a22bae89dd3ea, stripped
libclrjit.so:                                      ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=b3050876013379e1abd2c8ab67a3b649c901550c, stripped
libcoreclr.so:                                     ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=beb05fea73aadf25e83d75f7cb06a638becb31bd, stripped
libhostfxr.a:                                      current ar archive
libhostfxr.so:                                     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=955d0bd7775d52bc8aba0124aa2e6f7b0e059ae1, stripped
libhostpolicy.so:                                  ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=bd4f445a772be690487d98d63be8aa0d7013dad0, stripped
libjitinterface_x64.so:                            ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=7aca7857d900236e5fd0925419afd2a1510541f5, stripped
libmscordaccore.so:                                ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=fae13f6b4f0a887250f6e2f9bbfa986c0b493edd, stripped
libmscordbi.so:                                    ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e07ae986afb11b697f758994cb3dd069fe6cd801, stripped
libnethost.a:                                      current ar archive
libnethost.so:                                     ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, BuildID[sha1]=fb0e1694b1521c04c74755a2210c8b8988298f81, stripped
libSystem.Globalization.Native.a:                  current ar archive
libSystem.Globalization.Native.so:                 ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=411160e3995dfc8aa04d8929dc5958ae61e4d6c4, stripped
libSystem.IO.Compression.Native.a:                 current ar archive
libSystem.IO.Compression.Native.so:                ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e539b4dd805290e3a19c8551ab766cc7d7d0bafd, stripped
libSystem.IO.Ports.Native.a:                       current ar archive
libSystem.IO.Ports.Native.so:                      ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=d41c2ed5987fd9f7ffe8f03c853673dc24f8a397, stripped
libSystem.Native.a:                                current ar archive
libSystem.Native.so:                               ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=e732c3d8b2d37c4584bcd8470f018a1493394bc0, stripped
libSystem.Net.Security.Native.a:                   current ar archive
libSystem.Net.Security.Native.so:                  ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=3ecfe261dd743c77c13bc8ed53a811d1e0c4c6c3, stripped
libSystem.Security.Cryptography.Native.OpenSsl.a:  current ar archive
libSystem.Security.Cryptography.Native.OpenSsl.so: ELF 64-bit LSB shared object, x86-64, version 1 (FreeBSD), dynamically linked, BuildID[sha1]=b3f59bd1083874d1c0db7fc528b3c8452d11be21, stripped
nethost.h:                                         C source, ASCII text
singlefilehost:                                    ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, for GNU/Linux 2.6.32, BuildID[sha1]=7d2abe4df7dfc9c4f875865784d606267bc42c77, stripped,

net6RTM (the last crossbuild from that branch I have done) only contains one executable: crossgen2 and that is correctly FreeBSD

EDIT: did a quick crossbuild from Linux x86_64 -> FreeBSD ARM64. The contents of the Crossgen2 nupkg (and tarball) are just as odd: everything but dotnet and singlefilehost are FreeBSD ARM aarch64. Those two are Linux ARM aarch64

[sec]

Broken System.Private.CoreLib those make it work OOTB (for runtime part) - https://github.com/dotnet/runtime/pull/71486/commits/951c92afc499fddb19846bfd108bad5f3a086ae8 and https://github.com/dotnet/runtime/pull/71486/commits/98d39cd008c6160eb2feec60fad712d7b24661c1

As for SDK to fail somewhere inside libunwind - @Thefrank would you mind doing SDK build in Debug or share the steps I can reproduce to crossbuild it myself (so I won't bug you) ? :)

As for those Linux files inside packages - that's weird, becuase there is FreeBSD version of singlefilehost and dotnet produced during build, but the Linux one is being packed inside - all of the below have their correct FreeBSD, but are inside dotnet-crossgen2-7.0.0-dev-freebsd-arm64.tar.gz:

./singlefilehost:                                    ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=000c66b0f0a90e33362b538857a1fe07858d95d2, stripped
./libhostpolicy.so:                                  ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=f7f923736c5fa480faec666385f8ae7cb6191905, stripped
./libnethost.so:                                     ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=ec8c97ecfbcf5bb77d4fa8a5fcb3e56cb7bc13e4, stripped
./dotnet:                                            ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=eb185bf4cb8984561b845c57ea43484c3ed2b40a, stripped
./libhostfxr.so:                                     ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, BuildID[sha1]=827719e673ae0b16ade2b80907cf2c39e6ae517a, stripped

At least crossgen2 inside is FreeBSD, bad news it doesn't work:

Assert failure(PID 42015 [0x0000a41f], Thread: 105128 [0x19aa8]): pThread
    File: /home/ubuntu/runtime/src/coreclr/vm/threads.inl Line: 43
    Image: /usr/home/sec/temp/cros/crossgen2

Abort (core dumped)

[Thefrank]

@sec https://github.com/Thefrank/dotnet-freebsd-crossbuild/releases/tag/7.0.100-preview.7.22355.99-freebsd-arm64

I don't mind making the SDKs. This also reminds me that I should update my build script (from the above repo) to allow for arm64 builds.

For platforms like FreeBSD, the installer branch is basically a fancy fetch+sed+tar... unless you want to generate native binaries...which we don't under a crossbuild, as afaik, does not work for FreeBSD.

[sec]

@Thefrank Thanks, but can't find libcoreclr.so.dbg anywhere :) (my guess it's not packed into SDK) - so the backtrace is still unknown.

Maybe someone more expierenced in this topic could help to debug the issue (something with threads, as crossgen2 fail in there) ? :)

[Thefrank]

@sec check dotnet-runtime-7.0.0-preview.6.22329.5-freebsd-arm64.tar.gz

[sec]

@Thefrank Nope, libcoreclr.so.dbg is still not there:

sec@amper:~/sdk % tar --list --file dotnet-runtime-7.0.0-preview.6.22329.5-freebsd-arm64.tar.gz|grep libcoreclr
./shared/Microsoft.NETCore.App/7.0.0-preview.6.22329.5/libcoreclr.so

I don't think *.so.dbg are packed into tar's

[Thefrank]

I linked the wrong one dotnet-runtime-symbols-freebsd-arm64-7.0.0-preview.6.22329.5.tar.gz needs to say "symbols"

EDIT: @sec on a different note: do you have any suggestions on cheap arm64 hardware that freebsd runs on?