Haiku: add platform support

jessicah commented 3 years ago

Opening an issue to track work on adding platform support for Haiku: https://www.haiku-os.org/.

Any tips on how cross-compiling should work would be appreciated. Am currently starting with './build.sh mono+libs -arch x64 -os haiku -c debug -cross, and slowly adding Haiku targets to build and about to start some work on adding platform support insrc/libraries/Common/src/Interop`.

Thus far it hasn't required a functioning cross-compiler yet, have only just started.

dotnet-issue-labeler[bot] commented 3 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

En3Tho commented 3 years ago

What are benefits of having Haiku as an os target vs other os'es?

am11 commented 3 years ago

Here is a quick list without debugging shell or cmake scripts / how-to-get-there steps (if you need to know something specific, i can comment it separately):

add crossbuild toolchain support in: eng/common/cross/build-rootfs.sh and eng/common/cross/toolchain.cmake
update eng/native/configure*.cmake files for compiler options under eng/native/ directory
update groups with Calculate* labels in Directory.Build.props file at the root of this repo
for mono, update src/mono/mono.proj, src/mono/CMakeLists.txt and src/mono/Directory.Build.props files

Then run eng/common/cross/build-rootfs haiku x64 to build the rootfs enviornment and ROOTFS_DIR=$(pwd)/.tools/x64 ./build.sh -cross -arch x64 -os haiku to get started with code-only changes. From time to time, you may need to update local configurations of projects being ported, but the aforementioned central configuration take care of most of the wiring.

Later on if you want to make a persistent environment, you can get a docker tag from https://github.com/dotnet/dotnet-buildtools-prereqs-docker, with rootfs at a fixed location: /crossrootfs/x64. We did this step for illumos/SmartOS very early to make development environment stateless and build recipe sharable with other community members.

jessicah commented 3 years ago

Thanks @am11, is mono+libs the right target to start with? Also, what's the difference between mono in this repo, and https://github.com/mono/mono?

am11 commented 3 years ago

is mono+libs the right target to start with?

Yes, imo, after the common configurations are done, focusing first on mono and then libraries would be the fastest way to get .NET 6 working on Haiku. Going by git grep -i haiku, you can already tell there is some related code from the initial mono port in dotnet/runtime repo.

Later you can look into corehost (dontet(1)) to make the build binaries usable with .NET SDK etc. It would also make it much easier to then port coreclr runtime.

what's the difference between mono in this repo, and https://github.com/mono/mono?

mono/mono (OSS) aims to be compatible with .NET Framework 4.7 (closed source). Mono subset (src/mono) in this repository started off as a port of same mono runtime (minus the legacy components of .NET Framework 4.7, which are not supported in .NET 5) and implements common runtime interface as src/coreclr. The initial focus was to support various targets including, but not limited to, Xamrian, Mobile and WebAssembly that coreclr does not support today.

The core .NET libraries in this repo are tested against both (coreclr and mono) runtime flavors. The goal is to make both runtimes drop-in replacements of each other, and they can (and do) overlap when it comes to platform targets. Officially the focus for runtime/src/mono so far has been on mobile-like targets, but mono RT built from this repo runs desktop apps just fine (to do that it currently requires two manual cp steps which hopefully may no longer be needed in the future 😁).

jessicah commented 3 years ago

@am11 Running into a weird issue with configuration in src/libraries/Native:

https://github.com/jessicah/dotnet-runtime/blob/haiku/src/libraries/Native/Unix/System.Globalization.Native/CMakeLists.txt#L22

  CMake Error at System.Globalization.Native/CMakeLists.txt:24 (message):
    Cannot find utypes.h, try installing libicu-dev (or the appropriate package
    for your platform)

It feels like it's not searching the configured sysroot. Invoking build.sh as follows: ROOTFS_DIR=/home/jessica/build/haiku-sysroot ./build.sh mono+libs -arch x64 -os haiku -c debug -cross --ninja.

jessica@homecloud:~/build/haiku-sysroot/boot/system/develop/headers$ find -name utypes.h
./unicode/utypes.h

Not really sure what I'm doing wrong. All the checks being run in configure.cmake appear to be invoking the Haiku cross-compiler correctly. Unfortunately I can't find a log that mentions the utype.h search. It's definitely for cross-compiling, as libicu-dev is installed on host Linux.

Logs in artifacts/obj/native/net6.0-Haiku-Debug-x64/CMakeFiles: https://gist.github.com/jessicah/7f8693f773d7ddec8c51f2df15da0221

Additionally, if I comment out the search and just hard code them as succeeding, then in configure.cmake, the checks for HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS and HAVE_SET_MAX_VARIABLE both compile and pass, so the sysroot is configured correctly for compile/link, just the builtin CMake functions are not working correctly.

am11 commented 3 years ago

Could you try adding:

+++ eng/common/cross/toolchain.cmake
--- eng/common/cross/toolchain.cmake

elseif(CMAKE_SYSTEM_NAME STREQUAL "Haiku")
    set(CMAKE_SYSROOT "${CROSS_ROOTFS}")
+    include_directories(SYSTEM ${CROSS_ROOTFS}/boot/system/develop/headers)
+    set(CMAKE_SYSTEM_PREFIX_PATH "${CROSS_ROOTFS}")
else()

to see if it gives clean result. I remember having similar issue with ICU on illumos, and tweaking toolchain.cmake helped. Your observation is spot on, the flags and definitions used by cmake introspection are different than those used for the actual compilation. IIRC, that include_directories sets it for both.

NattyNarwhal commented 3 years ago

I did the PAL port and (attempt at) Mono runtime stuff in mono/mono; ping me if you need me.

jessicah commented 3 years ago

@am11 no luck, unfortunately :-/

am11 commented 3 years ago

I think it is ok to continue with a workaround at least for now. :)

Additionally, if I comment out the search and just hard code them as succeeding, then in configure.cmake, the checks for HAVE_UDAT_STANDALONE_SHORTER_WEEKDAYS and HAVE_SET_MAX_VARIABLE both compile and pass, so the sysroot is configured correctly for compile/link, just the builtin CMake functions are not working correctly.

best place to do this hardcoding is eng/native/tryrun.cmake and set cache value. In case cmake tryrun is new to you, cmake invokes it basically before anything else (even cmake's own modules), which is why the custom way of platform detection (like /HaikuConfig.h in your case) is needed.

if the cmake check is based on static condition or existance of symbol or header, then we use set_cache_value($VAR_NAME 0) or set_cache_value($VAR_NAME 1)
- as you notice in that file that this is normally not needed. usually it's misconfiguration if static analysis of header files or libraries is giving undesired results.
if the cmake check is based on try_compile, compiling and running a piece of code to identify something (which is not possible in cross-compilation), then we append _EXITCODE to $VAR_NAME.

jessicah commented 3 years ago

Hacked my way further, but running into an issue with src/libraries/shims that I can't piece together from the project/build files:

/home/jessica/source/rt2/src/libraries/shims/manual/System.forwards.cs(8,88): error CS0234: The type or namespace name 'ZLibException' does not exist in the namespace 'System.IO.Compression' (are you missing an assembly reference?) [/home/jessica/source/rt2/src/libraries/shims/manual/System.csproj]
/home/jessica/source/rt2/src/libraries/shims/manual/System.forwards.cs(9,77): error CS0234: The type or namespace name 'CookieVariant' does not exist in the namespace 'System.Net' (are you missing an assembly reference?) [/home/jessica/source/rt2/src/libraries/shims/manual/System.csproj]
/home/jessica/source/rt2/src/libraries/shims/manual/System.forwards.cs(10,77): error CS0234: The type or namespace name 'PathList' does not exist in the namespace 'System.Net' (are you missing an assembly reference?) [/home/jessica/source/rt2/src/libraries/shims/manual/System.csproj]

jessicah commented 3 years ago

@am11 With https://github.com/jessicah/dotnet-runtime/commit/10e4e8bf41e375c24b0086809f9046540d978e70 it "finishes", but I'm not sure what I have :p

It seems like the mono build is targeted for Linux, although the .so files not related to mono are linked against the Haiku libraries. I'm guessing mono isn't configured to build for Haiku, and probably some other parts are like this too :-/

I'll have a deeper look into tryrun.cmake, perhaps this is the cause of CMake's find_path and friends failing...

am11 commented 3 years ago

For mono, it would require a patch like https://github.com/dotnet/runtime/commit/a47b8dca77fe6b5f9514cfced139b00a8243e5a1 (it used to be autoconf based, now mono build in this repo has been moved to cmake).

tryrun.cmake

usually i run the configure-only step natively on platform (like OpenIndiana for illumos) to collect all the expected configure values then run the same with cross toolchain on linux, the diff is what goes in the tryrun.cmake (minus a things which are optional, but it doesn't hurt to have the entire diff in).

jessicah commented 3 years ago

A lot of that patch looks like fixes for building Illumos on Illumos, as most of these are HOST variables.

Anyway, I've just noticed this in the build output (the CMAKE variables it outpus), that I didn't see previously:

jessi@unbuntu:~/source/dotnet-runtime$ cat ../run
#!/bin/bash

ROOTFS_DIR=~/build/haiku-sysroot ./build.sh mono+libs -arch x64 -os haiku -c debug -cross --ninja

jessi@unbuntu:~/source/dotnet-runtime$ ../run
__DistroRid: haiku.r1-x64
  Determining projects to restore...
  Tool 'coverlet.console' (version '1.7.2') was restored. Available commands: coverlet
  Tool 'dotnet-reportgenerator-globaltool' (version '4.5.8') was restored. Available commands: reportgenerator
  Tool 'microsoft.dotnet.xharness.cli' (version '1.0.0-prerelease.21357.4') was restored. Available commands: xharness
  Tool 'microsoft.visualstudio.slngen.tool' (version '5.0.5') was restored. Available commands: slngen

  Restore was successful.
  All projects are up-to-date for restore.
  Determining projects to restore...
  All projects are up-to-date for restore.
  RuntimeConfigParser -> /home/jessi/source/dotnet-runtime/artifacts/bin/RuntimeConfigParser/Debug/net6.0/RuntimeConfigParser.dll
  JsonToItemsTaskFactory -> /home/jessi/source/dotnet-runtime/artifacts/bin/JsonToItemsTaskFactory/Debug/net472/JsonToItemsTaskFactory.dll
  RuntimeConfigParser -> /home/jessi/source/dotnet-runtime/artifacts/bin/RuntimeConfigParser/Debug/net472/RuntimeConfigParser.dll
  JsonToItemsTaskFactory -> /home/jessi/source/dotnet-runtime/artifacts/bin/JsonToItemsTaskFactory/Debug/net6.0/JsonToItemsTaskFactory.dll
  Microsoft.NET.Runtime.MonoTargets.Sdk -> /home/jessi/source/dotnet-runtime/artifacts/packages/Debug/Shipping/Microsoft.NET.Runtime.MonoTargets.Sdk.6.0.0-dev.nupkg
  Determining projects to restore...
  All projects are up-to-date for restore.
  The CMake command line is the same as the last run. Skipping running CMake configure.
  Running 'TARGET_BUILD_ARCH=x64 cmake --build . --target install --config Debug' in '/home/jessi/source/dotnet-runtime/artifacts/obj/mono/Haiku.x64.Debug/'
  [0/1] Re-running CMake...
  CMAKE_SYSTEM_NAME=Linux
  CMAKE_SYSTEM_VARIANT=
  CMAKE_SYSTEM_PROCESSOR=x86_64
  TARGET_ARCH=x86_64
  CMAKE_CROSSCOMPILING=FALSE
  -- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)
  -- Configuring done
  -- Generating done
  -- Build files have been written to: /home/jessi/source/dotnet-runtime/artifacts/obj/mono/Haiku.x64.Debug
  [0/1] Install the project...
  -- Install configuration: "Debug"
  Stripping debug symbols from /home/jessi/source/dotnet-runtime/artifacts/obj/mono/Haiku.x64.Debug/out/lib/libcoreclr.so
  System.Private.CoreLib.Generators -> /home/jessi/source/dotnet-runtime/artifacts/bin/System.Private.CoreLib.Generators/netstandard2.0-Debug/System.Private.CoreLib.Generators.dll
  System.Private.CoreLib -> /home/jessi/source/dotnet-runtime/artifacts/bin/mono/Haiku.x64.Debug/IL/System.Private.CoreLib.dll
  /home/jessi/source/dotnet-runtime/src/libraries/Native/build-native.sh x64 Debug outconfig net6.0-Haiku-Debug-x64 -os Haiku ninja -cross

That looks like I'm missing something crucial, surely this should be compiling for the target, not the host?

am11 commented 3 years ago

A lot of that patch looks like fixes for building Illumos on Illumos, as most of these are HOST variables.

Yes, haiku would need a "similar" patch, so it gets CMAKE_SYSTEM_NAME=Haiku. Also, when rebuilding after making changes, I delete artifacts directory so cmake runs the configure again.

-- Could NOT find PkgConfig (missing: PKG_CONFIG_EXECUTABLE)

e.g. equivalent of https://github.com/dotnet/runtime/blob/a47b8dca77fe6b5f9514cfced139b00a8243e5a1/src/mono/mono.proj#L236 (path to pkgconfig in sysroot dir) will fix this error.

jessicah commented 3 years ago

Ah right, I forgot that happens, had been running --clean earlier, whoops, thanks for the reminder :)

am11 commented 3 years ago

Admittedly, Host, Target and Build are three confusing concepts. In MSVC world (coreclr, libraries and corehost), Host in cross compilation is the machine we are executing compiler on and Target is the architecture of target machine. However, in autoconf world, Host is what MSVC refers to as Target and Build is what MSVC refers to as Host. Target in autoconf is a special case, where compiler itself is built on a different architecture than Host and Build and we are pointing to its destination triplet:

In other words, when mono (which was and still is primarily autoconfig-like) is talking about host of cross-compiler, it means target by normal logic. Therefore, I just follow the nearby logic. :)

illumos and FreeBSD are the closest matches to Haiku, as both of them cross compile on amd64 for amd64.

jessicah commented 3 years ago

Ah okay. Well, it looks like it may have built the cross mono stuff.

Log: https://gist.github.com/jessicah/647f1e8e033c96eb2a47ed3f526e47dd

I guess I can try copying some stuff over to a Haiku install, and seeing what happens.

What will be the next steps, assuming I can get something to run on Haiku?

jessicah commented 3 years ago

Okay, copied artifacts to my Haiku, and the mono-sgen binaries under artifacts/obj/mono/Haiku.x64.Debug run :)

am11 commented 3 years ago

That was fast. Great progress! It took me way longer to get this far. 🤣

On Linux box, where you have cross compiled, you can run:

# if pwd is your dotnet-runtime directory
cd ..
dotnet-runtime/.dotnet new console -n helloworldapp
cd hellowwoldapp
../dotnet-runtime/.dotnet publish -c Release -o mypublishdir

then edit: mypublishdir/helloworldapp.runtimeconfig.json and change version line to "version": "6.0.0-dev".

Copy mypublishdir to Haiku system and run mono-sgen mypublishdir/helloworldapp.dll.

It may emit a huge diagnostics and fail, but that's not the product we are interested in. We want dotnet(1) to work. :)

A few notes about the setup:

if we extract function(locate_toolchain_exec exec var) out of this if/elseif block: https://github.com/dotnet/runtime/blob/38477909eeb6296664cc184d461035b1774ea2df/eng/common/cross/toolchain.cmake#L94 to share and use the same pattern for Haiku, that will make it selfcontained in a way that compiler detection would look something like: -- Check for working C compiler: /runtime/.tools/x64/bin/x86_64-haiku-gcc -- works instead of: -- Check for working C compiler: /usr/bin/clang-10 -- works the benefit of using toolchain from the sysroot is that cmake will be using all tools (linker, loader, objcopy etc.) from same original toolchain, rather than mix and match, which usually is best to avoid.
that will likely fix the ICU hack as well as krb5 lookup issue.

Next step would be to port src/native/corehost, i.e. ./build.sh -s host+packs -os haiku -cross -c Release which produces package tarballs under /artifacts/packages. I have only tried it with clr+mono+libs+host+packs, but not just mono+libs+host+packs alone.

The method I followed to test mono on amd64 was to copy artifacts/packages/Release/Shipping/dotnet-runtime-6.0.0-dev-illumos-x64.tar.gz tarball file and artifacts/bin/mono/illumos.x64.Release/ directory on OpenIndiana machine, extract .tar in ~/.dotnet directory and then overwrite libcoreclr.so and System.Private.CoreLib.dll from mono to ~/.dotnet/shared/Microsoft.NETCore.App/6.0.0-dev/libcoreclr.so and .dotnet/shared/Microsoft.NETCore.App/6.0.0-dev/System.Private.CoreLib.dll respectively, and then ran the published application with ~/.dotnet/dotnet helloworld.dll.

jessicah commented 3 years ago

Hmm, running mono-sgen gives:

* Assertion at /home/jessi/source/dotnet-runtime/src/mono/mono/metadata/assembly.c:3087, condition 'corlib' not met

Seems like it doesn't know where to look for the dlls. Tried setting MONO_PATH to various values, but no dice.

I'll look into using the location_toolchain_exec next, might help avoid installing symlinks in /usr/bin, since clang doesn't really know to look in non-standard locations with the current compile switches.

jessicah commented 3 years ago

Well, making heaps more progress on coreclr/corehost, but now the c++ compiler has decided it doesn't know what size_t is. My CMake craft isn't that good, so if you have any ideas, would be great :)

Log: https://gist.github.com/jessicah/b7f48ab99909f4aaf2a8e11579a5c3b3

am11 commented 3 years ago

Error is coming from system header. Looks like string.h is missing or some macro definition in cmake which controls including string.h in system headers.

I tried build-rootfs from haiku branch, but it is failing with:

Chmod1 bin.linuxx86/jam 
...updated 33 target(s)...
Building cross tools with 2 parallel jobs
Invalid argument: `--sysroot'

does haiku/configure require some patch to support --sysroot option?

jessicah commented 3 years ago

The patch is here: https://gist.github.com/jessicah/046dca21d4c3e10e8e358cff0978c166

And additional packages to extract are here: https://haiku.nz/files/

Also need to edit /boot/system/develop/headers/posix/unistd.h, and change { c } to { c, 0 } for ioctl macro.

am11 commented 3 years ago

Thanks. I applied this patch http://sprunge.us/D9SwK4 to your branch and got error during sysroot compilation. I am using ubuntu 20.04 vagrant box:

# sudo apt install -y git build-essential nasm ninja-build clang liblldb-dev cmake liblttng-ust-dev libicu-dev libssl-dev libkrb5-dev libnuma-dev libcups2-dev libgmp-dev libmpfr-dev libmpc-dev

$ eng/common/cross/build-rootfs.sh haiku x64

...
gcc: error: /home/vagrant/dotnet-runtime/.tools/rootfs/x64/tmp/buildtools/binutils/ld/deffilep.c: No such file or directory
gcc: fatal error: no input files
compilation terminated.
make[4]: *** [Makefile:2179: deffilep.o] Error 1
make[4]: *** Waiting for unfinished jobs....
make[5]: Leaving directory '/home/vagrant/dotnet-runtime/.tools/rootfs/x64/generated/cross-tools-x86_64-build/binutils/ld'
make[4]: Leaving directory '/home/vagrant/dotnet-runtime/.tools/rootfs/x64/generated/cross-tools-x86_64-build/binutils/ld'
make[3]: *** [Makefile:1782: all-recursive] Error 1
make[3]: Leaving directory '/home/vagrant/dotnet-runtime/.tools/rootfs/x64/generated/cross-tools-x86_64-build/binutils/ld'
make[2]: *** [Makefile:1057: all] Error 2
make[2]: Leaving directory '/home/vagrant/dotnet-runtime/.tools/rootfs/x64/generated/cross-tools-x86_64-build/binutils/ld'
make[1]: *** [Makefile:7137: all-ld] Error 2
make[1]: Leaving directory '/home/vagrant/dotnet-runtime/.tools/rootfs/x64/generated/cross-tools-x86_64-build/binutils'
make: *** [Makefile:849: all] Error 2

jessicah commented 3 years ago

Can't say I've ever seen a build error like that. These are the build requirements from the Haiku website: sudo apt install git nasm autoconf automake texinfo flex bison gawk build-essential unzip wget zip less zlib1g-dev xorriso libtool gcc-multilib python3

Also, I've tracked down where the issue is, just don't quite understand what it's doing:

size_t redefined to DUMMY_size_t in palinternal.h
size_t is defined in pal_mstypes.h
size_t is undefined later in palinternal.h

https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/src/include/pal/palinternal.h#L207 https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/inc/pal_mstypes.h#L561-L575 https://github.com/dotnet/runtime/blob/main/src/coreclr/pal/src/include/pal/palinternal.h#L428

I've added a conditional to not undefine size_t at 3, but that ends up with this sort of output:

  In file included from /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/include/pal/malloc.hpp:27,
                   from /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/cruntime/malloc.cpp:24:
  /home/jessi/build/haiku-sysroot/generated/cross-tools-x86_64/x86_64-unknown-haiku/include/c++/8.3.0/new:120:31: error: declaration of 'operator new' as non-function
   void* operator new(std::size_t) _GLIBCXX_THROW (std::bad_alloc)
                                 ^
  In file included from /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/include/pal/corunix.hpp:23,
                   from /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/cruntime/malloc.cpp:22:
  /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/include/pal/palinternal.h:207:16: error: 'DUMMY_size_t' is not a member of 'std'
   #define size_t DUMMY_size_t
                  ^~~~~~~~~~~~
  /home/jessi/source/dotnet-runtime/src/coreclr/pal/src/include/pal/palinternal.h:207:16: note: suggested alternative: 'size_t'

So the undefine kind of seems correct, or the definition is leaking into standard headers incorrectly. I'm definitely lost in the cycle of includes at this point! Heh.

jessicah commented 3 years ago

After more searching into the weeds, it looks like the issue is that Haiku doesn't use _SIZE_T_DEFINED, but instead __size_t__. So will push a commit for that soon :)

jessicah commented 3 years ago

EDIT: patched mcontext_t to provide cs, will look at whether can be merged to Haiku in the near future.

I'm about as far as I can go now, without patching Haiku. Haiku doesn't provide the CS register as part of mcontext_t, as found in https://github.com/haiku/haiku/blob/master/headers/posix/arch/x86_64/signal.h

However, it is defined in kernel land at https://github.com/haiku/haiku/blob/master/headers/private/kernel/arch/x86/64/iframe.h#L31 so could probably add it to mcontext_t easily enough.

Also unsure if https://github.com/jessicah/dotnet-runtime/blob/haiku/src/coreclr/pal/src/include/pal/context.h#L509 is the right definition, the rest seem correct comparing to other platforms.

am11 commented 3 years ago

Porting CoreCLR to a new Unix-y platform typically entails porting the HP libunwind to the platform as a first step, then the coreclr PAL (src/coreclr/pal) and making sure PAL tests are passing. The rest of the coreclr components (VM, JIT, tools) usually don't require lots of code changes if the ISA is supported (in Haiku's case x86_64 which is supported). However, it will require certain amount of debugging to make coreclr_initialize succeed (but that will come later).

jessicah commented 3 years ago

Yes, I've been adding code to libunwind, https://github.com/jessicah/dotnet-runtime/commit/520a875d44680955fc30c834f3acf34e2bb1eada

And been slowly slogging through PAL... and working around stuff that's not in Haiku (yet). I think corehost is done, it gets to install and fails because it thinks coreclr is done :p

am11 commented 3 years ago

Great, it will also require src/os-haiku.c. We can build that repo in isolation to make sure its autoreconf -i && ./configure && make && make check is good. Some regression tests (make check) might fail as they are failing for other platforms too in v1.5 (including x86_64 linux, which coreclr has a copy of), but if it is working for most part (i.e. unw_step is working correctly), it's good enough for our need.

It would be best to issue a PR to libunwind/libunwind repo and once merged, cherry-pick the patch to coreclr/src/pal/libunwind and add a note: https://github.com/dotnet/runtime/blob/b937677e8f8601848d29bc072a93cc0c6e21576d/src/coreclr/pal/src/libunwind/libunwind-version.txt#L7. With the next update of libunwind, that list will be updated.

jessicah commented 3 years ago

Yep, I'll work on a haikuporter recipe for libunwind, and try to get patches upstreamed.

Ah, I see what you mean by needing an os-haiku.c, just tripped over the undefined references at [980/1357] targets :)

jessicah commented 3 years ago

I'll come back to libunwind, the low level details are a bit beyond my grasp right now; so just hacked together some code that just returns errors.

EDIT: the linker errors were because the --start/--end-group flags were missing.

Anyway, I'm running into an error with linking for ilasm, getting undefined references to libmd related stuff. I've checked the CMakeLists.txt for any conditional compilation, but I couldn't find anything, so am genuinely baffled by this one.

The linking error: https://gist.github.com/jessicah/0503f8b6ae8e16b9f5e6236c931a39d0#file-output-log-L3801-L3812

It looks like the libmd is getting built: https://gist.github.com/jessicah/0503f8b6ae8e16b9f5e6236c931a39d0#file-output-log-L2222-L2491

Perhaps it's the visibility of HIDDEN that's the problem? See readelf output below:

jessi@unbuntu:~/source/dotnet-runtime$ for file in `find artifacts/ -name libmd*`; do echo $file; readelf -a -W $file | grep -e MetaDataDllGetClassObject; done

artifacts/obj/coreclr/Haiku.x64.Debug/md/staticmd/libmdstaticapi_ppdb.a
000000000000006a  000000a400000004 R_X86_64_PLT32         0000000000000000 MetaDataDllGetClassObject - 4
   164: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND MetaDataDllGetClassObject

artifacts/obj/coreclr/Haiku.x64.Debug/md/staticmd/libmdstaticapi.a
000000000000006a  000000a400000004 R_X86_64_PLT32         0000000000000000 MetaDataDllGetClassObject - 4
   164: 0000000000000000     0 NOTYPE  GLOBAL DEFAULT  UND MetaDataDllGetClassObject

artifacts/obj/coreclr/Haiku.x64.Debug/md/compiler/libmdcompiler_crossgen.a
   176: 0000000000000000   222 FUNC    GLOBAL HIDDEN    21 MetaDataDllGetClassObject

artifacts/obj/coreclr/Haiku.x64.Debug/md/compiler/libmdcompiler_ppdb.a
   174: 0000000000000000   222 FUNC    GLOBAL HIDDEN    21 MetaDataDllGetClassObject

jessicah commented 3 years ago

Well, I have a few binaries I can run, working through fixes, slowly :)

But come across a weird one I'm struggling to find any documentation on:

runtime_laoder: /Development/testing/mcd: Could not resolve symbol '_ZTH12t_ThreadType'

And passing through c++filt gives: TLS init function for t_ThreadType; and this is defined at https://github.com/jessicah/dotnet-runtime/blob/haiku/src/coreclr/vm/util.cpp#L33

am11 commented 3 years ago

If PAL is building without quirks and PAL tests are passing on Haiku, then these errors are interesting to investigate. Otherwise PAL is a prerequisite and tests are there to ensure that this very base layer is working as expected for rest of the stack.

jessicah commented 3 years ago

How do I run the PAL tests on Haiku? As the build script wants to download a dotnet SDK.

am11 commented 3 years ago

https://github.com/dotnet/runtime/blob/57bfe474518ab5b7cfe6bf7424a79ce3af9d6657/docs/workflow/testing/coreclr/unix-test-instructions.md#pal-tests these are basic C/C++ tests.

am11 commented 3 years ago

PAL tests are meant to guarantee that underlying platform's C/C++ runtime is working as expected. PAL, in general, is an abstraction layer that wraps standard C/C++ functions and exposes them with Win32 API names. The rest of components under coreclr use Win32 API names (as they were originally written for .NET Framework, which is Windows only).

In my setup, I have OpenIndiana VM (vagrant + virtualbox) running on MBP. Repo is cloned on macOS, I crossbuilt clr+paltests using docker container, which in turned was built using build-rootfs.sh and published to Microsoft container repository (steup is done in this git repo):

docker run -e ROOTFS_DIR=/crossrootfs/x64 -v ~/runtime:/runtime \
    mcr.microsoft.com/dotnet-buildtools/prereqs:ubuntu-18.04-cross-illumos-20210714135645-f13d79e \
    /runtime/build.sh clr.paltests -cross -gcc -os illumos

Then with vagrant/virtualbox, I have ~/runtime directory shared and mounted at /runtime (same as docker). To run all PAL tests: /runtime/src/coreclr/pal/tests/palsuite/runpaltests.sh /runtime/artifacts/bin/coreclr/$(uname).x64.Debug/paltests (takes a minute or two to run the tests and test results are stored in an xml file path to which will be shown on stdout)

So PAL test executable is an ELF binary which does not require .NET SDK to compile and has no runtime dependency on coreclr_initialize and friends. :)

jessicah commented 3 years ago

@am11 ah, this is very useful, thanks :) when I had tried to run any of the shell scripts, it had still wanted to download a dotnet sdk. Do the paths actually need to match between the two VMs, or is that just a nicety?

One other thing, would it be possible to use the stack unwinding from libgcc_s instead of libunwind? I'm not sure having two unwinders in binaries is the right thing. And libgcc_s is automatically linked into all Haiku binaries.

jessicah commented 3 years ago

I'm still running into the TLS issue, with the undefined TLS init symbols. It seems like the files using the extern thread_local variables are being compiled without the actual definition of the variables included, but I haven't yet been able to figure it all out. Looks like most of the thread local variables are in vm, but I haven't yet been able to find anything in the CMakeFiles that would hint at a configuration problem :-/

And trying to build clr.paltests results in an MSBuild error related to libhostpolicy.so, looking for artifacts/bin/linux-x64.Debug/corehost/libhostpolicy.so:

Is that meant to be generated? Or is this yet another build system config I've missed? What a repo to dig through...

  Microsoft.NETCore.App.Runtime.Composite ->
/home/jessi/.nuget/packages/microsoft.dotnet.sharedframework.sdk/6.0.0-beta.21357.3/targets/sharedfx.targets(180,5):
  error MSB4018: The "GenerateSharedFrameworkDepsFile" task failed unexpectedly. [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018: System.IO.FileNotFoundException: /home/jessi/source/dotnet-runtime/artifacts/bin/linux-x64.Debug/corehost/libhostpolicy.so [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018:    at System.Diagnostics.FileVersionInfo.GetVersionInfo(String fileName) in System.Diagnostics.FileVersionInfo.dll:token 0x6000027+0x1e [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018:    at Microsoft.DotNet.SharedFramework.Sdk.FileUtilities.GetFileVersion(String sourcePath) in /_/src/Microsoft.DotNet.SharedFramework.Sdk/src/FileUtilities.cs:line 30 [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018:    at Microsoft.DotNet.SharedFramework.Sdk.GenerateSharedFrameworkDepsFile.Execute() in /_/src/Microsoft.DotNet.SharedFramework.Sdk/src/GenerateSharedFrameworkDepsFile.cs:line 62 [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018:    at Microsoft.Build.BackEnd.TaskExecutionHost.Microsoft.Build.BackEnd.ITaskExecutionHost.Execute() in Microsoft.Build.dll:token 0x60015da+0x3e [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]
  error MSB4018:    at Microsoft.Build.BackEnd.TaskBuilder.ExecuteInstantiatedTask(ITaskExecutionHost taskExecutionHost, TaskLoggingContext taskLoggingContext, TaskHost taskHost, ItemBucket bucket, TaskExecutionMode howToExecuteTask) in Microsoft.Build.dll:token 0x600145a+0x809 [/home/jessi/source/dotnet-runtime/src/installer/pkg/sfx/Microsoft.NETCore.App/Microsoft.NETCore.App.Runtime.Composite.sfxproj]

Build FAILED.

am11 commented 3 years ago

when I had tried to run any of the shell scripts, it had still wanted to download a dotnet sdk.

In cross compilation, it downloads host system's SDK for any subset. If you are running commands on Ubuntu, then it will download linux-x64 SDK. If you want to only build coreclr directly on the target system (Haiku), then you can directly call the internal script which has option to bypass managed code compilation (which exempts downloading SDK):

runtime/src/coreclr/build-runtime.sh -skipgenerateversion -nopgooptimize -skipmanaged

In the initial port, this method is useful to figure out the cmake introspection results, which we can compare with cross compilation ones and make adjustments accordingly in eng/native/tryrun.cmake.

Do the paths actually need to match between the two VMs, or is that just a nicety?

It is just how currently I have it set up on local system (everything mounted at the root). It could be any path and we can also copy artifacts/ to target rather than mounting.

would it be possible to use the stack unwinding from libgcc_s instead of libunwind?

Currently nognu/HP libunwind (default for non-macOS and non-FreeBSD Unices) and LLVM libunwind (default for macOS and FreeBSD, optional for $others) are supported. I am not sure if it is possible or how much work it is to support GCC one. If libgcc_s exposes similar interface which nognu libunwind and llvm ibunwind share, then it should be relatively straightforward given we have other two implementations.

And trying to build clr.paltests results in an MSBuild error related to libhostpolicy.so, looking for artifacts/bin/linux-x64.Debug/corehost/libhostpolicy.so:

Interesting. because it should not use or even build libhostpolicy or corehost when running:

dotnet-runtime/build.sh clr.paltests -cross -gcc -os haiku

What a repo to dig through...

Yes, .NET is not a simple repo to port. It took FreeBSD years to port (and still in progress). Porting .NET libraries takes a lot of time, energy and effort. It builds != it works and that's just a tip of the iceberg. :)

jessicah commented 3 years ago

Is there a reason why PAL doesn't use and instead hardcodes values for various error codes in a bunch of places?

Because the mapping from error symbol to number is a) not defined; and b) completely different on Haiku, causing test failures due to the false assumption. Just wary of adding a check for Haiku and using Haiku's values, in case there's a reason for it.

am11 commented 3 years ago

Is there a reason why PAL doesn't use and instead hardcodes values for various error codes in a bunch of places?

I think for errnos, if we always want the platform specific value then we should just use the system header. If that is not possible, we could add a cmake check to get the system number and use that in #ifndef PAL_STDCPP_COMPAT case (where we redefine stuff). @janvorli, what do you think?

jessicah commented 3 years ago

@am11 @janvorli this is my current fix, FWIW: https://github.com/jessicah/dotnet-runtime/commit/7bd4f1d40b4d938b81eecbc77d80e145e005118b

It resolves the PAL testsuite errors, at any case. I did try searching for uses of errno as much as possible, and from what I can tell elsewhere, the sources use the macro definitions, the PAL test suite was about the only place I could see using raw errno values.

The int casts resolved a couple of signed-vs-unsigned warnings as errors, BTW.

am11 commented 3 years ago

Just a heads up, we also use errno conversion in libraries https://github.com/dotnet/runtime/blob/2a5bae8d50eda49580fcafedbe0a6480b6ff064f/src/libraries/Native/Unix/Common/pal_error_common.h (which will come later :slightly_smiling_face:)

janvorli commented 3 years ago

@jessicah We actually do use errno.h everywhere in the PAL implementation. pal.h header is an interface for the runtime. The runtime code cannot include any platform headers to prevent collisions with runtime / windows types, functions and defines. We never hit the problem you have with Haiku, as Linux, macOS, FreeBSD and other Unix OSes we have support for (and even Windows) use the same values for the couple of errnos we use in the runtime. Thus we've never had a problem with those couple of values we use in the runtime and that we define there if not found. We basically use just the ERANGE, EBADF, ENOMEM, EILSEQ and EINVAL, as you have found.

The PAL errno test defines the ENOENT since all PAL tests execute on top of the coreclr PAL like the runtime and so they cannot include any platform headers.

Edit: I've forgotten to mention that the case when PAL_STDCPP_COMPAT is defined is a bit different - in that case, we try to do our best to ensure that pal.h can be included side by side with stl headers. But this is only a best effort thing that's not used in the runtime and was added to support special scenarios. And it keeps breaking things from time to time. The ultimate goal is to eventually get rid of the PAL completely. That will require to remove all Windowsisms from the runtime implementation and moving the Windows specific stuff to a Windows minipal and Unix stuff to Unix minipal. Runtime would then use standard C/C++ functions for most of the stuff.

janvorli commented 3 years ago

I think for errnos, if we always want the platform specific value then we should just use the system header. If that is not possible, we could add a cmake check to get the system number and use that in #ifndef PAL_STDCPP_COMPAT case (where we redefine stuff). @janvorli, what do you think?

We cannot use the system header in the runtime as I've explained above. I think that for these few constants, I'd accept what @jessicah has in the commit.

trungnt2910 commented 2 years ago

What's the status of this port, and how can I reproduce and continue your work?

jessicah commented 2 years ago

@trungnt2910 it's basically blocked on porting libunwind: https://github.com/jessicah/libunwind/tree/haiku-support; in particular, I'm fairly sure exception handling needs to be working for use in .Net Core, which is currently still failing with make check.

Regarding libunwind: Note that a bunch of offsets and things for register/stack frame layouts and such are guesses in some cases, so there's quite likely to be errors in my current tree.

In terms of building .Net Core itself, it's been a while, I'll need to run a fresh build to try remember what commands I used to run on both Linux and Haiku. Thought I would've saved the commands to run somewhere, but it seems I failed to do so :-/

am11 commented 2 years ago

Great progress! Your cmake fixes were merged and available as part of cmake releases since v3.21.2. We can use latest cmake in Docker manifest when preparing dotnet-buildtools-prereqs-docker image (for cross compilation toolchain; host: linux-x64, target: haiku-x64).

currently still failing with make check

A note about make check in libunwind: it runs the regression tests and often does not give us 100% clean results on other platforms/architectures either. Moreover, the usage of libunwind in .NET Runtime is confined to certain parts of the library. Some components are not required for runtime (but they might be needed for other consumers of libunwind such as, julialang/julia, strace etc.). So it can be ported in parts; initial port followed by improvements/fixes needed, as we learn what does or doesn't work.

It might be a good idea to start upstreaming changes and then incrementally improve things gradually. Generally, smaller patches are quickly reviewed and merged.

dotnet / runtime

Haiku: add platform support #55803