search

jank-lang / jank

The native Clojure dialect hosted on LLVM
https://jank-lang.org
Mozilla Public License 2.0
1.69k stars 50 forks source link

feat(default.nix): Init in /compiler+runtime #94

Open haruki7049 opened 1 month ago

haruki7049 commented 1 month ago

Adding Nix-build support for /compiler+runtime directory.

haruki7049 commented 1 month ago

Hey @jeaye, In https://github.com/jank-lang/jank/pull/94/commits/7585782c8086b322f3f80a0d3ee273a5244f4a9c, An error is occured about LLVM supported version. However github.com/llvm/llvm-project has no version, named as 19.0.x. What LLVM version should I use?

CMake Error at cmake/llvm.cmake:78 (message):
  Found unsupported version: LLVM 19.1.1;

  Please set llvm_dir pointing to the llvm version 19.0 to 19.0.x build or
  installation folder
Call Stack (most recent call first):
  CMakeLists.txt:113 (include)
haruki7049 commented 1 month ago
jeaye commented 1 month ago

Hey @jeaye, In 7585782, An error is occured about LLVM supported version. However github.com/llvm/llvm-project has no version, named as 19.0.x. What LLVM version should I use?

CMake Error at cmake/llvm.cmake:78 (message):
  Found unsupported version: LLVM 19.1.1;

  Please set llvm_dir pointing to the llvm version 19.0 to 19.0.x build or
  installation folder
Call Stack (most recent call first):
  CMakeLists.txt:113 (include)

Hey! I'd say, in order to get things compiling, we can just patch that CMake check to allow your 19.1.1 version. Everyone working on jank, right now, compiles Clang/LLVM from source and it's roughly 19.0.0 but it has a particular fix which hasn't made it into the 19.x release yet. I have a ticket for that here: https://github.com/llvm/llvm-project/issues/111068

By using 19.1.1, we may run into some hard crashes when we run jank, but the main task here is getting it compiling with Nix. Using a different Clang version after we get things compiling should be much easier, especially if I can get that commit backported to 19.x.

jeaye commented 1 month ago

I went through our vcpkg.json to see what we can do about each of the deps. Looks like only one of them needs compilation. The majority can be global distro packages. A few we'll need to bring in as submodules (like compiler+runtime/third-party/bpptree and others).

Hopefully this is helpful.

haruki7049 commented 1 month ago

Tasks:

haruki7049 commented 1 month ago

Memo: What is the DCHECK?

/build/source/folly/logging/LoggerDB.cpp: In constructor 'folly::LoggerDB::LoggerDB()':
/build/source/folly/logging/LoggerDB.cpp:78:3: error: 'DCHECK' was not declared in this scope
   78 |   DCHECK(ret.second);
      |   ^~~~~~
/build/source/folly/logging/LoggerDB.cpp: In member function 'folly::LogCategory* folly::LoggerDB::createCategoryLocked(LoggerNameMap&, folly::StringPiece, folly::LogCategory*)':
/build/source/folly/logging/LoggerDB.cpp:473:3: error: 'DCHECK' was not declared in this scope
  473 |   DCHECK(ret.second);
      |   ^~~~~~
make[2]: *** [CMakeFiles/folly_base.dir/build.make:3366: CMakeFiles/folly_base.dir/folly/logging/LoggerDB.cpp.o] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [CMakeFiles/Makefile2:141: CMakeFiles/folly_base.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
jeaye commented 1 month ago

Hmm, so jank builds what it needs from folly as part of a custom target: https://github.com/jank-lang/jank/blob/main/compiler%2Bruntime/CMakeLists.txt#L323

Folly itself is a pain in the ass to build, so I've stripped out just the bits that jank needs and patched them to compile more easily. Are you building folly on its own, or using jank's CMake target for it?

haruki7049 commented 1 month ago

I was building folly only, thinking that I could just build folly as is. it is quite hard to use git submodule in Nix.

jeaye commented 1 month ago

I was building folly only, thinking that I could just build folly as is. it is quite hard to use git submodule in Nix.

I think getting Nix to clone recursively (or just work with submodules in some way) would be beneficial for two reasons:

  1. Lots of people will need to build jank without Nix and submodules will work easily for them
  2. We don't want the Nix build to be building from different sources than what's in the actual repo (i.e. the submodules)

You know Nix better than I do, for sure. What I know, though, is that:

  1. If we're just building from the local filesystem (i.e. submodules have already been initialized), Nix doesn't need to know anything about submodules
  2. If we're building from scratch, fetchGit supports a deepClone = true flag

Are we able to work with that?

haruki7049 commented 1 month ago

Now my local repository was not cloned using git clone --recurse-submodules {repo_url}, but simply cloned and used using git clone {repo_url}. Even in this case, to make nix-build available, it was taken from the URL.

Sure, if we have those files locally, we can reference them.

Make git clone --recurse-submodules {repo_url} mandatory?

But I'd like to say no, because I believe that building in Nix should be possible with or without submodules.

jeaye commented 1 month ago

Now my local repository was not cloned using git clone --recurse-submodules {repo_url}, but simply cloned and used using git clone {repo_url}. Even in this case, to make nix-build available, it was taken from the URL.

Sure, if we have those files locally, we can reference them.

Make git clone --recurse-submodules {repo_url} mandatory?

But I'd like to say no, because I believe that building in Nix should be possible with or without submodules.

Initializing submodules (either with git clone --recurse-submodules or git submodule update --recursive --init) is currently mandatory, outside of Nix. Since we want Nix to use the same sources, I think Nix should also use the submodules. Otherwise, maintaining the Nix versions of our dependencies will require duplicate effort. Whenever we update a submodule, we'll also need to update the Nix expression.

Submodules are pure and reproducible, like any git commit. A single git commit in jank represents not only a snapshot of jank's source but also all of its submodules.

Hopefully that's a good enough point. Thanks for working with me here. :slightly_smiling_face: すみません

haruki7049 commented 1 month ago

That may indeed be true. Ok, I'll take the method of using local submodules on Nix.

And, no need to apologize. It's okay. NO PROBLEM!! だいじょうぶ!!

haruki7049 commented 1 month ago

Wait, The folly submodule should contain files other than those listed below. If you do not write the file exclusion settings in the submodule, you will end up writing the source file exclusion in Nix as well.

set(
  folly_sources
  third-party/folly/folly/ScopeGuard.cpp
  third-party/folly/folly/SharedMutex.cpp
  third-party/folly/folly/concurrency/CacheLocality.cpp
  third-party/folly/folly/synchronization/ParkingLot.cpp
  third-party/folly/folly/synchronization/SanitizeThread.cpp
  third-party/folly/folly/lang/SafeAssert.cpp
  third-party/folly/folly/lang/ToAscii.cpp
  third-party/folly/folly/system/ThreadId.cpp
  third-party/folly/folly/system/AtFork.cpp
  third-party/folly/folly/detail/Futex.cpp
)
jeaye commented 1 month ago

Wait, The folly submodule should contain files other than those listed below. If you do not write the file exclusion settings in the submodule, you will end up writing the source file exclusion in Nix as well.

set(
  folly_sources
  third-party/folly/folly/ScopeGuard.cpp
  third-party/folly/folly/SharedMutex.cpp
  third-party/folly/folly/concurrency/CacheLocality.cpp
  third-party/folly/folly/synchronization/ParkingLot.cpp
  third-party/folly/folly/synchronization/SanitizeThread.cpp
  third-party/folly/folly/lang/SafeAssert.cpp
  third-party/folly/folly/lang/ToAscii.cpp
  third-party/folly/folly/system/ThreadId.cpp
  third-party/folly/folly/system/AtFork.cpp
  third-party/folly/folly/detail/Futex.cpp
)

The folly submodule contains more files, yes. The ones listed there are just the files we're compiling, but we still need other files, such as all of the headers that those source files include. I don't think we should be excluding any files using Nix.

haruki7049 commented 1 month ago

Then, How should I do for https://github.com/jank-lang/jank/pull/94#issuecomment-2394512590 ?

jeaye commented 1 month ago

So, I think you shouldn't need to do anything for folly. We already have a submodule for it and it already builds as part of jank. There shouldn't be anything you need to do there, unless you run into errors compiling the folly which is part of jank. But I believe the error you ran into was when you were compiling a different version of folly. Correct me if I'm wrong.

haruki7049 commented 1 month ago

Ah, you are referring to a submodule that is local. I finally understand. Thanks, Jeaye!!

haruki7049 commented 1 month ago

log.txt

The compile log is verrrry long, so I uploaded the log.txt. Which dependency file contains the reference to SHA256 that is written in this file?

jeaye commented 1 month ago

log.txt

The compile log is verrrry long, so I uploaded the log.txt. Which dependency file contains the reference to SHA256 that is written in this file?

SHA256 comes in from openssl.

haruki7049 commented 1 month ago

PROGRESS:

Now I have an error: stderr.log

I know that the <random> is had by stdcxx, but there is already pkgs.libcxx in buildInputs... What should I do...?

haruki7049 commented 1 month ago

PROGRESS:

Now I have an error: stderr.log

I know that the <random> is had by stdcxx, but there is already pkgs.libcxx in buildInputs... What should I do...?

@jeaye I've been doing some research and learned that this project creates a pre-compiled header called PCH. I'm assuming the error is due to default.nix not being able to use libcxx or something when building this PCH.

Any thoughts on this matter?

jeaye commented 1 month ago

PROGRESS:

Now I have an error: stderr.log

I know that the <random> is had by stdcxx, but there is already pkgs.libcxx in buildInputs... What should I do...?

Hey! You've made it very close! That error comes from us trying to compile our runtime PCH. jank has two PCHs: one used for AOT building of the compiler and one used to load at runtime, for JIT compiled code.

So, there are two different major C++ standard library implementations: libstdc++ and libc++. On Linux, we use libstdc++. This generally gets shipped with GCC.

This error comes from the Clang we're using not finding libstdc++ on startup. The script here involved is bin/build-pch. I suspect that the way we're running that clang++ is something that Nix doesn't like. We can try adding -v to the clang++ flags, to get more info about where it's looking for headers.

EDIT: Originally misread the problem. Updated my response.

jeaye commented 1 month ago

Hey @haruki7049! You don't need to worry about the merge conflicts. I'll take care of them when this gets merged. :slightly_smiling_face:

haruki7049 commented 1 month ago

https://cmake.org/cmake/help/latest/command/target_precompile_headers.html#command:target_precompile_headers

@jeaye I wonder if I can use cmake commands like above... I don't want to use shell scripts in cmake...

haruki7049 commented 1 month ago

to create incremental-pch

jeaye commented 1 month ago

https://cmake.org/cmake/help/latest/command/target_precompile_headers.html#command:target_precompile_headers

@jeaye I wonder if I can use cmake commands like above... I don't want to use shell scripts in cmake...

I looked into it, before making the script. CMake's PCH tooling is very limited. jank is already using it for the AOT PCH. But the JIT PCH requires different flags. Ultimately, a script seemed the only viable approach.

I'd be open to getting that entirely put into CMake, but I didn't succeed when I tried.

Is having it in a script causing trouble with Nix?

haruki7049 commented 1 month ago 
[87/97] Generating incremental.pch
FAILED: incremental.pch /build/source/build/incremental.pch
cd /build/source/build && /build/source/bin/build-pch /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1 /build/source/include /build/source/build/incremental.pch -std=gnu++20\ -DIMMER_HAS_LIBGC=1\ -DIMMER_TAGGED_NODE=0\ -DHAVE_CXX14=1\ -DFMT_HEADER_ONLY=1\ -DJANK_DEBUG\ -w -isystem\ /nix/store/ai4g3129rzppvqmxfb7fg6xh3ihhxyyz-clang-19.1.1-dev/include\ -isystem\ /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\ -I/build/source/include/cpp\ -I/build/source/third-party/nanobench/include\ -I/build/source/third-party/folly\ -I/build/source/third-party/bpptree/include\ -isystem\ /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include\ -isystem\ /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\
/build/source/include/cpp/jank/prelude.hpp:3:10: fatal error: 'random' file not found
    3 | #include <random>
      |          ^~~~~~~~
PLEASE submit a bug report to https://github.com/llvm/llvm-project/issues/ and include the crash backtrace, preprocessed source, and associated run script.
Stack dump:
0.      Program arguments: /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang++ -std=gnu++20 -DIMMER_HAS_LIBGC=1 -DIMMER_TAGGED_NODE=0 -DHAVE_CXX14=1 -DFMT_HEADER_ONLY=1 -DJANK_DEBUG -w -isystem /nix/store/ai4g3129rzppvqmxfb7fg6xh3ihhxyyz-clang-19.1.1-dev/include -isystem /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include -I/build/source/include/cpp -I/build/source/third-party/nanobench/include -I/build/source/third-party/folly -I/build/source/third-party/bpptree/include -isystem /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include -isystem /nix/store/wiyw2njh783ad5pil81xi1jwdxsqbxxx-llvm-19.1.1-dev/include\\ -I /build/source/include -I /build/source/include/cpp -Xclang -fincremental-extensions -Xclang -emit-pch -x c++-header -o /build/source/build/incremental.pch -c /build/source/include/cpp/jank/prelude.hpp
1.      /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include/boost/mp11/detail/mp_value.hpp:13:1: current parser token 'namespace'
2.      /nix/store/8xyxg1mp3f0bb8nsll3hbm4lwpisfwz2-boost-1.86.0/include/boost/unordered/detail/type_traits.hpp:36:1: parsing namespace 'boost'
 #0 0x00007fffeb5e38fe llvm::sys::PrintStackTrace(llvm::raw_ostream&, int) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xfe38fe)
 #1 0x00007fffeb5e167c llvm::sys::CleanupOnSignal(unsigned long) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xfe167c)
 #2 0x00007fffeb4dac58 CrashRecoverySignalHandler(int) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xedac58)
 #3 0x00007fffea0495c0 __restore_rt (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x405c0)
 #4 0x00007ffff47d3d80 clang::Decl::castFromDeclContext(clang::DeclContext const*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xfd3d80)
 #5 0x00007ffff50ee530 clang::Sema::PopDeclContext() (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x18ee530)
 #6 0x00007ffff51b629d clang::Sema::ActOnFinishNamespaceDef(clang::Decl*, clang::SourceLocation) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x19b629d)
 #7 0x00007ffff4506535 clang::Parser::ParseNamespace(clang::DeclaratorContext, clang::SourceLocation&, clang::SourceLocation) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xd06535)
 #8 0x00007ffff44eb2df clang::Parser::ParseDeclaration(clang::DeclaratorContext, clang::SourceLocation&, clang::ParsedAttributes&, clang::ParsedAttributes&, clang::SourceLocation*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xceb2df)
 #9 0x00007ffff45cd3ef clang::Parser::ParseExternalDeclaration(clang::ParsedAttributes&, clang::ParsedAttributes&, clang::ParsingDeclSpec*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xdcd3ef)
#10 0x00007ffff45cecb5 clang::Parser::ParseTopLevelDecl(clang::OpaquePtr<clang::DeclGroupRef>&, clang::Sema::ModuleImportState&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xdcecb5)
#11 0x00007ffff44b5172 clang::ParseAST(clang::Sema&, bool, bool) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0xcb5172)
#12 0x00007ffff68ad499 clang::FrontendAction::Execute() (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x30ad499)
#13 0x00007ffff68295fb clang::CompilerInstance::ExecuteAction(clang::FrontendAction&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x30295fb)
#14 0x00007ffff6941c5b clang::ExecuteCompilerInvocation(clang::CompilerInstance*) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x3141c5b)
#15 0x0000000000413a51 cc1_main(llvm::ArrayRef<char const*>, char const*, void*) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x413a51)
#16 0x000000000040d60a ExecuteCC1Tool(llvm::SmallVectorImpl<char const*>&, llvm::ToolContext const&) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40d60a)
#17 0x00007ffff64485e9 void llvm::function_ref<void ()>::callback_fn<clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const::'lambda'()>(long) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c485e9)
#18 0x00007fffeb4db093 llvm::CrashRecoveryContext::RunSafely(llvm::function_ref<void ()>) (/nix/store/2f53shlvx25ng9kbmlpar45fg0zrjj35-llvm-19.1.1-lib/lib/libLLVM.so.19.1+0xedb093)
#19 0x00007ffff6448cf1 clang::driver::CC1Command::Execute(llvm::ArrayRef<std::optional<llvm::StringRef>>, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char>>*, bool*) const (.part.0) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c48cf1)
#20 0x00007ffff640dc14 clang::driver::Compilation::ExecuteCommand(clang::driver::Command const&, clang::driver::Command const*&, bool) const (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c0dc14)
#21 0x00007ffff640e73d clang::driver::Compilation::ExecuteJobs(clang::driver::JobList const&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&, bool) const (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c0e73d)
#22 0x00007ffff6422da4 clang::driver::Driver::ExecuteCompilation(clang::driver::Compilation&, llvm::SmallVectorImpl<std::pair<int, clang::driver::Command const*>>&) (/nix/store/a5ximnyqqb2rn5x8w90ic1acr0js84cg-clang-19.1.1-lib/lib/libclang-cpp.so.19.1+0x2c22da4)
#23 0x0000000000410e52 clang_main(int, char**, llvm::ToolContext const&) (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x410e52)
#24 0x000000000040cff6 main (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40cff6)
#25 0x00007fffea03314e __libc_start_call_main (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x2a14e)
#26 0x00007fffea033209 __libc_start_main@GLIBC_2.2.5 (/nix/store/3dyw8dzj9ab4m8hv5dpyx7zii8d0w6fi-glibc-2.39-52/lib/libc.so.6+0x2a209)
#27 0x000000000040d055 _start (/nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin/clang+++0x40d055)
clang++: error: clang frontend command failed with exit code 139 (use -v to see invocation)
clang version 19.1.1
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /nix/store/6a2f67zl2j0nxi551hp7q4cpid67k3aw-clang-19.1.1/bin
clang++: error: unable to execute command: Segmentation fault (core dumped)
clang++: note: diagnostic msg: Error generating preprocessed source(s).
ninja: build stopped: subcommand failed.

This is the compile log by 2d45d67b4cf67567e58909e4d401f991a586cb38 . I want to take libstdcxx to the shell script, so I tried to change the stdenv's compiler, such as pkgs.llvmPackages_19.libstdcxxClang, but there is no effects.

I don't know the way to serve libstdcxx to the shell script...

haruki7049 commented 1 month ago

stderr.log

{
  pkgs ? import (builtins.fetchTarball {
    url = "https://github.com/NixOS/nixpkgs/archive/75b209227dff3cbfac19f510a62f9446c92beac4.tar.gz";
    sha256 = "166varzdamn2kkw5zslarzl6026q87wjnss1hlhdsq89hwvfgd2d";
  }) { },
}:

let
  lib = pkgs.lib;

  # Deps
  boost = pkgs.callPackage ./nix/boost.nix { };
  boehmgc = pkgs.callPackage ./nix/boehmgc.nix { };
  libzippp = pkgs.callPackage ./nix/libzippp.nix { };
  openssl = pkgs.callPackage ./nix/openssl.nix { };
in

pkgs.llvmPackages_19.stdenv.mkDerivation {
  pname = "jank-lang";
  version = "dev";
  src = lib.cleanSource ./.;

  nativeBuildInputs = with pkgs; [
    cmake
    git
    ninja
  ];

  patchPhase = ''
    patchShebangs --host "${pkgs.bash}" bin/build-pch
  '';

  buildInputs = [
    boost
    boehmgc
    libzippp
    openssl
    pkgs.doctest
    pkgs.double-conversion
    pkgs.readline
    pkgs.libzip
    pkgs.immer
    pkgs.cli11
    pkgs.magic-enum
    pkgs.fmt
    pkgs.llvmPackages_19.clang-unwrapped
    pkgs.llvmPackages_19.llvm
  ];

  cmakeBuildType = "Debug";
  CC = "${pkgs.llvmPackages_19.libstdcxxClang}/bin/clang";
  CXX = "${pkgs.llvmPackages_19.libstdcxxClang}/bin/clang++";

  cmakeFlags = [
    "-Djank_tests=on"
  ];
}

This default.nix can change default compiler, by changing CC and CXX. This can affect jank's compile but not incremental-pch.

jeaye commented 1 month ago

Hm, I think we can just try to compile the PCH outside of CMake first, using clang within your nix environment. Once we get that working, then we can hook it into CMake.

Right now, the script is expecting to receive the CLANG_INSTALL_PREFIX, which is the place where Clang lives. Are you setting that to be where Nix has clang?

What happens if you just invoke bin/build-pch from your Nix shell? It needs two args:

  1. The clang prefix
  2. The path to jank's compiler+runtime/include dir
jeaye commented 3 weeks ago

Hi @haruki7049! Just checking in to see the status of this and if you need any help.

haruki7049 commented 3 weeks ago

Hi @jeaye. Thanks for this mention. Perhaps are you in a hurry? If you are in a hurry, I am sorry for that.

I am interested in CI's result on main branch. It is not a good thing that most of the CI results on the main branch have failed. Are you know the reasons?

haruki7049 commented 3 weeks ago

Hey jeaye, can we build jank compiler without incremental-pch? I think that it is easy to build jank compiler without incremental-pch.

jeaye commented 3 weeks ago

Hi @jeaye. Thanks for this mention. Perhaps are you in a hurry? If you are in a hurry, I am sorry for that.

Nope, not in a hurry. No need to be sorry. Just trying to keep things progressing, since you've had really good progress!

I am interested in CI's result on main branch. It is not a good thing that most of the CI results on the main branch have failed. Are you know the reasons?

The reasons boil down to jank previously requiring devs to build clang/llvm from source, to get the latest fixes. This takes 2.5 hours to do in CI and makes debugging any other issues a real pain. The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet. Once it is, I can update CI to use that and then resolve anything else.

I agree that having main fail regularly is not ideal. It'll be fixed soon!

Hey jeaye, can we build jank compiler without incremental-pch? I think that it is easy to build jank compiler without incremental-pch.

jank will need the incremental PCH to work. We could build it as a CMake target, though. Perhaps Nix would like that more. We'd just need to use CMAKE_CXX_COMPILER to run the command as a custom CMake target.

We just need to make sure that the changes we make will work outside of Nix, too. Removing vcpkg is a big win for everyone, but we will still need to be able to access the necessary dependencies, either from the OS's package manager or from within the repo as a submodule.

haruki7049 commented 3 weeks ago

The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet.

Wait, Is this project using LLVM 19.1.2? Not LLVM 19.1.1?

jeaye commented 3 weeks ago

The latest llvm 19.1.2 release has everything we need, but it's not available as a binary for most distros yet.

Wait, Is this project using LLVM 19.1.2? Not LLVM 19.1.1?

When you started this PR, jank was using an unreleased version of LLVM. That's why we had to compile it from source. I told you not to worry about it, since the issues would show up as hard crashes at runtime but we should focus on getting things compiling first. Since then, 19.1.2 has been released with the necessary fixes, so we can go forward using that.

haruki7049 commented 3 weeks ago

@jeaye Can I close the pull request once I have found it very difficult to write up the default.nix for this project?　Instead, I am going to add some more shell.nix.

jeaye commented 3 weeks ago

How about I remove vcpkg (and add submodules) and the build-pch script? That should make things easier for you.

On October 29, 2024 12:20:31 AM PDT, haruki7049 @.***> wrote:

@jeaye Can I close the pull request once I have found it very difficult to write up the default.nix for this project?　Instead, I am going to add some more shell.nix.

-- Reply to this email directly or view it on GitHub: https://github.com/jank-lang/jank/pull/94#issuecomment-2443417468 You are receiving this because you were mentioned.

Message ID: @.***>

haruki7049 commented 3 weeks ago

If those could be done, it would be quite easy.... But are they feasible?

jeaye commented 3 weeks ago

Yeah. Give me a few days and I'll report back.

On October 29, 2024 4:29:27 PM PDT, haruki7049 @.***> wrote:

If those could be done, it would be quite easy.... But are they feasible?

-- Reply to this email directly or view it on GitHub: https://github.com/jank-lang/jank/pull/94#issuecomment-2445508541 You are receiving this because you were mentioned.

Message ID: @.***>

jeaye commented 3 weeks ago

Yeah. Give me a few days and I'll report back. … On October 29, 2024 4:29:27 PM PDT, haruki7049 @.> wrote: If those could be done, it would be quite easy.... But are they feasible? -- Reply to this email directly or view it on GitHub: #94 (comment) You are receiving this because you were mentioned. Message ID: @.>

I've moved the incremental PCH compilation out of the script and into cmake as a custom target in https://github.com/jank-lang/jank/commit/319f0b25893457c593b933923fa17b104ad6fd28

Hopefully that helps. I'll work on removing vcpkg next. That'll take longer.

haruki7049 commented 3 weeks ago

Commit hash: 3f60ab54f24bf0daf53746a414afdc00839241ad Command: nix-build Directory: /compiler+runtime

This error is generated. What is this? err.log

nix-shell -p nix-info --run "nix-info -m"

 - system: `"aarch64-darwin"`
 - host os: `Darwin 24.0.0, macOS 15.0.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/store/63lvc59rjz075kb1f11xww2x2xqpw1pn-source`
jeaye commented 3 weeks ago

Commit hash: 3f60ab5 Command: nix-build Directory: /compiler+runtime

This error is generated. What is this? err.log

nix-shell -p nix-info --run "nix-info -m"

 - system: `"aarch64-darwin"`
 - host os: `Darwin 24.0.0, macOS 15.0.1`
 - multi-user?: `yes`
 - sandbox: `no`
 - version: `nix-env (Nix) 2.18.2`
 - channels(root): `"nixpkgs"`
 - nixpkgs: `/nix/store/63lvc59rjz075kb1f11xww2x2xqpw1pn-source`

The error happens when building folly. From what I gather, the issue is because you're using libc++ instead of libstdc++. We can see that in the errors here:

In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/functional:552:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__functional/boyer_moore_searcher.h:27:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/vector:325:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/formatter_bool.h:16:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/concepts.h:16:
In file included from /nix/store/7x3an9qwvkmird3n67qw09jccmd1dd7g-libcxx-19.1.1-dev/include/c++/v1/__format/format_parse_context.h:16:

jank needs to be built with libstdc++. Generally, clang will use this by default on Linux, but Nix must be doing something weird.

jeaye commented 2 weeks ago

vcpkg is gone: https://github.com/jank-lang/jank/pull/122

We rely on boost and readline globally, while the rest has been brought in via submodules.

haruki7049 commented 2 weeks ago

Commit hash: c221ce92cc29391d3d186f96314a2302ff927834

Diff:

--- a/compiler+runtime/default.nix
+++ b/compiler+runtime/default.nix
@@ -30,6 +30,7 @@ pkgs.llvmPackages_19.stdenv.mkDerivation {
     pkgs.doctest
     pkgs.double-conversion
     pkgs.readline
+    pkgs.fmt
     pkgs.libzip
     pkgs.llvmPackages_19.clang-unwrapped
     pkgs.llvmPackages_19.llvm

err.log

 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.58, NixOS, 24.05 (Uakari), 24.05.5919.32e940c7c420`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.18.8`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`
haruki7049 commented 1 week ago

@jeaye I don't know, because I don't know C++, but is the fmt/format really imported?

Commit hash: c221ce92cc29391d3d186f96314a2302ff927834

System:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.31, NixOS, 24.11 (Vicuna), 24.11pre703931.4aa36568d413`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(root): `"nixos, nixos-hardware"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

err.log

jeaye commented 1 week ago

@jeaye I don't know, because I don't know C++, but is the fmt/format really imported?

Commit hash: c221ce9

System:

$ nix-shell -p nix-info --run "nix-info -m"
 - system: `"aarch64-linux"`
 - host os: `Linux 6.6.31, NixOS, 24.11 (Vicuna), 24.11pre703931.4aa36568d413`
 - multi-user?: `yes`
 - sandbox: `yes`
 - version: `nix-env (Nix) 2.24.10`
 - channels(root): `"nixos, nixos-hardware"`
 - nixpkgs: `/nix/var/nix/profiles/per-user/root/channels/nixos`

err.log

Looks like you may not be on the latest main. Your compilation line there is missing the include path to fmt. It was added here: https://github.com/jank-lang/jank/commit/81662734f4526f1f90b82c75d940851b87c5074b#diff-084c38e4be3e607f9cb1264d6a09cf1234173392296c938b42838cfffb28176dR236

haruki7049 commented 1 week ago

Hey @jeaye, Are you know this error's meaning?

https://gist.github.com/haruki7049/b04d1772c570f6019bd02808a0fc3647#file-err-log

Commit hash: 3e3a3bd753bd4142ca099b27abf91a1da315c093

jeaye commented 1 week ago

Hey @jeaye, Are you know this error's meaning?

https://gist.github.com/haruki7049/b04d1772c570f6019bd02808a0fc3647#file-err-log

Commit hash: 3e3a3bd

The error is saying that the incremental PCH we're loading was compiled with different flags than the jank binary. So when the jank binary tries to load the incremental PCH at runtime, clang raises this error.

I saw this before and I think I fixed it by ensuring all the same compilation flags are used for both jank and the incremental.pch. Though, I think I've only seen it when using a prebuilt clang/llvm for building jank. I've never been able to fully get that working, for one reason or another.

jeaye commented 1 week ago

If that incremental.pch is the only thing getting in the way, perhaps we can try without it. There would be a severe performance penalty for startup time, though. That's why I'm hopeful that we can get it loaded normally.