Closed oxinabox closed 4 years ago
First issue!
On non-Linux you can build without the web interface (which is the reason for the hinotify dep) using stack build --flag dex:-web
.
I'll add another target to the makefile with a comment.
Closing, but please re-open if that didn't work for you. It would be nice to figure out the macOS equivalent to Linux's inotify. PRs welcome.
Well we got further this time 🙂
This time it was llvm-hs-7.0.1
that blocked me
JuicyPixels > Registering library for JuicyPixels-3.3.3.1..
-- While building package llvm-hs-7.0.1 using:
/private/var/folders/1r/4cjj1xk90qbb6njj7dn_cmt40000gn/T/stack2402/llvm-hs-7.0.1/.stack-work/dist/x86_64-osx/Cabal-2.4.0.1/setup/setup --builddir=.stack-work/dist/x86_64-osx/Cabal-2.4.0.1 configure --user --package-db=clear --package-db=global --package-db=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/pkgdb --libdir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/lib --bindir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/bin --datadir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/share --libexecdir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/libexec --sysconfdir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/etc --docdir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/doc/llvm-hs-7.0.1 --htmldir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/doc/llvm-hs-7.0.1 --haddockdir=/Users/oxinabox/.stack/snapshots/x86_64-osx/19f92992da84ddd872e7f776ca44f7ddcdce0f7b17815902723d4090d435093e/8.6.5/doc/llvm-hs-7.0.1 --dependency=Cabal=Cabal-2.4.1.0-4t2ut7bCQNuEj8DDES6BZk --dependency=array=array-0.5.3.0 --dependency=attoparsec=attoparsec-0.13.2.2-6XZWdvg7dAKGuzy18iXIM3 --dependency=base=base-4.12.0.0 --dependency=bytestring=bytestring-0.10.8.2 --dependency=containers=containers-0.6.0.1 --dependency=exceptions=exceptions-0.10.2-2mS9GUVmJpmKSw5CuaN6ik --dependency=llvm-hs-pure=llvm-hs-pure-7.0.0-3a3CINtVYv5LBG1xtLCQFN --dependency=mtl=mtl-2.2.2 --dependency=template-haskell=template-haskell-2.14.0.0 --dependency=transformers=transformers-0.5.6.2 --dependency=utf8-string=utf8-string-1.0.1.1-Geq8jdOv4Q3LkcQoEOWDVv -f-debug -fshared-llvm --exact-configuration --ghc-option=-fhide-source-paths
Process exited with code: ExitFailure 1
Progress 38/45
(I don't think I can reopen, but you can)
I assume you have LLVM 7 installed on your system? (Stack won't do that for you.) I know @apaszke managed to build on Mac so it must be possible! I'll get my hands on a Mac and try to build it and report back if I learn anything.
I've also had trouble building llvm-hs
, but on Ubuntu:
llvm-hs > setup: The program 'llvm-config' version ==7.0.* is required but the version
llvm-hs > found at /usr/bin/llvm-config is version 7.1.0
What's the best way to install llvm-7.0.*
or to get around the version check?
I assume you have LLVM 7 installed on your system?
Yep, see above:
~/ProgO/dex-lang(git:master)❱ ✘ ❱ llvm-config --version
7.1.0
Installed via
brew install llvm@7
then adding
usr/local/opt/llvm@7/bin/
to my PATH
Hmm, it might be that I need to do something to add some libraries, e.g. with LD_PATH
.
@sjakobi , maybe try fiddling with the extra-deps in stack.yaml? I think I must have pinned those versions to match whatever LLVM version is on my system.
I'd welcome any suggestions for how to make building more robust. It's not something I'm very familiar with.
I've now installed llvm-8-dev via https://apt.llvm.org/, which appears to be compatible with llvm-hs-8.0.0. stack build
and stack test
now succeed with the following patch:
diff --git a/dex.cabal b/dex.cabal
index b397bc1..1be8656 100644
--- a/dex.cabal
+++ b/dex.cabal
@@ -21,7 +21,7 @@ library
Actor, DeShadow, Cat, Flops, Normalize,
Interpreter, RenderHtml, Plot
build-depends: base, containers, mtl, binary, bytestring,
- time, tf-random, llvm-hs-pure ==7.*, llvm-hs ==7.*,
+ time, tf-random, llvm-hs-pure >= 7 && < 8.1, llvm-hs >= 7 && < 8.1,
aeson, megaparsec, warp, wai,
parser-combinators, http-types, prettyprinter, text,
data-binary-ieee754, blaze-html, cmark, diagrams-lib,
diff --git a/stack.yaml b/stack.yaml
index ebedbc7..878920f 100644
--- a/stack.yaml
+++ b/stack.yaml
@@ -10,5 +10,5 @@ packages:
- .
extra-deps:
-- llvm-hs-7.0.1
-- llvm-hs-pure-7.0.0
+- llvm-hs-8.0.0
+- llvm-hs-pure-8.0.0
However:
≻ stack exec -- dex repl
Pass 'Interprocedural Sparse Conditional Constant Propagation' is not initialized.
Verify if there is a pass dependency cycle.
Required Passes:
fish: “stack exec -- dex repl” terminated by signal SIGSEGV (Address boundary error)
Eesh, I've never seen anything like that before. I've had a universally good experience with LLVM and llvm-hs so far. It's probably something on my end (I haven't tested with LLVM 8). I wonder what happens if you disable the LLVM passes by commenting out this line?
I wonder what happens if you disable the LLVM passes by commenting out this line?
Same result. :/
Sorry I can't be of more help. I don't know much about LLVM and it's my first time building llvm-hs
.
Now I'm suddenly having troubles too. When I make the project I get the following warning Warning: the following files would be used as linker inputs, but linking is not being done: cbits/libdex.so
And then when I try to run dex I get:
.../dex-lang/.stack-work/install/x86_64-linux/bf780b400f72a5a59241a8970e88cd31804e452b83a4a636831cdecd2d1c8c55/8.6.5/bin/dex: error while loading shared libraries: cbits/libdex.so: cannot open shared object file: No such file or directory
but if I ls the cbits directory I do find libdex.so
I thought I'd try updating everything to LLVM 8 but I can't get llvm-hs-8.0.0 to build ...
Stack does a really nice job of managing Haskell dependencies. So it's predictable that the build issues are around our single non-Haskell dependency.
Does anyone have ideas for how we could do this better? Building LLVM from source? Using a container?
Building from source looks impractical for most people, maybe containers would be a good idea
@joaogui1, I see that warning sometimes too and I don't think it's a problem. Try running dex
from the project root directory (yes, it'd be nice if you didn't have to).
@dougalm that worked, thanks :)
It would be nice to be on LLVM 8, I started trying to get this to build with the JuliaLang cross-compiler infastructure: BinaryBuilder. but @staticfloat informs me that that is setup for LLVM 8, and I would need to add a binary provider for LLVM 7, to use that. Which seems unfun.
If I can get BinaryBuilder going with this it would be nice, because then can just provide pre-built binaries for every platform; rather than needing to use make
.
(Not sure how much pain stack and haskell would give. Hopefully, can just use the stack installer script.)
@joaogui1, I see that warning sometimes too and I don't think it's a problem. Try running
dex
from the project root directory (yes, it'd be nice if you didn't have to).
Wouldn't it be possible to add cbits/libdex.so to /usr/local/lib and then when installing dex make it reference /usr/local/lib/cbits/libdex.so instead of just cbits/libdex.so? Or create an enviroment variable with the path to libdex.so (when the person runs the makefile) and then tell dex to use that enviroment variable? The first option would require sudo and the second would be a little bit less stable I guess, what do you think?
@oxinabox, that sounds promising! Thanks for offering to look into it.
It would be nice to be on LLVM 8
Yes, definitely. Let's move to LLVM 8 if we can solve the build issues.
then can just provide pre-built binaries for every platform; rather than needing to use make.
That sounds like the right long-term goal for sure. For now, though, the people wanting to build Dex will (hopefully) be people interested in helping develop it. So they'll need to be able to make changes to the Haskell source and recompile.
Dex might be different from Julia in this regard. AIUI, a lot of Julia is implemented in Julia. Dex is implemented in Haskell, even pieces like AD, and I don't see that changing. It's hard to imagine improving on Haskell as a language for writing compilers. There may eventually be a need for a standard library of numerical routines written in Dex. But even that is quite far off.
@joaogui1, yes, both of those would work, but I prefer the second option because I don't think we need Dex to be available system-wide for now. I'd be happy to accept a PR for the readme that adds instructions for setting up the environment variable in your bashrc.
Same here on Mac. I have tried to:
brew install llvm-hs/llvm/llvm-7
(brew install llvm@7
will install llvm-7.1) and sudo port install llvm-7.0
All produced the same error.
Dex might be different from Julia in this regard. AIUI, a lot of Julia is implemented in Julia.
That shouldn't matter for purposes of BinaryBuilder, it is in theory a general purpose cross-compilation enviroment tool that happens to be maintained by the julia project. It is mostly used for building dependencies of julia and of julia packages. Like building GTK, and Tcl etc. I don't think anything based on Haskell yet has been built, so that might be a challange. But I'll give it a shot, should put the system through its paces.
For now, though, the people wanting to build Dex will (hopefully) be people interested in helping develop it. So they'll need to be able to make changes to the Haskell source and recompile.
Thats true. In this case it might help me, because it has containerization built-in so I don't need to worry about so much Mac OS weirdness (in theory).
But indeed, its isn't a real overall solution even if I can make it work.
Looks like my quest to get this to build with BinaryBuilder is blocked for now, because of https://github.com/commercialhaskell/stack/issues/2387
On a more positiive note: I now have DexLang working for myself. Because I gave up on Mac and just spun up a Ubuntu 16 VM.
+1 for updating LLVM. I've actually never managed to build Dex on Mac exactly due to the LLVM version mismatch (7.1 installed, 7.0 expected). IIRC the LLVM website doesn't even have a packaged 7.0 version for Mac!
@apaszke Ah, my bad. I forget that you (like Lyndon) failed even after disabling the inotify stuff. Ok, sounds like the right move is to switch to LLVM 8.
@oxinabox, glad you managed to get a VM version working at least! Also, I noticed you're at Invenia Cambridge. Say hi to Christian Steinruecken from me!
Ok, I've updated to LLVM 8 on this branch, which includes a fix for @sjakobi's "not initialized" bug. I tried installing on a Mac, with LLVM 8 installed via brew install llvm@8
, and now I'm stuck on this bug, which appears to be upstream of llvm-hs in LLVM itself. Or maybe homebrew LLVM doesn't work with llvm-hs?
I'll keep trying, but maybe someone here with more Mac familiarity can figure it out now that we're on LLVM 8?
@oxinabox - did you have any luck building on Mac after we updated to LLVM 8?
I did not. Same error you were getting
I am currently building LLVM from source to see how that goes. Will post results on that.
The good news: I have dex running on Mac
The bad news. I can't repeat it. The first time I did it I kept a script so I would be able to repeat it. But apparently I missed something (or Dex master moved under me between my two runs).
My first shot seems to work fully But my second errors on start up of REPL
bash-3.2$ stack -- exec dex repl
dyld: lazy symbol binding failed: Symbol not found: _LLVMInitializeAMDGPUTargetInfo
Referenced from: /Users/oxinabox/progmisc2/dex-lang/.stack-work/install/x86_64-osx/1051de0f5f44af8b4795ed2f3a6b75cc8f74343cb71df5fea9088f8e9b40d75d/8.6.5/bin/dex
Expected in: /usr/local/lib/libLLVM.dylib
dyld: Symbol not found: _LLVMInitializeAMDGPUTargetInfo
Referenced from: /Users/oxinabox/progmisc2/dex-lang/.stack-work/install/x86_64-osx/1051de0f5f44af8b4795ed2f3a6b75cc8f74343cb71df5fea9088f8e9b40d75d/8.6.5/bin/dex
Expected in: /usr/local/lib/libLLVM.dylib
Abort trap: 6
I am just checking over my script on how I built it to make sure that that is right -- I might have missed a step the second time
Hooray, that's great news! Thank you for working on this. Sounds like we're very close to a reproducible Mac build.
# Set envvars as desired
export PROJECT=/Users/oxinabox/dex_building/round6
export LLVM_DIR="$PROJECT"/llvm/
export LLVM_INSTALL="$LLVM_DIR"/installed
export DEX_SRC="$PROJECT"/dex-lang
#### Build LLVM ####
# as per https://github.com/llvm-hs/llvm-hs/tree/llvm-9#building-from-source
# Download LLVM
mkdir -p $LLVM_DIR
cd $LLVM_DIR
wget http://releases.llvm.org/9.0.0/llvm-9.0.0.src.tar.xz
tar -xzf llvm-9.0.0.src.tar.xz
export LLVM_SRC=$LLVM_DIR/llvm-9.0.0.src/
# Build LLVM (this will take a while)
mkdir builddir
cd builddir
cmake $LLVM_SRC -DCMAKE_INSTALL_PREFIX=$LLVM_INSTALL \
-DLLVM_BUILD_LLVM_DYLIB=True -DLLVM_LINK_LLVM_DYLIB=True
cmake --build . -j 4
cmake --build . --target install
# Special Mac things
cd $LLVM_INSTALL/lib
ln -s -F libLLVM.dylib libLLVM-9.dylib
install_name_tool -id $PWD/libLTO.dylib libLTO.dylib
install_name_tool -id $PWD/libLLVM.dylib libLLVM.dylib
install_name_tool -change '@rpath/libLLVM.dylib' $PWD/libLLVM.dylib libLTO.dylib
## Now set this as your LLVM
export PATH="$LLVM_INSTALL"/bin:$PATH
llvm-config --libdir # Check this matches to `$"LLVM_INSTALL"/lib
#### Build Dex
mkdir -p $DEX_SRC
git clone https://github.com/google-research/dex-lang.git $DEX_SRC
cd $DEX_SRC
make -j4
Building LLVM takes literally hours. So not great.
(update: LD_LIBRARY_PATH does not need to be set, I made some other mistake, see below)
# None of the above need to be set except to set `LD_LIBRARY_PATH`
export LD_LIBRARY_PATH=$"$LLVM_INSTALL"/lib:$LD_LIBRARY_PATH
stack -- exec dex repl
I would really like to get rid of the need to set the LD_LIBRARY_PATH
.
If you don't do that then it gets the dyld: lazy symbol binding failed
when you start it.
Something needs to be changes somewhere so dex uses statically linked libraries I think?
I tried running llvm-config --link-static
before building dex
,
but that did not seem to work.
I am no expert on LLVM build/linking stuff.
I know its possible, because I still have the dex build I made yesterday that works fine without setting the LD_LIBRARY_PATH
. I just can't recreate it.
Ok, so what is going on is that the path for the custom build of LLVM is not getting put in to the dex binary.
Which right now is built at
/Users/oxinabox/progmisc2/dex-lang/.stack-work/install/x86_64-osx/1051de0f5f44af8b4795ed2f3a6b75cc8f74343cb71df5fea9088f8e9b40d75d/8.6.5/bin
looking at the good one I built yesterday, under ❱ otool -l -v dex | less
it has:
name /Users/oxinabox/progmisc/llvm-8.0.0.installed/lib/libLLVM.dylib (offset 24)
Where as todays has:
name @rpath/libLLVM.dylib (offset 24)
And no LC_RPATH
defined. So I guess that is why it goes looking in system shared library locations.
If I go there and run
install_name_tool -add_rpath "$LLVM_INSTALL"/lib dex
to add a LC_RPATH
then that makes it work.
Ok, after much fiddling, and learning things I never wanted to know about linking :joy: I have concluded my original instructions were infact correct. And I just messed up the
# Special Mac things
cd $LLVM_INSTALL/lib
ln -s libLLVM.dylib libLLVM-8.dylib
install_name_tool -id $PWD/libLTO.dylib libLTO.dylib
install_name_tool -id $PWD/libLLVM.dylib libLLVM.dylib
install_name_tool -change '@rpath/libLLVM.dylib' $PWD/libLLVM.dylib libLTO.dylib
step.
Once can verify that that step that was done correctly via: otool -D
You should get:
~/progm/llvm-/lib❱ ✔ ❱ pwd
/Users/oxinabox/progmisc2/llvm-8.0.0.installed/lib
~/progm/llvm-/lib❱ ✔ ❱ otool -D libLLVM.dylib
libLLVM.dylib:
/Users/oxinabox/progmisc2/llvm-8.0.0.installed/lib/libLLVM.dylib
Rather than:
~/progm/backu/lib❱ ✔ ❱ otool -D libLLVM.dylib
libLLVM.dylib:
@rpath/libLLVM.dylib
Wow, heroic work! I'll link to your "how-to" message in the README. Thanks for digging into this!
I am trying to see if I can workout a way to do it with some prebuilt LLVM libraries because building LLVM takes ages
Excellent work on the tutorial! Unfortunately, I don't have the 15+GB disk space on my mac to build LLVM. I look forward to prebuilt libraries etc.
I have given bup on building and distributing LLVM libraries for now, after making no progress for a while. Mac OS Catelina is also increasing hostile to distributing ... anything with the latest increases requiring you to send them any binaries to sign.
You might want to buy a 32GB USB stick or something.
https://github.com/google-research/dex-lang/pull/61 removes the need for Stack, which was issues with musl, and thus with the BinaryBuilder cross compilation infastructure i was working with before. This means I might be able to get it working (if i can find time) so we can have be built binaries for Mac.
@oxinabox awesome! Please keep in mind that it does require Cabal 3, but I think that's the default you get from Homebrew
I have updated the instructions above so it now works with LLVM v9
I found that I was able to use the llvm@9
libraries from Homebrew if I passed the Cabal flag to llvm-hs
to use static linking. Given that a makefile
is used, the simplest way seemed to be to add the following to stack.yaml
:
flags:
llvm-hs:
shared-llvm: false
That flag is described in the llvm-hs
documentation.
It also requires compiling with /usr/local/opt/llvm@9/bin
on the PATH
, so that llvm-hs
can find the llvm-config
binary. (There's a note about why Homebrew doesn't add this path by default when llvm@9
is installed; it comes down to the fact that they want to allow multiple LLVM versions to be installed simultaneously.)
Apparently, linking statically against LLVM is not the "preferred" approach, but there's a Homebrew issue (and again here) where the Homebrew maintainers are refusing a patch to fix the dylib problem we're encountering, and saying it can only be addressed upstream. While that all gets sorted out, static linking seems like a solution with minimal fuss.
@lancelet that is awesome! Can you please try wrapping this path in if os(darwin)
and adding that to our Cabal file if that works?
@apaszke sure, I'll take a look to see if that's possible. It might take me a few days to get to it though.
Of course, take your time! Thanks for helping us figure it out
I'm not sure that there's any cabal-native way to add this path when building.
The core problem is that llvm-config
is a system binary required by llvm-hs
(link to its cabal file), rather than by Dex itself. So while there might (perhaps) be ways to hack in the additional path within the llvm-hs
cabal file, AFAICT, it's well outside the scope of the Dex cabal file to specify what path llvm-hs
might use when building.
I've read through as much of the Cabal docs as I can on this, but there might be something I've missed. I haven't come across anything that seems as though it would provide this facility, however.
https://github.com/google-research/dex-lang/issues/2#issuecomment-557793009 <- That's a good writeup @oxinabox Thanks very much. I managed to get as far as step 136/138 by following your guide after running make -j4
to build Dex:
.../.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-osx/Cabal-2.4.0.1 build --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
Progress 136/138
make: *** [build] Error 1
@8bitmp3 can you please post some more context of the failure?
Hey @apaszke , yes sure. These are the steps I took.
Install stack with curl -sSL https://get.haskellstack.org/ | sh
Install LLVM 9 as follows:
# 1. Create a directory for LLVM 9
mkdir llvm
cd llvm
# 2. Download and unpack LLVM 9.0.0
curl -O https://releases.llvm.org/9.0.0/llvm-9.0.0.src.tar.xz
tar -xzf llvm-9.0.0.src.tar.xz
# 3. Set up a build directory for LLVM
mkdir builddir
# 4. Set up separate source and installed directories
mkdir llvm-9.0.0.src
mkdir installed
# 5. Create useful environment variables
export LLVM_DIR="/Users/{USER}/llvm"
export LLVM_INSTALL="$LLVM_DIR"/installed
export LLVM_SRC=$LLVM_DIR/llvm-9.0.0.src/
# 6. Build/configure LLVM
cd builddir
cmake $LLVM_SRC -DCMAKE_INSTALL_PREFIX=$LLVM_INSTALL -DLLVM_BUILD_LLVM_DYLIB=True -DLLVM_LINK_LLVM_DYLIB=True
cmake --build . -j 4
cmake --build . --target install
cd $LLVM_INSTALL/lib
ln -s -F libLLVM.dylib libLLVM-9.dylib
install_name_tool -id $PWD/libLTO.dylib libLTO.dylib
install_name_tool -id $PWD/libLLVM.dylib libLLVM.dylib
install_name_tool -change '@rpath/libLLVM.dylib' $PWD/libLLVM.dylib libLTO.dylib
export PATH="$LLVM_INSTALL"/bin:$PATH
llvm-config --libdir
# 8. Install Dex
mkdir dex-source
cd dex-source
export DEX_SRC="/Users/{USER}/dex-source"
git clone https://github.com/google-research/dex-lang.git
cd dex-lang
make -j4
After running the last command—make -j4
—the output indicated that there were 138 "steps" (?) and all was going well until it hit 136/138:
.../.stack/setup-exe-cache/x86_64-osx/Cabal-simple_mPHDZzAJ_2.4.0.1_ghc-8.6.5 --builddir=.stack-work/dist/x86_64-osx/Cabal-2.4.0.1 build --ghc-options " -fdiagnostics-color=always"
Process exited with code: ExitFailure 1
Progress 136/138
make: *** [build] Error 1
cc @oxinabox
I just tried compiling dex
on a Mac and can report that just installing stack
, llvm@9
using Homebrew (and making sure that it's on the PATH
and in CFLAGS/LDFLAGS
), and editing the stack.yml
file as stated in this comment worked perfectly!
@8bitmp3 please make sure that llvm-config
(from llvm@9
) is on your PATH
. Otherwise building the llvm-hs
package that we depend on will fail.
This is the llvm-config --libdir # Check this matches to $"LLVM_INSTALL"/lib
step.
If I was better at shell scripting that would be a assertion
I was having some trouble with OSX installation. The following 2 fixes got me through to a successful make tests
1) I ran into problems with C99 and aligned_alloc
. I simply replaced line 18 of cbits/libdex.c
, i.e.
char* result = aligned_alloc(alignment, nbytes);
with the following alternative POSIX alternative call
char* result = NULL;
if(posix_memalign(&result, alignment, nbytes) != 0)
result = NULL; // make sure errors are correctly reported
2) I followed the instructions on the llvm-hs
project page for how to install on OS X. They recommended a specific homebrew-based installation of LLVM that their project appears set up to detect. (command brew install llvm-hs/llvm/llvm-9
at this time) Maybe directing Dex users to their installation instructions (rather than this thread?) might be more reliable?
I get part-way through the
make
set and I hit an error relating tohinotify-0.4
System Details: