search

clash-lang / clash-compiler

Haskell to VHDL/Verilog/SystemVerilog compiler
https://clash-lang.org/
Other
1.43k stars 151 forks source link

No mechanism for standalone binary install #965

Closed Araneidae closed 3 years ago

Araneidae commented 4 years ago

I would like to investigate running Clash ... but (let's say) I don't have root on my target workstation. In the past with other systems I have been able to run a binary distribution in a local prefix (or build into a local prefix), but somehow Clash insists that I install a foreign package manager. This is a problem.

Let's say I have neither Nix nor Snap installed, that neither of these packages is going to be installed, I have no root access, and admin support is not going to be forthcoming. What are my options? By the way, ghc runs just fine.

martijnbastiaan commented 4 years ago

If GHC runs, would it be safe to assume you've got cabal installed too?

In that case, you can use cabal to run clash:

Fetch latest 1.0.x:

curl -s -L https://github.com/clash-lang/clash-compiler/archive/1.0.tar.gz | tar xz cd clash-compiler-1.0 cabal new-update

Run clash:

cabal new-run clash -- examples/Calculator.hs --vhdl

We can't offer a standalone binary for Clash (right now), as Clash depends on the exact version of GHC it was built with.

martijnbastiaan commented 4 years ago

Woops, the last command should have been

cabal new-run clash -- -iexamples examples/Calculator.hs --vhdl

Araneidae commented 4 years ago

Thanks for the very quick response! As you say, I had a little think and looked at the sources, and got what looks like a working clash (I've built against master as of today) with the following commands:

$ ghcup install 8.4.4
$ ghcup set 8.4.4
$ cabal update
$ cabal new-run --write-ghc-environment-files=always -- clash

After this, your Calculator example works.

Is there a cleaner way of running clash than with cabal new-run each time?

Alas, I'm very rusty with the tooling. I fiddle with Haskell about once a year, so I was very pleasantly surprised that all the builds ran without a hitch.

Presumably I should switch to 1.0 and stick with that while I try and make sense of things.

martijnbastiaan commented 4 years ago

Try this to install:

cabal new-install clash --installdir=${HOME}/bin

Then create these shell-scripts

clashi.sh:

#!/bin/bash
SOURCEDIR=${HOME}/clash-compiler-1.0
BINDIR=${HOME}/bin

export clash_ghc_datadir=${SOURCEDIR}/clash_ghc
export clash_lib_datadir=${SOURCEDIR}/clash_lib
export clash_prelude_datadir=${SOURCEDIR}/clash_prelude

${BINDIR}/clashi -package-env ${SOURCEDIR}/.ghc.environment.x86_64-linux-8.8.1 $@

clash.sh:

#!/bin/bash
SOURCEDIR=${HOME}/clash-compiler-1.0
BINDIR=${HOME}/bin

export clash_ghc_datadir=${SOURCEDIR}/clash_ghc
export clash_lib_datadir=${SOURCEDIR}/clash_lib
export clash_prelude_datadir=${SOURCEDIR}/clash_prelude

${BINDIR}/clash -package-env ${SOURCEDIR}/.ghc.environment.x86_64-linux-8.8.1 $@

Almost works on my machine. At first run it gives me an error:

<no location info>: error:
    Clash only works with the exact clash-prelude it was built with.
Clash was built with: clash-prelude-1.1.0-189457ad4525d4af545fff79572722beba436b21b84e33704551ecd778262bb1
So can't run with:    clash-prelude-1.1.0-inplace

Which can be "fixed" by replacing clash-prelude-1.1.0-inplace with clash-prelude-1.1.0-189457ad4525d4af545fff79572722beba436b21b84e33704551ecd778262bb1 in .ghc.environment.x86_64-linux-8.8.1. Note that you would probably need to replace 8.8.1 with 8.4.4.

As you can see, it's not ideal, but maybe it works for you.

Araneidae commented 4 years ago

I built with 8.4.4 because the readme says

we currently recommend running 8.4

Is this out of date now? Shall I try with an 8.8 release?

I'll stick with cabal-run for now while I try to make sense of things ... oh, it doesn't work outside the clash-compiler directory, that could rapidly get annoying. I think that's the only issue I need to workaround.

martijnbastiaan commented 4 years ago

Yeah, that's outdated, thanks for catching. Both 8.4 and 8.8 should work fine. edit: It's just 8.6 that's causing problems with documentation generation. 8.8 should be a bit quicker to compile than 8.4, so you might want to switch.

I'll stick with cabal-run for now while I try to make sense of things ... oh, it doesn't work outside the clash-compiler directory, that could rapidly get annoying. I think that's the only issue I need to workaround.

Ok! The scripts should work independent of cwd.

Araneidae commented 4 years ago

Turns out that 8.8 is not quite ready, doesn't seem to play well with cabal (https://github.com/haskell/cabal/issues/6327), so guess I'd better stick with 8.4 for now. There was an interesting comment that's news to me: "it's generally risky to use .1 versions of GHC"; I'll bear that in mind...

martijnbastiaan commented 4 years ago

"it's generally risky to use .1 versions of GHC"; I'll bear that in mind...

Speaking of risky... :-) Master is not considered stable. We do welcome you to use it though, as we'd rather see bug reports earlier than later.

@Araneidae I've fixed an error in the scripts I posted, so if you're using it, please re-copy.

Araneidae commented 4 years ago

Hmm. Running clash new-install fails mysteriously:

$ cabal new-install clash 
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-ghc-1.1.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-lib-1.1.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-prelude-1.1.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-testsuite-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-benchmark-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-profiling-prepare-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-profiling-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-term-0.0.0.tar.gz
cabal: filepath wildcard 'src/cbits/cosim_vpi.vpi' does not match any files.

I presume I'm running sensible versions of stuff:

$ cabal --version
cabal-install version 3.0.0.0
compiled using version 3.0.0.0 of the Cabal library 
$ ghc --version
The Glorious Glasgow Haskell Compilation System, version 8.4.4
martijnbastiaan commented 4 years ago

As a workaround:

rm -r clash-cosim

We're trying to reproduce it.

martijnbastiaan commented 4 years ago

As a more sensible workaround:

cabal new-build all

before running cabal new-install clash.

Araneidae commented 4 years ago

Where's a good place to talk about clash in general terms? I've probably got lots of questions, and here doesn't really seem like the right place.

Oh, that was interesting: I did git checkout v1.0.1 followed by your cabal new-run ... example, and it fails with:

Linking /scratch/git/clash-compiler/dist-newstyle/build/x86_64-linux/ghc-8.4.4/clash-ghc-1.0.1/x/clash/build/clash/clash ...
Loaded package environment from /scratch/git/clash-compiler/.ghc.environment.x86_64-linux-8.4.4

<no location info>: error:
    Clash only works with the exact clash-prelude it was built with.
Clash was built with: clash-prelude-1.0.1-inplace
So can't run with:    clash-prelude-1.1.0-inplace

Next cabal new-build all next (still on v1.0.1 checked out), and it fails with:

Completed    vty-5.25.1 (lib)
cabal: Failed to build clash-cosim-0.0.0 (which is required by
exe:clash-testsuite from clash-testsuite-0.0.0). The failure occurred during
the configure step.

Where is build stored? I'm guessing under the dist-newstyle directory; guess I should erase that and the .ghc.environment... file and try again.

Edit: No change. Guess the next step is to try head again?

Edit: Same issue on head, and (unsurprisingly) deleting clash-cosim hardly helps...! I'll see if I can figure out how to remove the dependency.

martijnbastiaan commented 4 years ago

Where's a good place to talk about clash in general terms? I've probably got lots of questions, and here doesn't really seem like the right place.

See: https://github.com/clash-lang/clash-compiler#community. Slack is where most of the core developers are.

Oh, that was interesting: I did git checkout v1.0.1 followed by your cabal new-run ... example, and it fails with:

Yeah.. vty.. Try git checkout 1.0 instead. Does this fix your issues?

Araneidae commented 4 years ago

Alas:

$ git checkout 1.0
$ rm -rf .ghc.environment.x86_64-linux-8.4.4 dist-newstyle/
$ cabal new-build all
Resolving dependencies...
Build profile: -w ghc-8.4.4 -O1
In order, the following will be built (use -v for more details):
...
[65 of 65] Compiling Clash.Backend.SystemVerilog ( src/Clash/Backend/SystemVerilog.hs, /scratch/git/clash-compiler/dist-newstyle/build/x86_64-linux/ghc-8.4.4/clash-lib-1.0.1/build/Clash/Backend/SystemVerilog.o )
cabal: Failed to build clash-cosim-0.0.0 (which is required by
exe:clash-testsuite from clash-testsuite-0.0.0). The failure occurred during
the configure step.

Just as a sanity check:

$ git status
On branch 1.0
Your branch is up to date with 'origin/1.0'.

nothing to commit, working tree clean
martijnbastiaan commented 4 years ago

Either there should be an error message in the ..., or you can expose it with cabal new-build clash-cosim.

Araneidae commented 4 years ago

Sorry, I did spot the relevant error after scrolling back. Running cabal new-build clash-cosim says:

$ cabal new-build clash-cosim
Build profile: -w ghc-8.4.4 -O1
In order, the following will be built (use -v for more details):
 - clash-cosim-0.0.0 (lib:clash-cosim) (first run)
Configuring clash-cosim-0.0.0...
Failed to execute vvp. System reported:

    vvp: readCreateProcessWithExitCode: runInteractiveProcess: exec: does not exist (No such file or directory)

Vvp is part of the iverilog package. On Ubuntu systems, run:

    sudo apt install iverilog

CallStack (from HasCallStack):
  error, called at /scratch/git/clash-compiler/dist-newstyle/build/x86_64-linux/ghc-8.4.4/clash-cosim-0.0.0/setup/setup.hs:88:15 in main:Main

What on earth is iverilog? Another dependency. Damn. Do I really need this? (It's not an available package on my system...) A pretty concealed dependency, too :( I don't suppose I can fake it, either. Does this rabbit hole go much deeper, do you know?!

Edit: Well, at least it seems to configure and build smoothly enough.

Thank you very much for the help so far!

martijnbastiaan commented 4 years ago

If you don't need cosim (verilog cosimulation), you can remove the clash-cosim directory. It should then skip building it.

martijnbastiaan commented 4 years ago

Also: if you're not sure, you don't need it :P

Araneidae commented 4 years ago

Well, it doesn't like it if I just rebuild having deleted clash-cosim:

$ rm -rf clash-cosim/
$ cabal new-build all
Resolving dependencies...
cabal: Could not resolve dependencies:
[__0] trying: clash-testsuite-0.0.0 (user goal)
[__1] rejecting: clash-testsuite:-cosim (constraint from config file, command
line flag, or user target requires opposite flag selection)
[__1] trying: clash-testsuite:+cosim
[__2] unknown package: clash-cosim (dependency of clash-testsuite +cosim)
[__2] fail (backjumping, conflict set: clash-cosim, clash-testsuite,
clash-testsuite:cosim)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: clash-testsuite,
clash-testsuite:cosim, clash-cosim

So it looks like clash-testsuite references it somehow. I'll clean and rebuild to be sure. Edit: nope, same error.

Edit: Hmm: I see cosim as a flag in testsuite/clash-testsuite.cabal defaulting to False ... and yet something is setting it :(

martijnbastiaan commented 4 years ago

Well, we were doing new-build all to workaround the clash-cosim sdist error, so I think you can just do cabal new-install clash now :).

Araneidae commented 4 years ago

Oh, I totally don't understand what cabal is up to! So is cabal ... all setting the cosim flag? I have to rebuild again of course, will try the install once that's done. That'll have to be tomorrow...

Edit: So I'm guessing there's a list of targets that cabal will build for me when I say all ... is there a direct way to get that list from cabal? I'm guessing something like cabal new-build clash-lib clash-benchmark ... clash-prelude clash-ghc would be the answer?

Damn damn damn:

$ cabal new-install clash
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-ghc-1.0.1.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-lib-1.0.1.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-prelude-1.0.1.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-testsuite-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-benchmark-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-profiling-prepare-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-profiling-0.0.0.tar.gz
Wrote tarball sdist to
/scratch/git/clash-compiler/dist-newstyle/sdist/clash-term-0.0.0.tar.gz
cabal: filepath wildcard 'src/cbits/cosim_vpi.vpi' does not match any files

and removing clash-cosim again doesn't fix this (I get the "Could not resolve dependencies" error instead).

martijnbastiaan commented 4 years ago

Edit: @leonschoorl 's instructions are better

leonschoorl commented 4 years ago

Try replacing cabal.project with just:

packages:
  ./clash-ghc/*.cabal,
  ./clash-lib/*.cabal,
  ./clash-prelude/*.cabal
Araneidae commented 4 years ago

Build and install worked, will make the scripts work and report back tomorrow (I hope!)

Araneidae commented 4 years ago

Looking pretty promising, but not quite there. I had to do at least one run with --write-ghc-environment-files=always to get the .ghc.environment.x86_64-linux-8.4.4 file (so that's what that switch does), and for a while I thought I wasn't getting the clash-prelude error message -- it doesn't seem to kick in until I actually try synthesising.

Still doesn't quite work. I've taken a snapshot of the .ghc.environment... file and am using that in clash.sh so I can compare the behaviour of cabal new-run clash and clash.sh, and I get the following result when running clash.sh:

$ clash.sh -iexamples examples/Calculator.hs --vhdl
Loaded package environment from /home/mga83/bin/clash.ghc.environment.x86_64-linux-8.4.4
Loaded package environment from /home/mga83/bin/clash.ghc.environment.x86_64-linux-8.4.4
GHC: Parsing and optimising modules took: 3.939s
GHC: Loading external modules from interface files took: 1.157s
GHC: Parsing annotations took: 0.002s

<no location info>: error:
    Clash error call:
    No BlackBox definition for 'Clash.Signal.Internal.unsafeToReset' even though this value was annotated with 'HasBlackBox'.
CallStack (from HasCallStack):
  error, called at src/Clash/Primitives/Util.hs:126:11 in clash-lib-1.0.1-ee216d56e0e8ab3dc597cb128f93b411ee8894d461d602a0a6af3613ebb8ddfb:Clash.Primitives.Util

Compare this with:

$ cabal new-run clash -- -iexamples examples/Calculator.hs --vhdl
Up to date
Loaded package environment from /scratch/git/clash-compiler/.ghc.environment.x86_64-linux-8.4.4
GHC: Parsing and optimising modules took: 3.774s
GHC: Loading external modules from interface files took: 0.589s
GHC: Parsing annotations took: 0.001s
Clash: Parsing and compiling primitives took 0.872s
GHC+Clash: Loading modules cumulatively took 6.116s
Clash: Compiling Calculator.topEntity
Clash: Using cached result for: Calculator.topEntity
Clash: Compiling: Calculator.testBench
Clash: Using cached result for: Calculator.testBench
Clash: Total compilation took 6.126s

For reference, here is my clash.sh:

#!/bin/bash
SOURCEDIR=/scratch/git/clash-compiler
BINDIR=$HOME/.cabal/bin
PATH=$HOME/.ghcup/bin:"$PATH"

export clash_ghc_datadir=$SOURCEDIR/clash_ghc_datadir
export clash_lib_datadir=$SOURCEDIR/clash_lib_datadir
export clash_prelude_datadir=$SOURCEDIR/clash_prelude_datadir

HERE="$(dirname "$0")"

$BINDIR/clash -package-env \
    "$HERE"/clash.ghc.environment.x86_64-linux-8.4.4 "$@"

and here is what I've done to the environment file:

$ diff .ghc.environment.x86_64-linux-8.4.4 ~/bin/clash.ghc.environment.x86_64-linux-8.4.4 
10c10
< package-db dist-newstyle/packagedb/ghc-8.4.4
---
> package-db /scratch/git/clash-compiler/dist-newstyle/packagedb/ghc-8.4.4
54c54
< package-id clash-prelude-1.0.1-inplace
---
> package-id clash-prelude-1.0.1-c13e38a00780910d1021278944944eea30bce330ce4f5b1e70b42649a6dd955c

(Using the original .ghc... with just the clash-prelude hack doesn't change this behaviour.) I'm currently using branch 1.0, but I guess this doesn't make much difference?

martijnbastiaan commented 4 years ago

Please recopy the contents for clash.sh from this issue :). Your script seems to be the old one, with _datadir still in it.

Araneidae commented 4 years ago

Aha! There was also a second bug in your scripts, which I was wondering at: I had to change _ to - to match the actual directory names, and now it works :)

For reference, here's the final working version of clash.sh:

#!/bin/bash
SOURCEDIR=/scratch/git/clash-compiler
BINDIR=$HOME/.cabal/bin
PATH=$HOME/.ghcup/bin:"$PATH"

export clash_ghc_datadir=$SOURCEDIR/clash-ghc
export clash_lib_datadir=$SOURCEDIR/clash-lib
export clash_prelude_datadir=$SOURCEDIR/clash-prelude

HERE="$(dirname "$0")"

$BINDIR/clash -package-env \
    "$HERE"/clash.ghc.environment.x86_64-linux-8.4.4 "$@"

So to wrap up:

  1. Modify cabal.project as advised above (basically remove everything except the core definitions).
  2. Install with cabal new-install clash.
  3. Run clash in-place at least once with --write-ghc-environment-files=always to get a .ghc.environment... file.
  4. Modify (copy of) .ghc... file as advised above.
  5. The original sources need to remain accessible.
  6. Use the wrapper scripts.

I guess this is good enough for me for now, but I won't close this issue as clearly either clash itself can be improved, or these instructions can be put somewhere.

Thank you ever so much for all your help. Now I'll try and make sense of clash itself!

martijnbastiaan commented 4 years ago

Aha! There was also a second bug in your scripts, which I was wondering at:

Oh no :see_no_evil: . Thanks for catching that!

I guess this is good enough for me for now, but I won't close this issue as clearly either clash itself can be improved, or these instructions can be put somewhere.

I'll move it to the clash-compiler repository, so we won't lose track of it.

Now I'll try and make sense of clash itself!

Good luck and have fun :)

martijnbastiaan commented 3 years ago

You can now try Clash by using a Clash starter project. This only needs Stack or Cabal+GHC, both of which can be installed without root permissions.