Cabal build error with documentation enabled for ghc-8.6.5

[mouse07410]

MacOS Catalina 10.15.4, Xcode-11.4, Cabal-3.2.0.0, GHC-8.8.3 (8.6.5 and 8.10.1 are also installed). Current HIE master.

Successfully built hie-8.8.3:

$ ./cabal-hie-install hie
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - hie-install-0.8.1.0 (lib) (first run)
 - fake-package-0 (exe:script) (first run)
Configuring library for hie-install-0.8.1.0..
Preprocessing library for hie-install-0.8.1.0..
Building library for hie-install-0.8.1.0..
[1 of 8] Compiling BuildSystem      ( src/BuildSystem.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/BuildSystem.o )
. . . . .
Completed    haskell-ide-engine-1.3 (exe:hie-wrapper)
Installing   haskell-ide-engine-1.3 (exe:hie)
Completed    haskell-ide-engine-1.3 (exe:hie)
Symlinking 'hie'
Symlinking 'hie-wrapper'
Copied executables hie-wrapper, hie, hie-8.8 and hie-8.8.3 to /Users/ur20980/.cabal/bin
Build completed in 6m35s

Tried to rebuild hie-8.6.5 (because it's v1.1). Failed:

$ ./cabal-hie-install hie-8.6.5
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (exe:script) (configuration changed)
Configuring executable 'script' for fake-package-0..
Preprocessing executable 'script' for fake-package-0..
Building executable 'script' for fake-package-0..
[1 of 1] Compiling Main             ( Main.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script-tmp/Main.dyn_o )
Linking /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script ...
# cabal (for check)
# git (for submodules)
# git (for submodules)
Options:
    Verbosity level: Info
# cabal (for hie-8.6.5)
# cabal (for hie-8.6.5)
Wrote tarball sdist to
/Users/ur20980/src/haskell-ide-engine/dist-newstyle/sdist/haskell-ide-engine-1.3.tar.gz
Wrote tarball sdist to
/Users/ur20980/src/haskell-ide-engine/dist-newstyle/sdist/hie-plugin-api-1.3.tar.gz
. . . . .
Completed    haskell-src-exts-1.22.0 (lib)
Haddock      ghc-lib-parser-8.8.3.20200224 (lib)
Installing   ghc-lib-parser-8.8.3.20200224 (lib)
Completed    ghc-lib-parser-8.8.3.20200224 (lib)
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.3 and exe:hie from
haskell-ide-engine-1.3).
Symlinking 'hie'
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-2524b75f:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
* Raised the exception:
/Users/ur20980/.cabal/bin/hie: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

Update

Setting the current GHC to 8.6.5 (and removing dist-newstyle/ to ensure clean rebuild) did not help - same error in the same place:

$ ./cabal-hie-install hie-8.6.5
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - Cabal-3.0.2.0 (lib) (requires build)
 - binary-orphans-1.0.1 (lib) (requires build)
. . . . .
Installing   cryptonite-0.26 (lib)
Completed    cryptonite-0.26 (lib)
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.3 and exe:hie from
haskell-ide-engine-1.3).
Symlinking 'hie'
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-54f4c067:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
* Raised the exception:
/Users/ur20980/.cabal/bin/hie: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

[jneira]

The log looks a little bit weird, cause it says the script is able to create the symlinks for hie and hie-wrapper but then failed to copy the hie one to hie-8.6.5 and hie-8.6:

https://github.com/haskell/haskell-ide-engine/blob/f2c727b284587484b7ca0ab4096336190c3834c6/install/src/Cabal.hs#L77-L78

Does hie exist in /Users/ur20980/.cabal/bin after the failed install? To know your cabal version could be useful too.

[fendor]

Maybe symlinks are handled differently on macos? If it works on GHC 8.8.3...

[mouse07410]

To know your cabal version could be useful too.

Silly me. It's Cabal-3.2.0.0 from Hackage (as anything I build from Cabal GitHub repo does not work).

Does hie exist in /Users/ur20980/.cabal/bin after the failed install?

I don't remember - but hie-wrapper certainly does not. I'll do another build and report.

Also, the Haddock failure message in the failed install - was it a warning or an error?

Update

There are only symlinks to hie and hie-wrapper in ~/.cabal/bin - but they point at non-existing directories in ~/.cabal/store/ghc-8.6.5`:

$ 
$ ll ~/.cabal/bin/hie ~/.cabal/bin/hie-wrapper
lrwxr-xr-x  1 ur20980  179427977  51 Apr 16 08:10 /Users/ur20980/.cabal/bin/hie@ -> ../store/ghc-8.6.5/hskll-d-ngn-1.3-5bd1c25a/bin/hie
lrwxr-xr-x  1 ur20980  179427977  59 Apr 16 08:10 /Users/ur20980/.cabal/bin/hie-wrapper@ -> ../store/ghc-8.6.5/hskll-d-ngn-1.3-e9d018cf/bin/hie-wrapper
$ ll ~/.cabal/store/ghc-8.6.5/hskll-d-ngn-1.3-5bd1c25a ~/.cabal/store/ghc-8.6.5/hskll-d-ngn-1.3-e9d018cf
ls: /Users/ur20980/.cabal/store/ghc-8.6.5/hskll-d-ngn-1.3-5bd1c25a: No such file or directory
ls: /Users/ur20980/.cabal/store/ghc-8.6.5/hskll-d-ngn-1.3-e9d018cf: No such file or directory
$ 

[jneira]

No totally sure but it seems we could close this one.

[mouse07410]

Actually, it looks like in the process of fixing one we broke another. Now ./cabal-hie-install hie-8.8.3 fails too:

$ rm -rf dist-newstyle
$ ./cabal-hie-install hie-8.8.3 2>&1 | tee cabal-build-8.8.3.txt
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - hie-install-0.8.1.0 (lib) (first run)
 - fake-package-0 (exe:script) (first run)
Configuring library for hie-install-0.8.1.0..
Preprocessing library for hie-install-0.8.1.0..
Building library for hie-install-0.8.1.0..
[1 of 8] Compiling BuildSystem      ( src/BuildSystem.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/BuildSystem.o )
[2 of 8] Compiling Print            ( src/Print.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Print.o )
[3 of 8] Compiling Version          ( src/Version.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Version.o )
[4 of 8] Compiling Stack            ( src/Stack.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Stack.o )
[5 of 8] Compiling Env              ( src/Env.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Env.o )
[6 of 8] Compiling Help             ( src/Help.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Help.o )
[7 of 8] Compiling Cabal            ( src/Cabal.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/Cabal.o )
[8 of 8] Compiling HieInstall       ( src/HieInstall.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/build/HieInstall.o )
Preprocessing library for hie-install-0.8.1.0..
Running Haddock on library for hie-install-0.8.1.0..
Haddock coverage:
Warning: 'stack' is out of scope.
    If you qualify the identifier, haddock can try to link it anyway.
Warning: 'ghc' is out of scope.
    If you qualify the identifier, haddock can try to link it anyway.
Warning: 'cabal' is out of scope.
    If you qualify the identifier, haddock can try to link it anyway.
  25% (  1 /  4) in 'HieInstall'
  Missing documentation for:
    Module header
    defaultMain (src/HieInstall.hs:14)
    buildIcuMacosFix (src/HieInstall.hs:98)
Warning: HieInstall: could not find link destinations for:
    VersionNumber
Documentation created:
/Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/hie-install-0.8.1.0/doc/html/hie-install/index.html
Configuring executable 'script' for fake-package-0..
Preprocessing executable 'script' for fake-package-0..
Building executable 'script' for fake-package-0..
[1 of 1] Compiling Main             ( Main.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script-tmp/Main.dyn_o )
Linking /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script ...
# cabal (for check)
# git (for submodules)
# git (for submodules)
Options:
    Verbosity level: Info
# cabal (for hie-8.8.3)
# cabal (for hie-8.8.3)
cabal: Could not resolve dependencies:
[__0] trying: cabal-helper-1.0.0.0 (user goal)
[__1] next goal: haskell-ide-engine (user goal)
[__1] rejecting: haskell-ide-engine-1.4 (conflict: cabal-helper==1.0.0.0,
haskell-ide-engine => cabal-helper>=1.1 && <1.2)
[__1] fail (backjumping, conflict set: cabal-helper, haskell-ide-engine)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-helper, haskell-ide-engine

Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-2524b75f:Development.Shake.Internal.Args
* Depends on: hie-8.8.3
  at command, called at src/Cabal.hs:28:13 in hie-install-0.8.1.0-inplace:Cabal
* Raised the exception:
Development.Shake.command, system command failed
Command line: cabal v2-install -w /Users/ur20980/.ghcup/bin/ghc-8.8.3 --write-ghc-environment-files=never --installdir /Users/ur20980/.cabal/bin --max-backjumps=5000 exe:hie exe:hie-wrapper --overwrite-policy=always --project-file=cabal.project-8.8.3 -v1
Exit code: 1
Stderr:
cabal: Could not resolve dependencies:
[__0] trying: cabal-helper-1.0.0.0 (user goal)
[__1] next goal: haskell-ide-engine (user goal)
[__1] rejecting: haskell-ide-engine-1.4 (conflict: cabal-helper==1.0.0.0,
haskell-ide-engine => cabal-helper>=1.1 && <1.2)
[__1] fail (backjumping, conflict set: cabal-helper, haskell-ide-engine)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-helper, haskell-ide-engine

And no, this issue isn't fixed either:

$ git pull --recurse-submodules
remote: Enumerating objects: 101, done.
remote: Counting objects: 100% (101/101), done.
remote: Compressing objects: 100% (40/40), done.
remote: Total 78 (delta 54), reused 50 (delta 33), pack-reused 0
Unpacking objects: 100% (78/78), 10.69 KiB | 185.00 KiB/s, done.
From https://github.com/haskell/haskell-ide-engine
   6dc7d430..51b97395  master     -> origin/master
 * [new tag]           1.4        -> 1.4
Updating 6dc7d430..51b97395
Fast-forward
 Changelog.md                                | 31 +++++++++++++++++++++++++++++++
 cabal.project                               | 10 +++-------
 haskell-ide-engine.cabal                    | 10 +++++-----
 hie-plugin-api/Haskell/Ide/Engine/Cradle.hs |  5 +++--
 hie-plugin-api/hie-plugin-api.cabal         |  2 +-
 src/Haskell/Ide/Engine/Plugin/Package.hs    |  5 +++--
 stack-8.4.2.yaml                            | 16 ++++++++++------
 stack-8.4.3.yaml                            | 16 ++++++++++------
 stack-8.4.4.yaml                            | 16 ++++++++++------
 stack-8.6.4.yaml                            | 18 ++++++++++++------
 stack-8.6.5.yaml                            | 13 +++++++------
 stack-8.8.2.yaml                            |  9 ++++-----
 stack-8.8.3.yaml                            | 16 ++++++++--------
 stack.yaml                                  | 16 ++++++++--------
 14 files changed, 115 insertions(+), 68 deletions(-)
$ time ./cabal-hie-install hie-8.6.5
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (exe:script) (configuration changed)
Configuring executable 'script' for fake-package-0..
Preprocessing executable 'script' for fake-package-0..
Building executable 'script' for fake-package-0..
[1 of 1] Compiling Main             ( Main.hs, /Users/ur20980/Src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script-tmp/Main.dyn_o )
Linking /Users/ur20980/Src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script ...
# cabal (for check)
# git (for submodules)
# git (for submodules)
Options:
    Verbosity level: Info
# cabal (for hie-8.6.5)
# cabal (for hie-8.6.5)
Wrote tarball sdist to
/Users/ur20980/Src/haskell-ide-engine/dist-newstyle/sdist/haskell-ide-engine-1.4.tar.gz
Wrote tarball sdist to
/Users/ur20980/Src/haskell-ide-engine/dist-newstyle/sdist/hie-plugin-api-1.4.tar.gz
Resolving dependencies...
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - io-storage-0.3 (lib:io-storage) (requires build)
 - Diff-0.4.0 (lib) (requires build)
 - SHA-1.6.4.4 (lib) (requires build)
 - StateVar-1.2 (lib) (requires build)
 - blaze-markup-0.8.2.5 (lib) (requires build)
 - base-compat-batteries-0.11.1 (lib) (requires build)
 - bsb-http-chunked-0.0.0.4 (lib) (requires build)
 - byteorder-1.0.4 (lib:byteorder) (requires build)
 - bytestring-trie-0.2.5.0 (lib) (requires build)
 - clock-0.7.2 (lib) (requires build)
. . . . .
Installing   haskell-src-exts-1.22.0 (lib)
Completed    haskell-src-exts-1.22.0 (lib)
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).
Symlinking 'hie'
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-2524b75f:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
* Raised the exception:
/Users/ur20980/.cabal/bin/hie: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

To remind:

$ stack --version && cabal --version && shake --version && ghcup list
Version 2.3.1, Git revision de2a7b694f07de7e6cf17f8c92338c16286b2878 (8103 commits) x86_64 hpack-0.33.0
cabal-install version 3.2.0.0
compiled using version 3.2.0.0 of the Cabal library 
Shake build system, version 0.18.5

   Tool  Version Tags                                 Notes
✗  ghc   7.10.3  base-4.8.2.0                              
✗  ghc   8.0.2   base-4.9.1.0                              
✗  ghc   8.2.2   base-4.10.1.0                             
✗  ghc   8.4.1   base-4.11.0.0                             
✗  ghc   8.4.2   base-4.11.1.0                             
✗  ghc   8.4.3   base-4.11.1.0                             
✗  ghc   8.4.4   base-4.11.1.0                             
✗  ghc   8.6.1   base-4.12.0.0                             
✗  ghc   8.6.2   base-4.12.0.0                             
✗  ghc   8.6.3   base-4.12.0.0                             
✗  ghc   8.6.4   base-4.12.0.0                             
✓  ghc   8.6.5   base-4.12.0.0                             
✗  ghc   8.8.1   base-4.13.0.0                             
✗  ghc   8.8.2   base-4.13.0.0                             
✔✔ ghc   8.8.3   recommended,base-4.13.0.0        
✓  ghc   8.10.1  latest,base-4.14.0.0             
✗  cabal 2.4.1.0                                           
✗  cabal 3.0.0.0                                           
✔✔ cabal 3.2.0.0 latest,recommended      
✔✔ ghcup 0.1.5   latest,recommended      
$ 

[fendor]

To me this looks like you need to update your hackage-index? E.g. cabal update?

[mouse07410]

To me this looks like you need to update your hackage-index? E.g. cabal update?

Unlikely, because I'm doing it regularly on a weekly (when not daily) basis.

But to oblige ;-)

$ cabal update
Downloading the latest package list from hackage.haskell.org
To revert to previous state run:
    cabal v2-update 'hackage.haskell.org,2020-05-15T18:47:38Z'
$ time ./cabal-hie-install hie-8.8.3
Resolving dependencies...
Build profile: -w ghc-8.8.3 -O1
In order, the following will be built (use -v for more details):
 - fake-package-0 (exe:script) (configuration changed)
Configuring executable 'script' for fake-package-0..
Preprocessing executable 'script' for fake-package-0..
Building executable 'script' for fake-package-0..
[1 of 1] Compiling Main             ( Main.hs, /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script-tmp/Main.dyn_o )
Linking /Users/ur20980/src/haskell-ide-engine/dist-newstyle/build/x86_64-osx/ghc-8.8.3/fake-package-0/x/script/build/script/script ...
# cabal (for check)
# git (for submodules)
# git (for submodules)
Options:
    Verbosity level: Info
# cabal (for hie-8.8.3)
# cabal (for hie-8.8.3)
cabal: Could not resolve dependencies:
[__0] trying: cabal-helper-1.0.0.0 (user goal)
[__1] next goal: haskell-ide-engine (user goal)
[__1] rejecting: haskell-ide-engine-1.4 (conflict: cabal-helper==1.0.0.0,
haskell-ide-engine => cabal-helper>=1.1 && <1.2)
[__1] fail (backjumping, conflict set: cabal-helper, haskell-ide-engine)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-helper, haskell-ide-engine

Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-2524b75f:Development.Shake.Internal.Args
* Depends on: hie-8.8.3
  at command, called at src/Cabal.hs:28:13 in hie-install-0.8.1.0-inplace:Cabal
* Raised the exception:
Development.Shake.command, system command failed
Command line: cabal v2-install -w /Users/ur20980/.ghcup/bin/ghc-8.8.3 --write-ghc-environment-files=never --installdir /Users/ur20980/.cabal/bin --max-backjumps=5000 exe:hie exe:hie-wrapper --overwrite-policy=always --project-file=cabal.project-8.8.3 -v1
Exit code: 1
Stderr:
cabal: Could not resolve dependencies:
[__0] trying: cabal-helper-1.0.0.0 (user goal)
[__1] next goal: haskell-ide-engine (user goal)
[__1] rejecting: haskell-ide-engine-1.4 (conflict: cabal-helper==1.0.0.0,
haskell-ide-engine => cabal-helper>=1.1 && <1.2)
[__1] fail (backjumping, conflict set: cabal-helper, haskell-ide-engine)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-helper, haskell-ide-engine

real    0m12.886s
user    0m7.657s
sys 0m2.199s

[fendor]

Very weird, I do not understand why cabal doesnt pick cabal-helper-1.1.0.0... I want an "explain build-plan" command >_>

---

Wait.... --project-file=cabal.project-8.8.3 what is that? In the command

cabal v2-install -w /Users/ur20980/.ghcup/bin/ghc-8.8.3 --write-ghc-environment-files=never --installdir /Users/ur20980/.cabal/bin --max-backjumps=5000 exe:hie exe:hie-wrapper --overwrite-policy=always --project-file=cabal.project-8.8.3 -v1

Is that yours? In the current master, no file such as cabal.project-8.8.3 exists, so, it should not be given to the cabal command? Or is that some weird bug in the installation script? If it is yours, maybe the index-state is set in there?

[mouse07410]

Wait.... --project-file=cabal.project-8.8.3 what is that?

$ ll cabal.project*
-rw-r--r--  1 ur20980  179427977  400 May 16 11:29 cabal.project
-rw-r--r--  1 ur20980  179427977  787 Apr  3 00:03 cabal.project-8.10.1
-rw-r--r--  1 ur20980  179427977  507 Apr  7 14:44 cabal.project-8.8.3
-rw-r--r--  1 ur20980  179427977  767 Mar 13 09:33 cabal.project-8.8.3-old
$ cat cabal.project-8.8.3
packages:
         ./
         ./hie-plugin-api/

         -- ./submodules/HaRe

source-repository-package
    type: git
    location: https://github.com/DanielG/cabal-helper.git
    tag: a18bbb2af92e9b4337e7f930cb80754f2408bcfd

tests: true

package haskell-ide-engine
  test-show-details: direct

-- Match the flag settings we use in stac builds
constraints:
        haskell-ide-engine +pedantic,
        hie-plugin-api     +pedantic

write-ghc-environment-files: never

index-state: 2020-04-06T20:27:36Z

In the command

cabal v2-install -w /Users/ur20980/.ghcup/bin/ghc-8.8.3 --write-ghc-environment-files=never --installdir /Users/ur20980/.cabal/bin --max-backjumps=5000 exe:hie exe:hie-wrapper --overwrite-policy=always --project-file=cabal.project-8.8.3 -v1

Is that yours?

Hmm... I'm sure I don't know enough to come up with this kind of stuff on my own. Probably, it was suggested to me in the course of resolving another issue back in the beginning of April...

...maybe the index-state is set in there?

As a matter of fact, indeed it was. I removed this file (or, rather, renamed it), and re-started the build for hie-8.8.3. It built fine.

. . . . .
Completed    haskell-ide-engine-1.4 (exe:hie)
Symlinking 'hie'
Symlinking 'hie-wrapper'
Copied executables hie-wrapper, hie, hie-8.8 and hie-8.8.3 to /Users/ur20980/.cabal/bin
Build completed in 10m16s

real    10m25.405s
user    21m19.262s
sys 1m51.354s

So, the only remaining problem is fixing the build for hie-8.6.5 (the subject of this issue). BTW, it is my understanding that I don't need to reset the default system GHC to 8.6.5 to do ./cabal-hie-install hie-8.6.5?

[fendor]

BTW, it is my understanding that I don't need to reset the default system GHC to 8.6.5 to do ./cabal-hie-install hie-8.6.5?

Yes, that is taken care for you. If you change the GHC on the path, it only influences what runs the installation script

So, the only remaining problem is fixing the build for hie-8.6.5 (the subject of this issue).

Is that still an issue? I used it quite often in the last weeks without problems on linux and windows

[jneira]

BTW, it is my understanding that I don't need to reset the default system GHC to 8.6.5 to do ./cabal-hie-install hie-8.6.5?

No, the script uses the --with-compiler=ghc-8.6.5 cabal flag to use the appropiate version

So the last error with ghc-8.6.5 was related with:

There are only symlinks to hie and hie-wrapper in ~/.cabal/bin - but they point at non-existing directories in ~/.cabal/store/ghc-8.6.5`

It is so, maybe the cabal store is corrupted and wipe it out ~/.cabal/store/ghc-8.6.5 (and the current symlinks) could help to make the build succesfult

[mouse07410]

...maybe the cabal store is corrupted...

Maybe, but I've tried this on two different machines...

...wiping out ~/.cabal/store/ghc-8.6.5 (and the current symlinks) could help to make the build successful

I really hate this remedy, and unfortunately it hasn't been effective (looks like Shake is screwed up):

Symlinking 'hie'
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-2524b75f:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
* Raised the exception:
/Users/ur20980/.cabal/bin/hie: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

BTW, when do I need to explicitly ./cabal-hie-install data?

What happens if I install hie-8.8.3 first, and hie-8.6.5 after that? It looks like it would re-symlink hie to hie-8.6.5? Or must I always install for "older" versions first?

[mouse07410]

I see no error or warning other than for haddock-api-2.22.0 documentation.

. . . . .
Installing   haskell-src-exts-1.22.0 (lib)
Completed    haskell-src-exts-1.22.0 (lib)
Haddock      ghc-lib-parser-8.8.2.20200205 (lib)
Installing   ghc-lib-parser-8.8.2.20200205 (lib)
Completed    ghc-lib-parser-8.8.2.20200205 (lib)
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).
Symlinking 'hie'
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shk-0.18.5-61e1ad34:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
* Raised the exception:
/Users/ur20980/.cabal/bin/hie: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

I used it quite often in the last weeks without problems on linux and windows

I hear you. First, this is MacOS, and more importantly, second - are you sure you built it with documentation?

Here's my ~/.cabal/config:

-- This config file was generated using the following versions
-- of Cabal and cabal-install:
-- Cabal library version: 3.2.0.0
-- cabal-install version: 3.2.0.0

repository hackage.haskell.org
  url: https://hackage.haskell.org/
  -- secure: True
  -- root-keys:
  -- key-threshold: 3

-- default-user-config:
-- require-sandbox: False
-- ignore-sandbox: False
-- ignore-expiry: False
http-transport: curl
-- nix: False
-- local-no-index-repo:
remote-repo-cache: /Users/ur20980/.cabal/packages
-- local-repo:
-- logs-dir: /Users/ur20980/.cabal/logs
world-file: /Users/ur20980/.cabal/world
-- store-dir:
-- verbose: 1
-- compiler: ghc
-- cabal-file:
-- with-compiler:
-- with-hc-pkg:
-- program-prefix: 
-- program-suffix: 
-- library-vanilla: True
library-profiling: False
shared: True
-- static:
executable-dynamic: True
-- executable-static: False
-- profiling:
-- executable-profiling:
-- profiling-detail:
-- library-profiling-detail:
optimization: True
-- debug-info: False
-- library-for-ghci:
-- split-sections: False
-- split-objs: False
-- executable-stripping:
library-stripping: False
-- configure-option:
-- user-install: True
-- package-db:
-- flags:
extra-include-dirs: /opt/local/include,/usr/local/include
-- deterministic:
-- cid:
extra-lib-dirs: /opt/local/lib/liconv,/opt/local/lib,/usr/local/lib
-- extra-framework-dirs:
extra-prog-path: /Users/ur20980/.cabal/bin
-- instantiate-with:
-- tests: False
-- coverage: False
-- library-coverage:
-- exact-configuration: False
. . . . .
-- allow-older: False
-- allow-newer: False
-- write-ghc-environment-files:
documentation: True
-- doc-index-file: $datadir/doc/$arch-$os-$compiler/index.html
-- target-package-db:
-- max-backjumps: 4000
-- reorder-goals: False
-- count-conflicts: True
. . . . .
-- root-cmd:
symlink-bindir: /Users/ur20980/.local/bin
build-summary: /Users/ur20980/.cabal/logs/build.log
-- build-log:
remote-build-reporting: anonymous
-- report-planning-failure: False
-- per-component: True
-- one-shot: False
-- run-tests:
jobs: 12
-- keep-going: False
. . . . .
-- install-method:
installdir: /Users/ur20980/.cabal/bin

Update

You really ought to start CI-building with documentation: True. I just validated that the only way to accomplish successful build of hie-8.6.5 is to disable documentation in the global Cabal config file ~/.cabal/config. Attempts to do something like ./cabal-hie-install hie-8.6.5 --disable-documentation fail the same way as shown above.

Screwing up my ~/.cabal/config and disabling documentation there, however, resulted in a successful build. Go figure:

. . . . .
Completed    ormolu-0.0.3.1 (lib)
Installing   hlint-2.2.11 (lib)
Completed    hlint-2.2.11 (lib)
Starting     haskell-ide-engine-1.4 (lib)
Building     haskell-ide-engine-1.4 (lib)
Installing   haskell-ide-engine-1.4 (lib)
Completed    haskell-ide-engine-1.4 (lib)
Starting     haskell-ide-engine-1.4 (exe:hie)
Starting     haskell-ide-engine-1.4 (exe:hie-wrapper)
Building     haskell-ide-engine-1.4 (exe:hie-wrapper)
Building     haskell-ide-engine-1.4 (exe:hie)
Installing   haskell-ide-engine-1.4 (exe:hie-wrapper)
Completed    haskell-ide-engine-1.4 (exe:hie-wrapper)
Installing   haskell-ide-engine-1.4 (exe:hie)
Completed    haskell-ide-engine-1.4 (exe:hie)
Symlinking 'hie'
Symlinking 'hie-wrapper'
Copied executables hie-wrapper, hie, hie-8.6 and hie-8.6.5 to /Users/ur20980/.cabal/bin
Build completed in 10m06s

$ hie-8.6.5 --version
Version 1.4 x86_64 ghc-8.6.5

[jneira]

Thanks for the detailed report, i would say that it seems to be a cabal bug. I mean, a documentation error or warning should not be cause a symlink/copy failure (if it is the cause) but not sure what could we do here to fix it.

Btw i can reproduce in my windows 10:

Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).
C:\sd\ghc-8.6.5\haskell-ide-e_-1.4-27a9161f04eb682e2fe38758622f89eca8c95b7e\bin\hie.exe: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)
Error when running Shake build system:
  at want, called at src\Development\Shake\Internal\Args.hs:83:69 in shake-0.18.5-0587b1b818f9ea6593f365b450ae926b455955e2:Development.Shake.Internal.Args
* Depends on: hie-8.6.5
  at command, called at src\\Cabal.hs:28:13 in hie-install-0.8.1.0-inplace:Cabal
* Raised the exception:
Development.Shake.command, system command failed
Command line: cabal v2-install -w D:\bin\stack\x86_64-windows\ghc-8.6.5\bin\ghc-8.6.5.exe --write-ghc-environment-files=never --installdir D:\bin --max-backjumps=5000 exe:hie exe:hie-wrapper --overwrite-policy=always --project-file=cabal.project --install-method=copy -v1
Exit code: 1
Stderr:
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).
C:\sd\ghc-8.6.5\haskell-ide-e_-1.4-27a9161f04eb682e2fe38758622f89eca8c95b7e\bin\hie.exe: copyFile:atomicCopyFileContents:withReplacementFile:copyFileToHandle:openBinaryFile: does not exist (No such file or directory)

Well maybe fixing the doc error in haddock-api-2.22.0 or use a newer version could fix it...

[jneira]

Well maybe fixing the doc error in haddock-api-2.22.0 or use a newer version could fix it...

I am afraid that it is not feasible to use haddock-api >= 2.23.0 with ghc-8.6.5 due to its base + ghc constraints

[mouse07410]

I mean, a documentation error or warning should not be cause a symlink/copy failure (if it is the cause)

It is not a symlink failure. Shake (?) presumably screws up something, so the directories containing hie and hie-wrapper that Cabal then symlinks - in fact do not exist, aka do not get created. So, the entire build appears to be failing at least for those components - not just the failure to symlink something successfully built.

Btw i can reproduce in my windows 10

Well, that's a step forward. But are you saying yo can't reproduce it on Linux? I would expect that all it would take is setting documentation: True.

What I do think is Cabal's bug - is that it appears to ignore --disable-documentation in the ./cabal-hie-build hie-8.6.5 --disable-documentation. For the build to succeed, it has to be disabled in ~/.cabal/config. Consequently, I think that if you try to CI this by adding --enable-documentation to the build command, it's not sufficient.

[fendor]

It is not a symlink failure.

It is, cabal v2-install seems to fail

Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-1.4 and exe:hie from
haskell-ide-engine-1.4).

But the error code seems wrong (e.g. 0 instead of non-zero), so shake tries to set the symlinks, but there is no source file, etc...

But are you saying yo can't reproduce it on Linux?

@jneira uses windows for their workstation, it is just the confirmation that it is reproducible.

What I do think is Cabal's bug

That is not a Cabal bug but a short-coming of the installation script.

[jneira]

But are you saying yo can't reproduce it on Linux?

Well, not sure where we wrote that "it is no reproduced on linux". In my case i could check only the ci and in fact it does not fail for cabal + ghc-8.6.5 + linux. In any case that would be before we were aware that the other piece of the puzzle is set documentation: true in the cabal config or --enable-documentation in the cli. Sorry if you think that we had to know it when we said it is not reproducible in linux (wherever it was).

That is not a Cabal bug but a short-coming of the installation script.

Yeah the installation doesn't let you pass arguments to the build tools (cabal or stack) used internally. It is not a bug cause it was not designed to do it, although it is a feature that we have considered at some point. The workaround is to change the global config or the cabal.project file.

It is not a symlink failure What I do think is Cabal's bug

I wanted to mean that it is a cabal bug cause it attempts to symlink (i already suspected that symlink was not broken in general 😛 ) using a source that doesnt exist. Cabal should either interrupt the build due the doc error or generate the executable and symlink it. Its actual behaviour dont seem correct for me.

A solution could be downgrade haddock-api to workaround the cabal bug, in the hope a previous compatible version, if exists, does not trigger the haddock error.

[jneira]

A solution could be downgrade haddock-api to workaround the cabal bug, in the hope a previous compatible version, if exists, does not trigger the haddock error.

I've just tested that cabal install -w ghc-8.6.5 --constraint="haddock-api < 2.22.0" --enable-documentation fails with me (also with older haddock-api versions) so i am afraid that workaround is not feasibe.

So, again, i think that we can do much more here to solve that.

[mouse07410]

the other piece of the puzzle is set documentation: true in the cabal config or --enable-documentation in the cli

Yep...

...the installation doesn't let you pass arguments to the build tools (cabal or stack) used internally. It is not a bug cause it was not designed to do it, although it is a feature that we have considered at some point

1. You're saying that the reason --disable-documentation did not work for me is because it was not passed to the build tools, despite this $@ thing at the end of the invocation in the cabal-hie-install script?

   $ cat ~/src/haskell-ide-engine/cabal-hie-install
   #!/bin/sh
   cabal v2-run ./install.hs --project-file install/shake.project $@

2. Would you please re-consider passing arguments...?

I've just tested that cabal install -w ghc-8.6.5 --constraint="haddock-api < 2.22.0" --enable-documentation fails with me

Unpleasant...

The workaround is to change the global config or the cabal.project file.

I'd like to strongly urge you to add cabal.project-8.6.5 file to your repo, with the appropriate parameter(s). Why not me adding it to my local? Because when I did that with cabal.project-8.8.3 it ended up breaking things when the configs diverged. Thus I really want it maintained as a part of the project, so any divergence/discrepancy is caught by CI or such.

[jneira]

You're saying that the reason --disable-documentation did not work for me is because it was not passed to the build tools, despite this $@ thing at the end of the invocation in the cabal-hie-install script?

That is right, there are two phases:

1. You compile and run the script with cabal or stack
   • The arguments you pass in that first call affects to the compilation of the script itself
   • F.e., if you call ./cabal-hie-install --with-compiler ghc-8.8.3 hie-8.6.5, you are using ghc-8.8.3 to compile the script that will use, in turn, ghc-8.6.5 to build haskell-ide-engine
   • When you run a script with cabal or stack you can pass arguments to the script at runtime, separating them with --. But those arguments are the standard ones for an application using the shake build systems (you can see them with ./cabal-hie-install -- --help)
   • You can add your custom arguments to that ones if we would want to pass them towards the build tools.
2. The script, at runtime, make calls to cabal and stack with the arguments fixed for the shake of simplicity:
   • The same script calls cabal of stack and they have incompatible args, they should be separated or wrapped
   • It calls several build tool commands: build, install, exec, run and no always you want to pass all arguments to all commands.
   • And we know there is a workaround: changing the cabal.project or the stack.yaml

So it is not trivial, although it would not be rocket science neither.

I'd like to strongly urge you to add cabal.project-8.6.5 file to your repo, with the appropriate parameter(s)

I am afraid that i dont know what parameter could we set to make the build succesful but documentation: false to let users to have documentation:true in their cabal global config (by default it is alreadyfalse), and i am not sure if that change, for that use case, worths create a new file.

[mouse07410]

I am afraid that i don't know what parameter could we set to make the build succesful but documentation: false to let users to have documentation:true in their cabal global config

That's exactly what I'm suggesting.

and i am not sure if that change, for that use case, worths create a new file.

It is worth it because if later on you find you need to add some parameter, or remove - this file would be under your maintenance. It reduces the chances of a user creating this file on his own, and getting screwed because the codebase and/or the situation changes.

[jneira]

@mouse07410 fair enough, i've tried to add it but i am afraid that we've hit another cabal bug trying to workaround the previous cabal bug: https://github.com/haskell/cabal/issues/6824

[jneira]

Fortunately in the issue they have made me remember a cabal feature to acomplish that:

package haddock-api
  documentation: False

😄

[jneira]

@mouse07410 i've merged the pr and the cabal.project-8.6.5 is in master, i've tried locally and cabal-hie-install hie-8.6.5works for me know

[mouse07410]

I confirm that it works. Thank you!