cabal install run immediately after cabal build unnecessarily rebuilds the project

[joeyh]

Describe the bug

I have just run cabal build. No source files have changed. I run cabal install (with some options), and it rebuilds the entire project from scratch.

To Reproduce

cabal unpack propellor-5.9.1
cd propellor-5.9.1
cabal build
cabal install --install-method=symlink --installdir=. exe:propellor --overwrite-policy=always

I notice that, if I cabal build exe:propellor, the cabal install does not re-build. The default cabal build of both executables and the library differing from what's it's being asked to install seem to be what causes the re-build. I think it should be possible to build everything and then only install part of what was build, without a rebuild.

System information

cabal-install version 3.0.0.0
compiled using version 3.0.1.0 of the Cabal library 

[joeyh]

I should mention that, while the above seems like suboptimal behavior, what I'm really wanting to do -- inexpensively symlinking the executable cabal has built to a place I need it -- is something cabal install seems ill-suited to.

Other performance problems include it loading the cabal package list, building the sdist (a 1.7 mb compressed tarball in my case), probably more.

What I'm probably going to use instead is cabal exec -- sh -c 'command -v propellor' to get the path to the binary and symlink it myself.

[andreasabel]

Same here. This behavior makes me hesitant to embrace v2-cabal yet, still preferring v1-cabal.

What would be a good workflow with v2-cabal? This looks natural:

cabal configure
cabal build
cabal install

but does not fly because the last step rebuilds everything---needlessly, it seems, isn't often copying/linking the executable all that is left to do?

[fgaz]

Context: While v1- was "autotools-like", v2- does not follow that model anymore and is more declarative instead. v2-install tries to be as reproducible as possible by doing a sdist before building and installing (and anecdotally this has helped people catch some bugs in their cabal files). This also has a simpler implementation, = less bugs.

@andreasabel So the sequence of commands for installing an executable is just

cabal install TARGET

Which builds the package one time only. build (and configure, which is optional) is only useful if you want to develop the package.

@joeyh for your usecase, the upcoming cabal 3.4 provides a list-bin command that inexpensively prints the paths of local executables, so you can do whatever you want with them without fragile exec hacks. Does that solve the issue?

[andreasabel]

@fgaz

build (and configure, which is optional) is only useful if you want to develop the package.

Indeed I develop https://github.com/agda/agda and want to install the executables after each build in order to test interactive Agda development with the newest bug-fixes in emacs. The v1 workflow was to run

cabal v1-install

in the working directory to get Agda to build and be installed. Of course, I will have to invoke this command several times until all the compilation error are fixed.

How do I proceed in v2?

I would run cabal build several times until my compilation errors are fixed, and then how to install the executables? cabal install is forbiddingly expensive, since compiling the 400 modules from scratch with aggressive optimization takes 30 minutes.

For me, the v1-install functionality works best. To port it to v2, could we have something like

cabal build --install-executables

that performs the installation with binaries built by build?

@phadej Since you ask what is holding me back from v2

I should mention that we don't test any of v1- commands anymore. The code is there, and if it works it is good, but if it doesn't, we wont actively pursue fixing it (rather concentrating on whatever issues are holding you from migration to v1-build).

the present issue is one that makes us stick to v1 in the Agda development.

[andreasabel]

My feature request would be to have this option of stack build:

--[no-]copy-bins         Enable/disable copying binaries to the local-bin-path
                           (see 'stack path') (default: disabled)

In the meantime, I have awked a workaround. I write

cabal build | tee >(copy-bins.awk)

which picks up the executable name from the Linking foo ... message of cabal build and links this executable in .cabal/bin.

copy-bins.awk:

#!/usr/local/bin/gawk --file

# To use with "cabal v2-build", e.g.
#
#   cabal build | tee >(copy-bins.awk)
#
# When "cabal build" outputs "Linking path/.../x/.../to/exe ..."
# we symlink this executable from our ".cabal/bin" directory.

match ($0, /^Linking (.+\/x\/.+) ...$/, res) {

  # CONFIGURE ME
  installdir = "~/.cabal/bin";

  # Capture the regex group (...) in src
  src = res[1];

  # Create the symlink
  ln_cmd = "ln -s -f "src" "installdir"/";
  print("[COPY-BINS]", ln_cmd);            # Tell what you are doing.
  system(ln_cmd);

}

[fgaz]

If the exec and list-bin+ ln -s workarounds aren't enough, we could consider accepting a pr that implements the conversion from local to store package, but there are a lot of details to think about (ex. rpath/relinking, data files, dependencies...). Even so, such a conversion has a fundamental problem over the above workarounds when doing iterative development: it would quickly increase the size of the store, and we have no garbage collection yet. I think list-bin plus ln -s is the best solution in this case.

[fgaz]

Some discussion in https://github.com/haskell/cabal/issues/7297#issuecomment-939285238 and partially in #7693 (but please let's use this ticket for this topic)

[mouse07410]

Please take a look at https://github.com/haskell/cabal/issues/7693#issuecomment-941214357, and feel free to respond here. Thanks!

[jneira]

After the info posted in https://github.com/haskell/cabal/issues/7297#issuecomment-939330830 and my tests in #7745 about the differences between build local packages and installing them

cannot be completely true, unit-ids (of local libraries) are different, Paths_ point to different directories, maybe something else I forget.

i would say that

• cabal install should behave as is (therefore continue rebuilding the local packages as remote ones), not only to reuse the code for cabal install local.or.remote.tar but to have a consistent behaviour across all target types.
  • it will make local libraries available in the store for other projects f.e.
• still i think we should add a cabal build --copy-bins to copy executables, tests and benches (and dynamic libs?) outside the build directory and make available the pattern cp $(cabal list-bin my exe) /some/path to users

[mouse07410]

First, yes - to be useful, --copy-bins must copy dynamic libraries in addition to copying the executables themselves. Otherwise, those executables would have zero chance to run.

Second, right now cabal list-bin appears broken. So, I assume cabal as-is would be unable to even locate the binaries to copy. cabal-plan does list the executables correctly - but it seems to have no idea about dynamic libs those executables depend on.

What are the effects of --enable-tests? Why cannot cabal v2-install build whatever it's (re-) building with that flag, and merely skip copying the <target>:test executables to the global store?

[Mikolaj]

Second, right now --list-bin appears broken in cabal.

Could you elaborate? (Best just point to ticket about that or open one).

What are the effects of --enable-tests?

Wrong ticket?

[mouse07410]

Second, right now cabal list-bin appears broken.

Could you elaborate? (Best just point to ticket about that or open one).

https://github.com/haskell/cabal/issues/7679

What are the effects of --enable-tests?

Wrong ticket?

No, an attempt to find a solution or a "working" workaround for this problem - when you build with --enable-tests, there's no way to avoid re-building a lot of dependencies during install. Based on this ticket, it looks like --enable-tests is not the only case when cabal rebuilds just-built stuff during install.

I agree that --copy-bins would've been a good workaround, if not a "total" solution - if it worked.

[jneira]

Afaik this was caused by #6906 which was fixed by #7753 with a regression test in #7759 Please open a new issue about improving list-bins if you think it is necessary

[jneira]

ugh i missed rebuild the project, sorry

[mouse07410]

Ping?

[Mikolaj]

@mouse07410: I'm afraid we haven't made any progress with designing --copy-bins nor splitting install or even incorporating cabal-env. That's my report for now. Do you have anything to report? Any new ideas?

[mouse07410]

Do you have anything to report? Any new ideas?

Alas, nothing new. :-( Frankly, I doubt I'm in the league - not enough Haskell or general FP experience or expertise. But it would be great if somebody figured how to fix --copy-bins and splitting install.

[mouse07410]

It seems that two problems are brought up here:

1. cabal install forces rebuild of several dependencies even when run immediately after a successful cabal build.

2. cabal --copy-bins omits dynamic libraries, thus ensuring that the copied executables become useless/broken (unable to run, being unable to load dynamic libraries they need).

Is there any progress in either of these two?

[Mikolaj]

I'm not aware of any progress, but I might have easily missed something. If you find any relevant ticket or a comment in another ticket, please kindly link here.