Open mouse07410 opened 2 years ago
This is intended:
It was introduced in #4879 to fix #4855. Things might have changed since then though (eg. https://github.com/haskell/cabal/pull/4886#issuecomment-365019271), judging by eg. #5143
To spell it out, you did --only-dependencies
and cabal is confused (do you want to run tests for the dependencies?), though perhaps it shouldn't be, see the ticket @fgaz mentioned.
To spell it out, you did
--only-dependencies
and cabal is confused (do you want to run tests for the dependencies?),
I want to build/install Agda, and run the tests. If I do build
and run tests first, and then install - it will recompile everything again, which is a fairly lengthy process. Doing it in one swoop via install makes it to compile the whole thing only once. And that's how it works with v1-install
. But since v1-
has unpleasant effect on the global environment, I'd much rather get it working with v2-
.
. . . though perhaps it shouldn't be, see the ticket @fgaz mentioned.
It would be great if this obstacle were taken care of.
I just tried removing
and there is no assertion failure, though tests aren't actually built. Maybe we just need the equivalent of #5143. I won't investigate this further right now though
I want to build/install Agda
So why do you do --only-dependencies
?
it will recompile everything again,
I hope that's not true (otherwise, please report). I you build
and test
and install
with the same flags, it should not rebuild any Haskell files, though it will re-link.
I hope that's not true (otherwise, please report). I you build and test and install with the same flags, it should not rebuild any Haskell files, though it will re-link.
unfortunately it's true for install: #6919
I hope that's not true (otherwise, please report). I you build and test and install with the same flags, it should not rebuild any Haskell files, though it will re-link.
unfortunately it's true for install: #6919
The example from #6919 is very sad but, if you squint, it's excluded by the "with the same flags" proviso (targets being not exactly flags, but being at least commandline arguments). And I've heard that stuff gets rebuilt without a good reason in other cases, which is why I'm fishing for a reproducible example.
It's still true even with the same flags: when installing, local packages are treated as remote and have to be rebuilt in the store
It's still true even with the same flags: when installing, local packages are treated as remote and have to be rebuilt in the store
So the "I notice that, if I cabal build exe:propellor
, the cabal install
does not re-build" success is not caused by the agreement of flags, but by cabal buil
treating local packages as remote in some cases?
The example from #6919 is very sad but, if you squint, it's excluded by the "with the same flags" proviso (targets being not exactly flags, but being at least commandline arguments).
Strictly speaking, those are different flags, albeit slightly. As I understand (please correct me if I'm wrong), --enable-tests
is not just about running the tests, it's about building in a different way that supports running the tests. Therefore, the resulting binaries for all the packages involved - the targets and their dependencies - will be different based on the presence or absence of --enable-tests
flag.
That's not unique to Haskell/Cabal - in C people debug and test with -g -O0
, but deploy to production with -O3
or better. Meaning that what was tested is not what actually runs in production. It is known to cause real problems in real life - for example: https://github.com/randombit/botan/issues/2802
. . . do you want to run tests for the dependencies?
To revisit this question - no, I don't really want to run tests on/for the dependencies - but these dependencies must be ready to support running tests on my project's target. I suspect, in the end there isn't much difference between whether I want to actually run tests for the dependencies, or merely build and install those dependencies in such a way that they can support their caller running tests.
Thus, I think that the correct behavior for Cabal in such a case should be - build the dependencies as if I were to run the tests for them, regardless of whether they are local or remote.
Comments, please?
I don't even understand the --only-dependencies flag here. It doesn't make a lot of sense in this workflow given how v-2 works. I think the flags are asking for something a bit nonsensical and the current behavior looks fine.
I think it is correct that install does not allow tests, since it doesn't make much sense.
To get the workflow you desire, you can just build with tests, then manually copy or symlink the binary. A future feature request could be to add a copy/symlink binary flag to the build of an executable.
I think it is correct that install does not allow tests, since it doesn't make much sense.
IMHO, on the contrary - it makes great sense to be able to run the test suite over the "production" (aka, installed) executable... At the very least, you'll know that the actual physical binary that passed or failed your tests is the same that is running your high-value tasks in deployment.
What am I missing?
@mouse07410: I'm pretty sure --enable-tests
does not affect how dependencies are built in any way other than adding testing framework dependencies to the mix and so possibly affecting versions of packges. Also, dependencies do not need to be prepared to make it possible for the main package to be tested. No preparation is needed [edit: again, except for adding the extra packages for test frameworks, etc.].
Regarding the ability to test exactly the thing you install, I think it makes sense, except that the installed executable would never be tested, because it's the libraries that are linked with the test component, executables are normally not used for tests. Generally, test component in cabal are rather unit tests for the code, while executables are tested using integration and similar tests running from the outside.
--enable-tests
does not affect how dependencies are built in any way other than adding testing framework dependencies to the mix and so possibly affecting versions of packges.
As I understand, both of the above mean - the binaries will be different. For an ideally simple code it may not matter - but I haven't seen such a code in real life.
Regarding the ability to test exactly the thing you install, I think it makes sense, except that the installed executable would never be tested, because it's the libraries that are linked with the test component, executables are normally not used for tests.
"Never say Never again!" ;-) This probably depends on what kind of tests you'd be running. They often - but not always - are unit tests.
As I understand, both of the above mean - the binaries will be different.
OK, you are right, in practice the compiled dependencies will be different. Moreover, if I wanted to test them afterwards, less would need to be rebuilt. However, I'm clarifying all these things in order to understand why you put --only-dependencies
in your commandline. Why is it?
"Never say Never again!" ;-) This probably depends on what kind of tests you'd be running. They often - but not always - are unit tests.
Haha, sure, but we need to focus on a particular use case. We don't have the resources to support everything. And we support only unit tests (more precisely, library tests) in cabal test components. Anybody using it to run integration tests (invoke an executable compiled in the same package) is on his own. Does it make sense?
We don't have the resources to support everything. And we support only unit tests (more precisely, library tests) in cabal test components. Anybody using it to run integration tests (invoke an executable compiled in the same package) is on his own. Does it make sense?
Hmm that sentence surprises me, f.e. in hls we have a few unit tests and lot, lot of integration tests in test stanzas which call the hls executable.
We use cabal build
with build-tool-depens
and everything works fine.
EDIT: i guess the sentence would make sense in the context of cabal v2-install
Hmm that sentence surprises me, f.e. in hls we have a few unit tests and lot, lot of integration tests in test stanzas which call the hls executable. We use
cabal build
withbuild-tool-depens
and everything works fine.
I may be wrong. But if I'm not, that means you are on your own. E.g., ensuring that the hls
exe is built before running the cabal test
probably requires some hacks? How do you do that?
I'm clarifying all these things in order to understand why you put
--only-dependencies
in your commandline. Why is it?
If I did that, I'd probably have a semi-reasonable explanation. Since it's a part of the installation setup of a large (too large for my comfort) Agda compiler that I'm only a user of (not a developer or a maintainer) - I've no clue why that argument is there. My layman's common sense suggests that it doesn't really belong, but...
I asked the question on the Agda repo (https://github.com/agda/agda/issues/4216#issuecomment-930443275), let's see what the answer is.
. E.g., ensuring that the
hls
exe is built before running thecabal test
probably requires some hacks? How do you do that?
build-tool-depends: haskell-language-server
do the trick (although it should be some other -tool-depends
, as mentioned in the issues about)
to workaround the (again) recompilation with --test-options
(recently fixed) we were using cabal run
I looked at the agda ticket. They use --only-dependencies with the build command, not the install command. Asking to install with --enable-tests set doesn't seem important to the use case they have. Similarly, they use --enable-tests with the build command, not the install command. I think this ticket is just confusing, and confused. Its not addressing a real world case.
I looked at the agda ticket. They use
--only-dependencies
with the build command, not the install command.
I don't think this is right, because with Cabal their build process does not use build
command at all.
The ticket talks about various (failed) experiments.
Similarly, they use
--enable-tests
with the build command, not the install command.
I don't know what you're basing this conclusion on. I say they do use --enable-tests
with the install command. One reason: their Makefile does not use build
command with Cabal - only with Stack, but that's a different story.
Their installation from the source (that includes unit and integration tests) is designed to run from Makefile. Here's what the successful installation log shows (I'd be happy to post the entire log, if anybody needs it):
========================= Installing dependencies using Cabal ============
time cabal v1-install --disable-documentation --builddir=./dist-2.6.3-ghc-9.0.1 --enable-tests -foptimise-heavily --only-dependencies -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS"
Resolving dependencies...
All the requested packages are already installed:
Use --reinstall if you want to reinstall anyway.
real 0m5.011s
user 0m1.467s
sys 0m1.002s
touch src/full/Agda/VersionCommit.hs
===================== Installing using Cabal with test suites ============
time cabal v1-install --disable-documentation --builddir=./dist-2.6.3-ghc-9.0.1 --enable-tests -foptimise-heavily -j1 --disable-library-profiling -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3
Resolving dependencies...
In order, the following will be installed:
Agda-2.6.3 *test (reinstall)(changes: QuickCheck-2.14.2 added,
filemanip-0.3.6.3 added, process-extras-0.7.4 added, tasty-1.4.1 added,
tasty-hunit-0.10.0.3 added, tasty-quickcheck-0.10.1.2 added,
tasty-silver-3.2.2 added, temporary-1.3 added, unix-compat-0.5.3 added)
Warning: Note that reinstalls are always dangerous. Continuing anyway...
[1 of 1] Compiling Main ( dist-2.6.3-ghc-9.0.1/setup/setup.hs, dist-2.6.3-ghc-9.0.1/setup/Main.o )
Linking ././dist-2.6.3-ghc-9.0.1/setup/setup ...
Configuring Agda-2.6.3...
Preprocessing executable 'agda-mode' for Agda-2.6.3..
Building executable 'agda-mode' for Agda-2.6.3..
. . . . .
[402 of 403] Compiling Agda.Compiler.Builtin ( src/full/Agda/Compiler/Builtin.hs, dist-2.6.3-ghc-9.0.1/build/Agda/Compiler/Builtin.o, dist-2.6.3-ghc-9.0.1/build/Agda/Compiler/Builtin.dyn_o )
[403 of 403] Compiling Agda.Main ( src/full/Agda/Main.hs, dist-2.6.3-ghc-9.0.1/build/Agda/Main.o, dist-2.6.3-ghc-9.0.1/build/Agda/Main.dyn_o )
ld: warning: directory not found for option '-L/usr/local/opt/icu4c/lib'
ld: warning: dylib (/opt/local/lib/libicuuc.dylib) was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib (/opt/local/lib/libicui18n.dylib) was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib (/opt/local/lib/libicudata.dylib) was built for newer macOS version (11.2) than being linked (11.0)
Preprocessing executable 'agda' for Agda-2.6.3..
Building executable 'agda' for Agda-2.6.3..
[1 of 1] Compiling Main ( src/main/Main.hs, dist-2.6.3-ghc-9.0.1/build/agda/agda-tmp/Main.dyn_o )
Linking ./dist-2.6.3-ghc-9.0.1/build/agda/agda ...
. . . . .
Preprocessing test suite 'agda-tests' for Agda-2.6.3..
Building test suite 'agda-tests' for Agda-2.6.3..
[ 1 of 63] Compiling Internal.Helpers ( test/Internal/Helpers.hs, dist-2.6.3-ghc-9.0.1/build/agda-tests/agda-tests-tmp/Internal/Helpers.dyn_o )
. . . . .
[63 of 63] Compiling Main ( test/Main.hs, dist-2.6.3-ghc-9.0.1/build/agda-tests/agda-tests-tmp/Main.dyn_o )
Linking ./dist-2.6.3-ghc-9.0.1/build/agda-tests/agda-tests ...
ld: warning: directory not found for option '-L/usr/local/opt/icu4c/lib'
ld: warning: dylib (/opt/local/lib/libicuuc.dylib) was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib (/opt/local/lib/libicui18n.dylib) was built for newer macOS version (11.2) than being linked (11.0)
ld: warning: dylib (/opt/local/lib/libicudata.dylib) was built for newer macOS version (11.2) than being linked (11.0)
Installing executable agda-mode in /Users/ur20980/.cabal/bin
Installing library in /Users/ur20980/.cabal/lib/x86_64-osx-ghc-9.0.1/Agda-2.6.3-Jmf8C4FL9f76mgnQviRUta
Installing executable agda in /Users/ur20980/.cabal/bin
Generating Agda library interface files...
... /Users/ur20980/src/agda/src/data/lib/prim/Agda/Builtin/Bool.agda
. . . . .
... /Users/ur20980/src/agda/src/data/lib/prim/Agda/Primitive/Cubical.agda
Installing executable agda-mode in /Users/ur20980/.cabal/bin
Installing library in /Users/ur20980/.cabal/lib/x86_64-osx-ghc-9.0.1/Agda-2.6.3-Jmf8C4FL9f76mgnQviRUta
Installing executable agda in /Users/ur20980/.cabal/bin
Completed Agda-2.6.3
real 26m40.947s
user 23m38.891s
sys 0m39.214s
You can see two v1-install
commands, one with --only-dependencies
, the other one - without; but both include --enable-tests
.
Again, here are just the "relevant" lines:
cabal v1-install --disable-documentation --builddir=./dist-2.6.3-ghc-9.0.1 --enable-tests -foptimise-heavily --only-dependencies -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS"
cabal v1-install --disable-documentation --builddir=./dist-2.6.3-ghc-9.0.1 --enable-tests -foptimise-heavily -j1 --disable-library-profiling -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3
I think this ticket is just confusing, and confused. Its not addressing a real world case.
The Agda ticket in question is not crisp (to say the least), as it tackles (mixes) several issues. Some of them have been long addressed, like, using cabal-install 3.0.0.0+
- current build succeeds with cabal-3.6.0.0
. But building from the source using v2-
commands has not been resolved. Part of the problem seems to be the --enable-tests
argument.
@mouse07410, I haven't looked at the Agda issue, so I can't comment about that, but just a remark that v1-install corresponds to v2-build. And v2-install is a completely different beast and in flux.
Edit: and we asked why you use --only-dependencies
with v2-install, while you asked (or the question was understood so) on the Agda issue why they use it with v1-install, which is akin to v2-build.
tbh I think the main culprit is v2-install recompile local packages after a v2-build, so fixing that adding a copy-bins to v2-build would make less desirable the original request @mouse07410 am I wrong about that?
tbh I think the main culprit is v2-install recompile local packages after a v2-build, so fixing that adding a copy-bins to v2-build would make less desirable the original request @mouse07410 am I wrong about that?
Yes, but... In my case, all the builds are dynamic. It implies that copying just the executables won't be enough - all the created binaries (executables and libraries) have to be copied with copy-bins
.
Along the same line, what would be the difference between v2-install
and v2-build --copy-bins
?
just a remark that
v1-install
corresponds tov2-build
. Andv2-install
is a completely different beast and in flux
Sorry, I do not understand. To me:
v?-build
is when you create the entire binary set (executable and the relevant libraries) somewhere locally, in your working directory;v?-install
is when all the binaries created above, are installed into a "global" (for this user) location, like ~/.cabal/bin
and ~/.cabal/store
, allowing all other packages/apps to use them;v1-install
installs the package into that global location without regard for the flags and version, thus possibly hurting other apps that want the installed package to be of a different version and/or compiled with different flags; v2-install
"rectifies" this by maintaining different copies based on flags and version.What am I missing?
* `v?-build` is when you create the entire binary set (executable and the relevant libraries) somewhere locally, in _your_ working directory;
Incorrect. v1-build
does not exist, v2-build
installs executables locally, but makes libraries available globally in store/.
* `v?-install` is when all the binaries created above, are installed into a "global" (for this user) location, like `~/.cabal/bin` and `~/.cabal/store`, allowing _all_ other packages/apps to use them;
Incorrect. v1-install
does roughly what you say and more, because it overwrites any deletes all different versions of the same libraries., The purpose of v2-install
(without --lib
flag) is only to make the executable globally available, nothing else.
* `v1-install` installs the package into that global location _without_ regard for the flags and version, thus possibly hurting other apps that want the installed package to be of a different version and/or compiled with different flags; `v2-install` "rectifies" this by maintaining different copies based on flags and version.
Almost correct, except that v2-build
is the one tasked with installing libraries, v2-install
has a different purpose.
Along the same line, what would be the difference between v2-install and v2-build --copy-bins?
Afaik internally v2-install makes and sdist (generating the source tarballs) of all packages, unpack them in $TMP and build them there, installing executables in the global location after that. As it does a build in a new location it has to recompile everything. I think I ve read somewhere the motivation of that but it seems it makes install command be more consistent and regular between the several targets it can handle: local and remote tarballs directly. The unique advantage I can think of is you build the effective sources included by sdist (which can be different from the located ones in the original dir)
So maybe it worths to keep it even if you add copy-bins, like stack.
v1-build
does not exist
Apparently, cabal-install 3.6.0.0
disagrees with you:
$ cabal --version
cabal-install version 3.6.0.0
compiled using version 3.6.1.0 of the Cabal library
$ cabal help | grep build
build Compile targets within the project.
report Upload build reports to a remote server.
v2-build Compile targets within the project.
v1-build Compile all/specific components.
v1-configure Prepare to build the package.
v1-clean Clean up after a build.
The purpose of
v2-install
(without--lib
flag) is only to make the executable globally available, nothing else.
Except, of course, that it must make sure the dependencies are installed as well, and if - as in my case - the build is dynamic, then all of the shared libraries from this package will also be installed.
But again, the goal is to install dependencies first, and then install the main package without having to recompile those dependencies again. The main package has to undergo tests, so it's built with --enable-tests
. Current Cabal does not seem to allow this workflow, which is a problem.
v1-build
does not existApparently,
cabal-install 3.6.0.0
disagrees with you:
I stand corrected.
Funny, I even have it in my old makefiles, but haven't heard users using it for ages (or never? I think one usually needs to use v1-copy after that and people don't need that granularity with v1-?).
The purpose of
v2-install
(without--lib
flag) is only to make the executable globally available, nothing else.Except, of course, that it must make sure the dependencies are installed as well, and if - as in my case - the build is dynamic, then all of the shared libraries from this package will also be installed.
Haskell executables are linked statically wrt Haskell libraries, so normally just the sole executable suffices. But you are right, in the dynamic case that's not enough and, I don't know how that works exactly, but cabal v2-install
needs to "make the executable globally available and runnable, nothing else".
But again, the goal is to install dependencies first, and then install the main package without having to recompile those dependencies again.
Do we mean, the goal of 'cabal v2-install --only-dependencies`? If so, we disagree, see above.
The main package has to undergo tests, so it's built with
--enable-tests
.
Makes sense.
Current Cabal does not seem to allow this workflow, which is a problem.
Are you sure? Did you try with cabal v2-build
? Or even cabal v2-build --only-dependencies
, given that it's useful in some circumstances?
Haskell executables are linked statically wrt Haskell libraries, so normally just the sole executable suffices.
Well, yes - but the fact that shared: True
and executable-dynamic: True
are part of the standard Cabal approach, and have been for a long time - I daresay my use case (aka, building dynamic) is "normal".
But you are right, in the dynamic case that's not enough and, I don't know how that works exactly, but
cabal v2-install
needs to "make the executable globally available and runnable, nothing else".
Yes - but, as I said above, "globally available and runnable" requires that all of the dependencies of the current targets and all of the dynamic libraries of the current targets are also installed globally. So, I'm not sure what you meant by "nothing else", as AFAIK, there is nothing else.
But again, the goal is to install dependencies first, and then install the main package without having to recompile those dependencies again.
Do we mean, the goal of 'cabal v2-install --only-dependencies`? If so, we disagree, see above.
I mean - the goal of the workflow that the developers of Agda compiler set up, and which I'm not competent enough to modify.
On the other hand, that workflow makes a lot of sense - if my package pulls a lot of dependencies, I do not want to have to recompile them all every time I tweak something in my package, especially if that tweak is only "minor" flags.
Current Cabal does not seem to allow this workflow, which is a problem.
Are you sure? Did you try with cabal
v2-build
? Or evencabal v2-build --only-dependencies
, given that it's useful in some circumstances?
I'm reasonably sure I did try that, and it recompiled everything when it came to building the "main" package (Agda itself). v2-build
placed everything in the local build dir, aka dist-newstyle//build/x86_64-osx/ghc-9.0.1/XXXXXX
. In other words, not globally. Though offhand, I can't validate what it's doing to dependencies - need to rush somewhere.
In addition to the listed problems, there were issues with @rpath
, which I'm saving for a better time and a separate discussion.
Haskell executables are linked statically wrt Haskell libraries, so normally just the sole executable suffices.
Well, yes - but the fact that
shared: True
andexecutable-dynamic: True
are part of the standard Cabal approach, and have been for a long time - I daresay my use case (aka, building dynamic) is "normal".
Seriously, are we disputing whether my use of "normal" was justified?
Seriously, are we disputing whether my use of "normal" was justified?
No, not really. ;-)
The question is whether the current Cabal supports the following workflow, and if it does - how to do it:
--enable-tests
flag.--enable-tests
flag and without having to recompile the dependencies.~/.cabal/bin
and ~/.cabal/store
.Cabal supports all but the last step, where you can copy binaries, but it will not copy shared libraries.
The alternate workflow, where one builds twice, once a build tests, and once with the "install" command, to install, will still use all the existing installed dependencies in the global store. This seems a minor inconvenience, at most.
I will repeat myself very clearly: despite the older (v1) agda instructions, installing with tests does not seem a workflow we should concern ourselves with supporting.
The alternate workflow, where one builds twice, once a build tests, and once with the "install" command, to install, will still use all the existing installed dependencies in the global store. This seems a minor inconvenience, at most.
Will this alternate workflow rebuild the dependencies in the global store? If not - then there's no inconvenience, as far as I'm concerned. What should the exact commands be then for these two builds (assuming one wants to only use v2-
commands)? Something like
cabal v2-build --only dependencies xxxxxxxxx
cabal v2-install --enable-tests xxxxxxxx
I really don't want Cabal to recompile all the dependencies just because install
that follows build
got an extra --enable-tests
flag.
cabal v2-build --enable-tests
(assuming you want tests enabled). If you want tests to run you then need to run cabal v2-test
. (If you build with tests enabled, but never run v2-test, there's no point). Subsequent calls to cabal v2-install
will not recompile dependencies, since they are all in the shared global store. Also note that in the v2- workflow, --only-dependencies
is not necessary -- cabal will plan and build dependencies, as well as manage them in the global store as necessary.
So in fact, assuming you don't actually want to run tests, then a single call to cabal v2-install
suffices.
Thank you! Much clearer. So, in your opinion I don't even need --only-dependencies
in the v2-
context?
A silly question: at the last cabal v2-install
do I include --enable-tests
flag? I confess to being more than a little confused.
No. All the enable-tests flag does is configure the package to build the test suite. But the test suite is, by design, not installed. So it makes no sense to enable tests when installing.
Sorry, no good.
Here's the build command:
time cabal v2-build --enable-tests --disable-documentation -foptimise-heavily -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3 2>&1 | tee cabal-build-out.txt
Here's the install command:
time cabal v2-install --disable-documentation -foptimise-heavily -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3 2>&1 | tee cabal-install-out.txt
cabal v2-build ... <flags>
succeeds, and gets me the libraries and the executables I need (see the attached log below). So far so good.
You can see what remote packages it thinks it needs to build:
$ time cabal v2-build --enable-tests --disable-documentation -foptimise-heavily -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3 2>&1 | tee cabal-build-out.txt
Resolving dependencies...
Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
- hashable-1.3.4.1 (lib) (requires download & build)
- time-compat-1.9.6.1 (lib) (requires build)
- these-1.1.1.1 (lib) (requires build)
- async-2.2.4 (lib) (requires download & build)
- scientific-0.3.7.0 (lib) (requires build)
- unordered-containers-0.2.14.0 (lib) (requires build)
- hashtables-1.2.4.1 (lib) (requires build)
- data-fix-0.3.2 (lib) (requires build)
- uuid-types-1.0.5 (lib) (requires build)
- case-insensitive-1.2.1.0 (lib) (requires build)
- strict-0.4.0.1 (lib) (requires build)
- tasty-silver-3.2.3 (lib) (requires build)
- attoparsec-0.14.1 (lib) (requires build)
- aeson-1.5.6.0 (lib) (requires build)
- Agda-2.6.3 (lib:Agda, exe:agda, exe:agda-mode, test:agda-tests) (configuration changed)
Downloading hashable-1.3.4.1
. . . . .
cabal v2-install ... <same flags as above, except for --enable-tests
rebuilds everything, when the whole idea is to avoid this extra build.
And you can see that the very same remote packages are now being rebuilt:
$ time cabal v2-install --disable-documentation -foptimise-heavily -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3 2>&1 | tee cabal-install-out.txt
Wrote tarball sdist to
/Users/ur20980/src/agda/dist-newstyle/sdist/Agda-2.6.3.tar.gz
Resolving dependencies...
Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
- hashable-1.3.4.1 (lib) (requires build)
- time-compat-1.9.6.1 (lib) (requires build)
- these-1.1.1.1 (lib) (requires build)
- async-2.2.4 (lib) (requires build)
- scientific-0.3.7.0 (lib) (requires build)
- unordered-containers-0.2.14.0 (lib) (requires build)
- hashtables-1.2.4.1 (lib) (requires build)
- data-fix-0.3.2 (lib) (requires build)
- uuid-types-1.0.5 (lib) (requires build)
- case-insensitive-1.2.1.0 (lib) (requires build)
- strict-0.4.0.1 (lib) (requires build)
- attoparsec-0.14.1 (lib) (requires build)
- aeson-1.5.6.0 (lib) (requires build)
- Agda-2.6.3 (exe:agda, exe:agda-mode) (requires build)
Starting hashable-1.3.4.1 (lib)
. . . . .
The only difference in flags is absence of --enable-tests
for the install. What is wrong? Why is cabal-3.6.0.0 doing that? Can you please fix it, so it doesn't rebuild what's already built?
Here are the logs: cabal-build-out.txt
$ time cabal v2-install --disable-documentation -foptimise-heavily -fenable-cluster-counting --ghc-options="+RTS -M6G -RTS" --program-suffix=-2.6.3 2>&1 | tee cabal-install-out.txt
Wrote tarball sdist to
/Users/ur20980/src/agda/dist-newstyle/sdist/Agda-2.6.3.tar.gz
Resolving dependencies...
Build profile: -w ghc-9.0.1 -O1
In order, the following will be built (use -v for more details):
- hashable-1.3.4.1 (lib) (requires build)
- time-compat-1.9.6.1 (lib) (requires build)
- these-1.1.1.1 (lib) (requires build)
- async-2.2.4 (lib) (requires build)
- scientific-0.3.7.0 (lib) (requires build)
- unordered-containers-0.2.14.0 (lib) (requires build)
- hashtables-1.2.4.1 (lib) (requires build)
- data-fix-0.3.2 (lib) (requires build)
- uuid-types-1.0.5 (lib) (requires build)
- case-insensitive-1.2.1.0 (lib) (requires build)
- strict-0.4.0.1 (lib) (requires build)
- attoparsec-0.14.1 (lib) (requires build)
- aeson-1.5.6.0 (lib) (requires build)
- Agda-2.6.3 (exe:agda, exe:agda-mode) (requires build)
Starting hashable-1.3.4.1 (lib)
Building hashable-1.3.4.1 (lib)
Installing hashable-1.3.4.1 (lib)
Completed hashable-1.3.4.1 (lib)
Starting these-1.1.1.1 (lib)
Starting scientific-0.3.7.0 (lib)
Starting unordered-containers-0.2.14.0 (lib)
Starting data-fix-0.3.2 (lib)
Starting async-2.2.4 (lib)
Starting uuid-types-1.0.5 (lib)
Starting case-insensitive-1.2.1.0 (lib)
Starting time-compat-1.9.6.1 (lib)
Starting hashtables-1.2.4.1 (lib)
Building data-fix-0.3.2 (lib)
Building unordered-containers-0.2.14.0 (lib)
Building scientific-0.3.7.0 (lib)
Building these-1.1.1.1 (lib)
Building time-compat-1.9.6.1 (lib)
Building async-2.2.4 (lib)
Building uuid-types-1.0.5 (lib)
Building case-insensitive-1.2.1.0 (lib)
Building hashtables-1.2.4.1 (lib)
Installing data-fix-0.3.2 (lib)
Installing async-2.2.4 (lib)
Installing case-insensitive-1.2.1.0 (lib)
Completed data-fix-0.3.2 (lib)
Installing uuid-types-1.0.5 (lib)
Completed async-2.2.4 (lib)
Installing these-1.1.1.1 (lib)
Completed case-insensitive-1.2.1.0 (lib)
Completed uuid-types-1.0.5 (lib)
Completed these-1.1.1.1 (lib)
Starting strict-0.4.0.1 (lib)
Installing scientific-0.3.7.0 (lib)
Installing time-compat-1.9.6.1 (lib)
Completed scientific-0.3.7.0 (lib)
Starting attoparsec-0.14.1 (lib)
Completed time-compat-1.9.6.1 (lib)
Building strict-0.4.0.1 (lib)
Building attoparsec-0.14.1 (lib)
Installing unordered-containers-0.2.14.0 (lib)
Installing hashtables-1.2.4.1 (lib)
Completed unordered-containers-0.2.14.0 (lib)
Completed hashtables-1.2.4.1 (lib)
Installing strict-0.4.0.1 (lib)
Completed strict-0.4.0.1 (lib)
Installing attoparsec-0.14.1 (lib)
Completed attoparsec-0.14.1 (lib)
Starting aeson-1.5.6.0 (lib)
Building aeson-1.5.6.0 (lib)
Installing aeson-1.5.6.0 (lib)
Completed aeson-1.5.6.0 (lib)
Starting Agda-2.6.3 (all, legacy fallback)
Building Agda-2.6.3 (all, legacy fallback)
. . . . . [build-for-install is still running]
@mouse07410 I am with you, each time I have to install hls I have to wait for two hours to see the end of the build To build and install it I have to plan it for the night, like my windows updates 😝
See https://github.com/haskell/cabal/issues/7297#issuecomment-939285238 for more info about the origin of build Vs install differences.
So from a user point of view the objective would be avoid hard to do rebuilds, no matter the underlying impl would be.
That said, don't you get a rebuild of local packages when you switch enable-tests between two cabal build
?
each time I have to install hls I have to wait for two hours to see the end of the build
Fur me as a user is a big inconvenience. For a big-project developer that uses CI it's a disaster, because the time CI allows the build+run to consume gets eaten up by this stupid rebuild. In the end, making it impossible to use CI. Not a good thing!
don't you get a rebuild of local packages when you switch enable-tests between two
cabal build
?
That's the whole point. I can understand why you don't want to install the tests you built. I do not understand why I cannot install the executables (and libraries) that were built with --enable-tests
.
I think that upon install --enable-tests
Cabal should install the main targets. I'm ok if it won't install the test-target.
Or am I missing something?
Please let's use #6919 to discuss whether install
should copy local executables and libraries built with build
or not.
cabal v2-install ... <same flags as above, except for --enable-tests rebuilds everything, when the whole idea is to avoid this extra build. And you can see that the very same remote packages are now being rebuilt
I cannot reproduce this. In the insall command, all remote packages are reused. Maybe there is a difference deep in the dependency trees of the two build plans you're getting that causes all that (justified in that case) recompilation. You could check that with cabal-plan.
I cannot reproduce this. In the insall command, all remote packages are reused. Maybe there is a difference deep in the dependency trees of the two build plans you're getting that causes all that (justified in that case) recompilation
I'm pretty sure that the complexity of the Agda project build contributes a lot to exposing this problem - but I say it's still a Cabal problem.
You could check that with
cabal-plan
Not sure I understand what you mean - but
$ cabal-plan list-bins
Agda:exe:agda /Users/ur20980/src/agda/dist-newstyle/build/x86_64-osx/ghc-9.0.1/Agda-2.6.3/build/agda/agda
Agda:exe:agda-mode /Users/ur20980/src/agda/dist-newstyle/build/x86_64-osx/ghc-9.0.1/Agda-2.6.3/build/agda-mode/agda-mode
happy:exe:happy /Users/ur20980/.cabal/store/ghc-9.0.1/hppy-1.20.0-37a15369/bin/happy
alex:exe:alex /Users/ur20980/.cabal/store/ghc-9.0.1/lx-3.2.6-c78ab9a4/bin/alex
$
happy
and alex
, apparently, are byproducts of this build, because
$ ll /Users/ur20980/.cabal/bin/alex
lrwxr-xr-x 1 ur20980 staff 45 Sep 28 13:13 /Users/ur20980/.cabal/bin/alex@ -> ../store/ghc-9.0.1/lx-3.2.6-64150b8d/bin/alex
$ ll /Users/ur20980/.cabal/store/ghc-9.0.1/lx-3.2.6-c78ab9a4/bin/alex
-rwxr-xr-x 1 ur20980 staff 883104 Sep 28 23:22 /Users/ur20980/.cabal/store/ghc-9.0.1/lx-3.2.6-c78ab9a4/bin/alex*
$
And, of course
$ cabal list-bin Agda:exe:agda
cabal: No or multiple targets given
$
One more concern:
$ cabal-plan list-bin Agda:exe:agda
/Users/ur20980/src/agda/dist-newstyle/build/x86_64-osx/ghc-9.0.1/Agda-2.6.3/build/agda/agda
$ otool -L `cabal-plan list-bin Agda:exe:agda`
/Users/ur20980/src/agda/dist-newstyle/build/x86_64-osx/ghc-9.0.1/Agda-2.6.3/build/agda/agda:
@rpath/libHSAgda-2.6.3-inplace-ghc9.0.1.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libHSbase-4.15.0.0-ghc9.0.1.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libHSghc-prim-0.7.0-ghc9.0.1.dylib (compatibility version 0.0.0, current version 0.0.0)
@rpath/libHSrts_thr-ghc9.0.1.dylib (compatibility version 0.0.0, current version 0.0.0)
/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1292.100.5)
$
The executable depends on the project-built shared library @rpath/libHSAgda-2.6.3-inplace-ghc9.0.1.dylib
, which is not listed/noticed/picked by cabal-plan
. It implies that it can't be copied manually via something like
$ cp `cabal-plan list-bin Agda:exe:agda` $DEST_DIR
Just tried this (though with ghc 8.10.4) and can't reproduce the deps being rebuilt when trying an install after a build with --enable-tests.
Note that it appears only a small subset of libs are being rebuilt, so its not a general issue, but something pertaining to those libs in particular.
This may have to do with the dynamic linking opts in your ~/cabal.config
. . . can't reproduce the deps being rebuilt when trying an install after a build with
--enable-tests
A sure reproducer for me in Agda repo itself - a bit on the big side, but should suffice as a proof...
All of my toy projects are trivially small, so not representative...
Note that it appears only a small subset of libs are being rebuilt, so its not a general issue, but something pertaining to those libs in particular.
Or, rather, a general issue impacting specific conditions. I doubt it's specific to individual libs/packages - probably, specific to what flags their project files define...?
This may have to do with the dynamic linking opts in your
~/cabal.config
No doubt, dynamic linking exacerbates the problem. Though I doubt it causes it.
I literally tried this with the agda repo. I couldn't reproduce.
Same and same.
You could check that with cabal-plan
Not sure I understand what you mean
if you run cabal-plan info
you get detailed information about the build plan. There's even a cabal-plan diff
command that could help you pinpoint the cause
You could check that with cabal-plan
Not sure I understand what you mean
if you run
cabal-plan info
you get detailed information about the build plan. There's even acabal-plan diff
command that could help you pinpoint the cause
My apologies for being dense. Say, I have a pristine/clean clone of the repo, ready for cabal build
and such. What would be the sequence of commands, including cabal-plan ...
that you suggest I run?
I literally tried this with the agda repo. I couldn't reproduce.
@gbaz and @fgaz would it be possible for you to re-try the same with (a) dynamic build (editing ~/.cabal/config
), and/or (b) the current GHC-9.0.1?
So not sure if my test is correct but i tried a raw cabal build -w ghc-8.10.7
and then cabal install -w ghc-8.10.7
and many dependencies has been rebuilt:
This is the cabal plan after cabal build
:
cabal plan after cabal install
And finally the output of cabal-plan in the temp dir used for build by cabal install
EDIT:
cabal install
is doneEdit: apparently, the below doesn't make sense apart of pointing out the connection to #7297, see two comments down.
.
@jneira: I do not claim to understand this, but @phadej says it's expected and hard to avoid that unit IDs differ between build and install: https://github.com/haskell/cabal/issues/7297#issuecomment-939330830
Given that, I guess it's unavoidable that rebuilds occur. Perhaps we need to dig deeper into the IDs and/or #7297.
Sanity check: the unit IDs in the plans you attached do differ, right?
Describe the bug Trying to manually (because included
Makefile
forcesv1-
commands) build Agda from source, I'm getting the following error:To Reproduce Steps to reproduce the behavior:
Expected behavior Successful build. I am getting it when I omit
--enable-tests
, but in that caseagda-tests
executable is not built (so, I can't run tests to validate the created executables).Note, that
cabal v2-build --enable-tests
seems to work fine (but it does not install the executables it builds).System information
cabal-3.6.0.0
,ghc-9.0.1