search

haskell / cabal

Official upstream development repository for Cabal and cabal-install
https://haskell.org/cabal
Other
1.62k stars 697 forks source link

Garbled output due to per-component parallelism #4126

Open 23Skidoo opened 7 years ago

23Skidoo commented 7 years ago

When building the log package with cabal new-build and GHC 7.8.4, I saw this:

[...]
[11 of 11] Compiling Log.Backend.ElasticSearch ( src/Log/Backend/ElasticSearch.hs, /home/refold/code/scrive/log/dist-newstyle/build/x86_64-linux/ghc-7.8.4/log-0.5.5/build/Log/Backend/ElasticSearch.o )
Configuring component test:log-test-integration from log-0.5.5
Configuring component test:log-test from log-0.5.5
Warning: The pWarning: aThcekage  has a verspackage hiaosn a  versriaonge fnor  arange  for da depenepdenedncye onn an interncaly on an in
ltibrarye:r nlaolg -any && ==0.5.5
, logl -any && ==i0.5.5. Tbhis rvaerrys:ion range  has nloo
semangtic meani ng an-da ncyan be r emoved.&
& ==0.5.5, log -any && ==0.5.5. This version range has no
semantic meaning and can be removed.
Preprocessing test suite 'log-test' for log-0.5.5...
Preprocessing test suite 'log-test-integration' for log-0.5.5...
[1 of 2] Compiling Test.ElasticSearch ( test/Test/ElasticSearch.hs, /home/refold/code/scrive/log/dist-newstyle/build/x86_64-linux/ghc-7.8.4/log-0.5.5/c/log-test/build/log-test/log-test-tmp/Test/ElasticSearch.o )
[...]

Looks like a synchronisation fail somewhere.

23Skidoo commented 7 years ago

Also interesting that the warning seems to be bogus: log-test and log-test-integration test suites don't have any constraint on the log library.

ezyang commented 7 years ago

This is going to be difficult to "debug" without a reproduceable test case. Perhaps there is some architectural rethink of how we do output in cabal-install that can eliminate this class of bugs entirely? IIUC we do put some effort into avoiding interleaving; how can we make sure it's impossible to forget?

23Skidoo commented 7 years ago

It happens relatively often on my machine when building the log package, but maybe it's just me.

BardurArantsson commented 7 years ago

It would obviously be a lot of quite tedious work, but my dream scenario would be https://hackage.haskell.org/package/concurrent-output

I imagine the build process output looking a lot like what's shown in https://buckbuild.com/static/buck-build-15fps.gif (perhaps one line per dependency currently building rather than per-file)

BardurArantsson commented 7 years ago

OK, so I've had a little think about this and looked a little bit into the code... I'm not sure the concurrent-output is really feasible as it stands[1], but I think we could make a little headway if we were to consider using a custom monad as the base monad instead of IO. To avoid having to do a huge refactor in one go we could have that monad be an instance of MonadIO, but require our custom OutputM (let's say) monad for anything that calls "warn" (etc.). I'm basically thinking something MTL-like such that explicit lifting could be avoided as much as possible. We could start with a simple "lock-around-every-output" type thing to at least ensure that there are no mangled single lines, at least. How does this sound? Does it sound reasonable?

One thing that I haven't thought about, but which may end up causing trouble would be any "bracket"-type constructs (and exception handling in general) since unfortunately doesn't really work with MonadIO. (At least without monad-control or similar. Even then it can make type signatures pretty horrendous.). OTOH, if all bracket usage is really just wrapping some short-lived IO-action (and not complicated stuff with callbacks), we might be able to get away with just having a liftIO around all usages of bracket. I'll try to a little bit investigation and have a look at a random sample of bracket usages in the current code to see if this might be a problem.

EDIT: [1] I imagine that any refactor to use Shake as per @ezyang's recent issue would perhaps make it a lot more feasible, but I imagine such a refactor would be... highly non-trivial? :)

BardurArantsson commented 7 years ago

Oh, and just as a little aside using a custom monad instead of IO has an additional nice little benefit of being able to elide the Verbosity parameter (almost) everywhere. That'd be a pretty nice little perk.

BardurArantsson commented 7 years ago

I'm just assigning myself for now. I'll do a little more investigating...

ezyang commented 7 years ago

@BardurArantsson If you do IO with environment, bracket/etc functions can be implemented perfectly. I think it's a worthwhile refactor but it's worth thinking about what things you are going to put in the environment. Might need multiple monads (at least, GHC does!)

phadej commented 4 years ago

@BardurArantsson do you still plan to investigate this? I'm remilestoning this to 3.4, as I don't expect this happen for 3.2

BardurArantsson commented 4 years ago

No, I've given up on trying to contribute to Cabal at this point. (Details available upon request, but I'll avoid giving them here to avoid needless drama.)

I believe the issue is still relevant, but I have no idea if anything has happened recently to address the problem, so it may have been solved. (Unlikely, but it might...)