Add config flag to control generation of .ghc.environment files

[hvr]

Some people are excited about .ghc.environment files, while others are mildly annoyed. I think there's probable cause to make this configurable on cabal's end.

To this end, I suggest (modulo bikeshed) to implement a flag, e.g. --pkg-environment-scope=LEVEL, where LEVEL can be one of

1. all (default & current behaviour, generate ghc env files with all transitive dependencies of project's (non-qualified) goals)
2. build-target (e.g. :pkg:Cabal or lib:Cabal or Cabal:test:parser-tests): generate ghc env files containing only the stated goal's /direct/ dependencies)
3. - (or none or off or disable?): disable generation of any ghc env files

This flag would also be persistable via cabal.project.(local).

Note: Item 2. is not the short-term goal of this feature request! I've included the 2. item mostly to motivate why it makes sense to design this to be a flag that's more than merely a boolean --{disable,enable}-package-environment-files flag.

Btw, a dual feature-request (allowing to opt-out from interpreting .ghc.environment files from GHC's side) has been filed at GHC #13753 -- however, that one will be too late for GHC 8.0.2 users.

Current status:

• Starting with GHC 8.4.3, ghc is more verbose when picking up pkg envs (see http://git.haskell.org/ghc.git/commit/00049e2dce93b1e468c3fde3287371eb988aafdc)
• Starting with GHC 8.4.4, ghc supports the special - environment which inhibits the default lookup logic (see http://git.haskell.org/ghc.git/commit/8f3c149d94814e4f278b08c562f06fc257eb3c43)

/cc @RyanGlScott

[ElvishJerricco]

@23Skidoo Are logical extensions of supposedly legacy features considered valuable?

[23Skidoo]

Merely saying that it's nothing new if you look at it this way.

[ElvishJerricco]

I guess it's not new for people who are used to installing stuff in the user package DB. But I think avoiding the user package DB is common practice, and these environment files are imposing the same behavior on people who avoid it.

[mikeplus64]

For what it's worth, I found this issue after running into mysterious nix-based build failures that turned out to be due to the .ghc.environment... files being auto-generated when I went into nix-shell --command 'cabal new-repl'.

For now good-old cabal repl and/or rming the file for nix is fine but it would be nice if this were configurable.

[hvr]

Sure, you can ask everyone to script their environment to have ghc and ghci be some magic scripts which do the DWIM behaviour -- but what does this achieve other than complicating everyone's life? you'd still end up with the commands ghc and ghci doing something special which tooling which naively assumes a different behaviour might likely choke on just as well.

And don't forget, this is about making the default experience on the shell convenient which is traditionally not deterministic; everything you do on the shell is traditionally stateful and obviously $CWD & $PATH sensitive. So to me this is a totally expected behaviour and it is a pragmatic design choice in the interest of DWIM and convenience.

If this breaks tooling it's merely because said tooling didn't take precautions to invoke ghc and ghci in such a way that setups the environment the way it needs it to be; if a tool naively calls ghc and ghci w/o specifying anyting else to shield against configuration files leaking in, in the past it already would have picked up a lot of contextual state which may or may not be in conflict with the tool's assumptions about the environment .

Btw, we just taught cabal new-install how to manipulate the stateful default package environment of GHC (which was a showblocker for making new-build default, as this corresponds to the old user pkg-db in the nix-style paradigm which we were lacking) and this also allows to generate custom $CWD-sensitive package environments. GHC supports these things, and cabal does too; in any case, such files can always be left behind, so it's the tooling's responsibility to become aware of these features and shield/protect itself against if it requires a more pristine ghc environment to operate in.

To summarise, don't require other tooling to create some "safe space" to protect your tool, instead make your tool more resilient/robust against undesired influences by being explicit about what it requires/expects about the environment.

[ElvishJerricco]

@hvr I feel like we're talking past each other here. Reading that response, it feels like I'm reading the same thing for the umpteenth time; i.e. it does not feel like you have actually addressed any of the arguments made against that position.

[mpickering]

This issue really needs to be resolved. I haven't read every comment on this thread but the environment files have broken far more than they have helped me.

I don't see the issue with a cabal command to enter an environment which has the package database this is how the whole python and nix ecosystem works. It is what people expect.

From my perspective is that once you have this automatic magic configuration there is NO option for users to get back to explicit behaviour as many desire. However, given the explicit behaviour, users who want a more automatic experience can use tooling such as direnv in order to automatically enter the correct shells.

[lspitzner]

@hvr

To summarise, don't require other tooling to create some "safe space" to protect your tool, instead make your tool more resilient/robust against undesired influences by being explicit about what it requires/expects about the environment.

I will point out that

1) The addition of package environment files as input to the compiler behaviour in ghc-8.0 was not mentioned in the release notes[1], nor in the migration guide[2]. 2) The fact that package environment files have an influence on the semantics of the GHC API is not documented in the API (and similarly nothing was mentioned in the release notes nor the migration guide). 3) The fact that package environment files affect the behaviour of GHC is to this date not mentioned in the man-page, nor in the --help output of ghc. It is "documented" in the users guide, if you happen to stumble upon the corresponding chapter[3]. Of course the users guide does not address the GHC API. 4) The fact that environment files are produced, as a side-effect, by cabal is not mentioned in the cabal commandline help, nor its man-page, nor its user's guide, nor was the change announced in the changelog [4].

It is not my intention to call out the relevant maintainers for this; it is an understandable oversight.

However, hvr, I strongly suggest you ensure that the the the tools and APIs under your control are explicit about their semantics in their documentation, before making suggestions regarding resilience and robustness to others.

if a tool naively calls ghc and ghci w/o specifying anyting else to shield against configuration files leaking in

Then, hvr, I expect an explanation as to how downstream tools are supposed to "be explicit" in a situation where one tool starts writing to a file and another starts changing its semantics based on that file. For all I can know, the next cabal version might start writing some new stuff into a ".we.love.implicit.inputs" file and the next GHC (API) version might start breaking randomly when it finds that file. There is no way to be proactively resilient against this. What does "being explicit" even translate to, here? Do you mean "rm -rf -- ./.*" to clear "environment files" in CWD? I really am at a loss what you could possibly mean.

And if we forget about the "proactively"/future-compatibility aspect of this, I still simply don't know what I am supposed to do in the current situation. How does one setup the environment appropriately? Should I delete the ".ghc.environment" file? Should I overwrite it in some fashion? Where is this documented? And how do you expect users to know this while it is not documented? And how do you expect users to keep track of which tools change which environment state in what way?

Every user of any interface naturally is "naive", because they don't know the thing as well as the authors of said interface. Indeed, that users (and by extension, the tools written by those users) are naive is a perfect reason for why not breaking expectations and implicit contracts is so important. So thanks for bringing up the phrase.

To summarize: Stop ignoring the users, don't make vague hints at potential solutions, and make sure that you are in a position to tell others what to do.

[1] https://downloads.haskell.org/~ghc/8.0.2/docs/html/users_guide/8.0.1-notes.html [2] https://ghc.haskell.org/trac/ghc/wiki/Migration/8.0 [3] https://downloads.haskell.org/~ghc/latest/docs/html/users_guide/packages.html#package-environments [4] https://hackage.haskell.org/package/cabal-install-2.2.0.0/changelog

[mikeplus64]

It is IMO much more unexpected that cabal-install commands alter the behaviour of ostensibly outside programs -- namely GHC and any tooling using it -- by default. @hvr It is more a problem of configuration files unexpectedly leaking out, than of them leaking in.

[23Skidoo]

Just a data point: I found implicit environments convenient in practice, because they made it easier to add out of the box ghcid support to a project I work on. However, they also broke a helper script in the same project.

[lspitzner]

@23Skidoo Were you able to fix said helper script? What did the solution look like? Is there any related information that might help others running into similar problems?

[23Skidoo]

I just added rm -f .ghc.environment.* to the beginning of the script. When we'll move to GHC 8.6, I'll change it to use -package-env -, as described in https://ghc.haskell.org/trac/ghc/ticket/13753. Alternatively, the new shebang feature in cabal 2.4 could also be used to solve this.

[lspitzner]

@23Skidoo

rm -f .ghc.environment.* is unfortunately insufficient in general, as there may be a package environment file in a parent directory (which easily may have been created by cabal-install).

Also,

When we'll move to GHC 8.6 ..

From the linked ticket:

milestone: 8.8.1 [bgamari:] These will not be addressed in GHC 8.6.

[23Skidoo]

It refers to the second part of the ticket. That patch is in 8.6, as you can see here: https://github.com/ghc/ghc/commit/8f3c149d94814e4f278b08c562f06fc257eb3c43.

[fgaz]

Another data point: I use env files a lot and I think they are really useful in a number of situations (I'll probably expand this in a blog post or two), and should definitely get publicized more, since few users know about them.

But, this is not enough to enable them by default IMO. They do cause problems sometimes, and the frustration when discovering the cause slightly outweighs the advantage of knowing they exist.

I think we all agree there should be a flag to enable/disable their generation though.

[lspitzner]

@23Skidoo I see. Am I correct in that the "master" version of the ghc users-guide is the most up-to-date one? Because it does not appear to reflect this addition. If so, might be worth to ping the relevant maintainer.

[23Skidoo]

Yep, looks like that patch didn't update the manual.

[lspitzner]

Global variables, side effects and dynamic typing are all really useful in a number of situations.

[fgaz]

I kinda like @alanz's idea too. Easy to implement and it'd add another colorful section to my bash prompt \o/

[23Skidoo]

@lspitzner

rm -f .ghc.environment.* is unfortunately insufficient in general, as there may be a package environment file in a parent directory (which easily may have been created by cabal-install).

Another thing you can do to ignore the implicit .ghc.environment. file is to use -hide-all-packages -package foo -package bar -package .... This is documented here. In fact, I've just converted my script to do this.

[lspitzner]

@23Skidoo I see you found the documentation which I already had linked above. It is sad that I need to point out how to work out around bugs in cabal to the very maintainer of cabal.

I have opened https://ghc.haskell.org/trac/ghc/ticket/15513 and https://ghc.haskell.org/trac/ghc/ticket/15541 to track some of the documentation deficiencies.

If anyone wants to open tickets about corresponding documentation of cabal-install, feel free.

[choener]

Unfortunately, the automatic generation of the .ghc.environment file is the cause of problems for me, as well. It is really unfortunate that they are being created automatically. I switch, with nix, between different versions of ghc (and everything below) and this not only breaks the repl, but full builds. For me, it would be much better, if they are only created if asked explicitly, the current behaviour is way too "surprising" for me.

[svenpanne]

I just wanted to say that I find the new implicit magic use of environment files by GHC a very bad idea, too. A few people already mentioned Python, so let's take a look how things work there:

The new officially sanctioned way to use Python + virtual environments is pipenv. It generates the actual, transitive list of packages to use (Pipfile.lock) from the description of packages your project needs (Pipfile). If you substitute "environment file" for "Pipfile.lock"and ".cabal file" for "Pipfile", you see the similarities with our problem at hand.

The crucial thing is: Python is blissfully unaware of any virtual environment unless you tell it explicitly that you want to use it. This is IMHO the only sensible way to do this, otherwise hell would break loose, given e.g. the many tools on a normal Ubuntu installation written in Python which would be broken in basically any virtual environment (unless you are extremely lucky). The same breakage happens already when you use runghc / ghci / tools using the GHC API.

If you want to use your Python virtual environment, you can do:

• Run pipenv shell, which will search upwards in the directory hierarchy for a virtual environment and will spawn a new shell in which Python will use that virtual env.
• Run pipenv run foobar, which is basically the same, but just for a single command foobar.
• Use direnv plus a suitable simple .envrc to switch into/out of the virtual env when you enter/leave the corresponding directory.
• etc. etc.

The main point is: If you want to pick up an virtual environment, you have to be explicit, and you have a choice of dead easy, well-documented, and widely known alternatives to do so, depending on your workflow. This works extremely well in practice, including tooling in editors etc. I totally fail to see why GHC/cabal stubbornly ignore the painful lessons learned in the Python world and repeat the same mistakes... 😞

[gbaz]

@choener It isn't clear to me how switching between ghc versions with nix could interact poorly with env files? In particular, as per https://downloads.haskell.org/~ghc/master/users-guide/packages.html#package-environments the searchpath for env files is indexed by ghc version, so different versions of ghc shouldn't end up sharing the same env files?

[choener]

@gbaz , sorry for the confusion; that is cabal not ghc. I've been playing with different versions of things lately. Unfortunately for me that increases the likelihood that I'm hit by this, since I have many local derivations using nixos packageSourceOverrides.

[gbaz]

Ok, so how does switching between different cabal versions cause a problem with this then? :-)

[choener]

The problem is, that once the environment file has been created by a sufficiently new cabal, it interferes with building as setup by packageSourceOverrides. @svenpanne has given a beautiful explanation, why having these files is not good, and I see the fallout from their creation in local builds with local dependencies.

[gbaz]

One other thought, occasioned by this blog post's (https://hexagoxel.de/postsforpublish/posts/2018-08-25-ghc-pkg-env-misfeature.html) point that:

"Invalid and even just outdated package environment files make ghc abort."

Perhaps we could change ghc such that if there is an error occasioned by an invalid command in the course of processing and env file then ghc could specifically point to the env file as inducing the problem? (Or did we patch ghc to do this already and I forgot?).

A big problem with the behavior seems to be just that when it goes wrong people get confused, so I suspect that better error messages will go a long way.

[gbaz]

@choener ok, so there isn't a distinct problem switching between ghcs or cabals at all, just the known complaint about interactions when using a sufficiently new cabal such that env files are generated at all. I'm not trying to be dismissive, I just want to be sure that people understand what the issues are or are not.

I really do think that a combination of a flag which nix haskell integration can set to turn off env generation combined with better error reporting as I suggested above should make this workflow much improved.

[choener]

@gbaz Yes! I wanted to give another voice towards making this opt-in. In particular since it seems, that currently you can not even opt-out (as far as I understand the ProjectOrchestration.hs). This in turn makes running cabal-install 2.2 rather frustrating, since I regularly break large builds just by doing cabal new-repl in a dependent package. I assume that this is partly due to the way how packageSourceOverrides works in nixos, and partially due to the way how I develop -- since I "never" use ghci, but only cabal new-repl and friends.

[mpickering]

I agree exactly with @svenpanne's clear comment https://github.com/haskell/cabal/issues/4542#issuecomment-415976537

Especially:

I totally fail to see why GHC/cabal stubbornly ignore the painful lessons learned in the Python world and repeat the same mistakes...

Any discussion of how nix works is not really relevant, it is just an example of the files breaking things. Of course it can be patched to ignore these files, and of course the GHC API can be changed to fix @lspitzner's problems but they do not fix the root cause of the issue.

[lspitzner]

@gbaz There is https://phabricator.haskell.org/D4689, included in 8.6.

From my perspective, this is no solution, on the contrary: It is another effect added to an operation that should be pure. Might be acceptable for UI, but for API it is really sad that you call some function and it spams to the host process's stderr.

[athas]

@gbaz There is https://phabricator.haskell.org/D4689, included in 8.6.

Does that solution make the breakage obvious (with default options) when using plain ghc? That was the situation in which I was bitten by this issue some days ago.

I think there are three factors that contribute to making this a misfeature: some tools create the .ghc.environment file automatically and (I think) silently, other tools automatically pick it up, and the file is hidden(!). Removing any of these factors would help the situation, but removing all of them would be best, I think. cabal ghc or cabal ghci is not so much more typing, and is much clearer to me.

[michaelpj]

Just chiming in as yet another person who wasted about half a day on mysterious breakage until I found this. I understand why it's convenient, but I agree with the others in this thread that subtly breaking users using "normal" compiler workflows is pretty bad.

(I tried to write this comment politely, but I feel like I should mention that this experience has actually made me very cross)

[ElvishJerricco]

@michaelpj your comment makes me think... Is it worth noting that, without trying to measure the ratio of positive / negative reactions to this feature, the magnitude of any negative reaction has consistently seemed much larger than the magnitude of any positive reaction? i.e. this feature seems either mildly convenient or massively frustrating, depending on who you ask. Regardless of whether you like the feature, are we willing to risk such frustration for the sake of minor convenience, just from a user-friendliness standpoint?

[gbaz]

The problem is that the comments here are people that have bitten by the rough edges. Discussion on an issue tracker is always dominated by people who have encountered issues prompting action -- there's no way to make any sort of judgment on that. We just need to fix the known and real outstanding issues, sooner rather than later, and then we can assess what problems (if any) remain.

[michaelpj]

We just need to fix the known and real outstanding issues, sooner rather than later, and then we can assess what problems (if any) remain.

Where I come from, when a feature is rolled out and causes breakage to a large number of users the response is a quick revert or gating it behind a flag until you can make a safe rollout plan. Somehow "just fixing the issues" always happens later rather than sooner.

(I also agree with the arguments in this thread that the feature seems strange to me. However, I don't mind that so long as I am not broken.)

I guess I'll come back to new-build in another six months.

[gbaz]

Indeed new-build is still experimental and only with the recent completion of the last gsoc is it nearing feature-complete. As with any open source project, patches and contributions are very welcome, but we're getting there with all this stuff, slowly but surely.

[ElvishJerricco]

@gbaz There's always going to be unknown information; we can't let that just completely stall the decision making process. This issue has been in a limbo state (read: de facto siding with current behavior) for quite some time now. We've seen several people come directly to this thread with the intention to display their support (edit: support of the feature, that is). I think that's at least something to go on.

[gbaz]

@ElvishJerricco I don't understand why you think this is in a limbo state. There is no endorsement of the current behavior. There is a change to two elements in GHC already -- improving warning messages to indicate use of env files and adding an opt-out flag to GHC directly (https://ghc.haskell.org/trac/ghc/ticket/13753). I think that ghc messages could be beefed up further in the case of errors -- but that really should be tracked in the GHC tracker.

Meanwhile, this issue tracks the ability to configure generation of the files or not -- that feature has not been implemented, and needs to be implemented. That's not limbo. That's "awaiting implementation."

The only dispute/question is if the better default behavior is to generate these files or not. Which is necessarily a question of choosing between use-cases. Since I think that the main issue with these files is in the presence of nix, and since nix already overrides many cabal settings, I think it is fine to have nix override the setting here as well.

(Yes there is also an issue with other tooling and the GHC api -- but there is a GHC tracker ticket for that too, as far as I know)

In any case, I think we've spent way more time discussing this improvement than making it, and I guess I'll try to cook up a patch in the next week or so, given that nobody who has chimed in on this thread with strong opinions has offered to make any patch themselves.

[ElvishJerricco]

Since I think that the main issue with these files is in the presence of nix, and since nix already overrides many cabal settings, I think it is fine to have nix override the setting here as well.

As I've stated before, Nix does not require any cabal settings to use cabal with a nix-shell-provided GHC-with-packages. But now, because of this feature, it does, and that means using CABAL_CONFIG to override stuff. But that's not a general solution, as it prevents users from getting settings that they actually do want from ~/.cabal/config. So they actually have to have a project-local setting, which means Nix can't make this the default behavior.

Furthermore, I don't think Nix is the main reason whatsoever. Sure, it's the main reason I discovered the issue, but the principle of it alone bothers me, and there have been reports from several other tools in several other use cases that this breaks. Sure, maybe there's an argument that the GHC API just needs a flag for it or something, but as long as this is the default behavior, people will frequently find themselves experiencing this kind of breakage. It is a perplexing situation when you encounter these problems for the first time, and it requires every tool author to have experienced that perplexity and anticipate it, or else their tool is likely to be broken.

That said, you're right about the limbo state. I apologize for disregarding the work that was done on GHC; that was not my intention. I was attempting to address specifically the decision about what the default behavior should be (edit: and whether that default would be changed in cabal or in GHC; about which I'm indifferent)

[svenpanne]

I think the main dispute is about whether or not GHC should pick up environment files by default. If they are generated by cabal or not is much less important. And the GHC ticket mentioned above is not the right way to fix the current state of affairs.

I'm very disappointed how this misfeature crept into GHC without any real discussion. The strange thing is: Tiny details are usually discussed on an epic scale, but something which heavily affects tons of people just went in. And yes, 99% of this discussion should be on the GHC issue tracker or on some mailing list...

[gbaz]

@svenpanne A discussion on glasgow-haskell-users sounds like a good idea. These files exist to be generated, and if cabal has this or that default setting, it hardly matters. If any tooling takes advantage of this ghc feature, similar issues will arise, and I think the main improvements to be made to make them work well will all tend to be on the GHC side.

@ElvishJerricco I'll try to sort out some of the questions I have about how the nix stuff works with configs in chat so as not to further clutter the thread.

[gbaz]

Hmm... actually, now that ghc's use of env files can be turned off by an env variable (by the ticket I linked) then nix can just set that and ignore messing with the config for this issue (though it still needs to mess with it in the new-build case for some other stuff, I think, at least for the time being).

[23Skidoo]

What if this config setting was made ternary instead of binary, enabling the feature by default only for versions of GHC that have a warning about env files and the -pkg-env - patch? I don't like the inconsistency, but on the other hand env files are not supported by all GHC versions anyway.

[lspitzner]

@23Skidoo As maintainer, would you be fine with merging a PR that reverts this feature (i.e. that removes the code that creates package environment files?)

This is in spirit of what @gbaz mentioned:

We just need to fix the known and real outstanding issues, sooner rather than later, and then we can assess what problems (if any) remain.

And indeed, if any people are still in favour of this feature, then by no means, they should be free to:

1) implement it in a way that does not break users, or 2) convince, in a public discussion, a majority of the respected members of this community that there is a sufficient advantage to breaking users, and that there is no non-breaking alternative to reaching this advantage.

But this can happen after fixing the real, outstanding issue, correct? Any discussion of how the config setting might look like, similarly, could be done afterwards.

[lspitzner]

I would like to point out that cabal officially supports ghc-8.0 through 8.4 (and a bit more, probably). This means that any "fixes" to this issue that might make it into ghc-8.6 are irrelevant.

[23Skidoo]

No, we shouldn't remove it, but we should definitely make it configurable.

[lspitzner]

@23Skidoo Thanks for the clear answer.

@gbaz There you have your "endorsement of the status quo"

[lspitzner]

Also, to make this clear: I specifically did not say "remove the feature". I said "revert it until the ones in favour implement it in an acceptable manner". Glancing over the difference is an unfair move.