Closed puredanger closed 4 years ago
Homebrew will use brewed dependencies where possible. The fact that other Java versions were previously possible was an unfortunate side-effect of java not being build from source as a formula. The current situation is the intended usage within homebrew. If people want to use a different dependency the best course of action is to maintain it in a tap.
Just a couple of points to highlight one more time:
openjdk
installed from dependency is not symlinked anywhere and will be used only if JAVA_HOME
is not setJAVA_HOME
to any Java you wantI guess this is good reason to not use brew ¯_(ツ)_/¯
I think we will probably just pull this out into our own tap then.
90% of the clojure community are known to use either JDK 8 or JDK 11 (https://clojure.org/news/2020/02/20/state-of-clojure-2020)
And clojure.org recommends running clojure on JDK8 or JDK11 at this time. It seems very strange to me that brew are packaging clojure against the platforms recomendations.
It's also not a purely theoretical issue I have clojure apps which use the clojure command line tools, and don't run on anything other than JDK 8 due to either dependencies on XMLGregorianCalendar; which was removed in JDK9, or now illegal reflective access calls because of the JDK9 module system.
If you'd like to use another Java installation feel free to set JAVA_HOME to any Java you want
If you'd like to use another Java installation feel free to set JAVA_HOME to any Java you want
The homepage for homebrew says:
Homebrew installs the stuff you need that Apple (or your Linux system) didn’t.
This is clearly incorrect in this instance.
The Clojure team has expressed what the correct dependency is, and if this is not what is installed then homebrew is clearly not doing the main thing that it's advertised to do. Homebrew will be installing something that the significant majority of users do not need (and is not sanctioned), and then expecting them to fix it.
I find this disappointing and makes me suspicious of how other packages are handled.
This is clearly incorrect in this instance.
You need Java to use Clojure, we install that dependency. Ergo "Homebrew installs the stuff you need"
The Clojure team has expressed what the correct dependency is
The team only indicated that the current dependency is incorrect, not which formula is the correct one. If the correct way is not to depend on any Java formula that breaks the flow for new users and is equally unacceptable.
Why push users into a default configuration that is neither recommended by clojure's core team or its community of practitioners? It strikes me that establishing the wrong default is going to cause new and existing users of the language problems.
Yes, people mentioned that. The next person who only says it's wrong without providing an alternative will be billed for the time I spend on this.
Yes, people mentioned that. The next person who only says it's wrong without providing an alternative will be billed for the time I spend on this.
The OP says:
the previous dependency was correct.
This commit should be reverted.
That seems like an alternative to me?
Please can we keep this civil without criticising volunteers for their work, whether we like it or not.
Personally I'd prefer the depedency was left as it was; but if an openjdk release must be forced on people picking a release of either 8 or 11 would I think be preferable to users of this formula.
I appreciate the homebrew maintainers' work and respect their policies, and I hope others will as well.
While I have my own opinions, I accept the decision here and Clojure will move their formula into their own tap, so this does not need to be an area of conflict. We've been contemplating this for a while anyways.
Clojure will move their formula into their own tap, so this does not need to be an area of conflict.
We will not be removing this formula from homebrew-core until it is no longer widely used. Of course Clojure can create their own tap and recommend that if they prefer.
If someone can point out specific, reproducible steps to demonstrate how this configuration is broken (rather than "unsupported", "wrong" or "not recommended") we can consider modifying the formula so that these problems are addressed.
Additionally:
- If you'd like to use another Java installation feel free to set
JAVA_HOME
to any Java you want
I'm not sure how this doesn't address the problem (please clarify if it does not). If you do not wish to use the installed openjdk
formula: simply set JAVA_HOME
and ignore it.
If this is simply "Homebrew installs a dependency it didn't before and I don't like that": that's par for the course with a package manager.
Note if you're part of this conversation it's not at all helpful to just 👎 comments without actually attempting to answer the several questions that need answering to reach an adequate resolution.
As a more meta-point I realise language communities operate independently of each other and can't be expected to keep up with what all the others are doing but this is not an issue specific to Clojure or Java. NodeJS and several other languages and applications regularly say "but this formula doesn't depend on X!" when it actually does.
It may not depend exclusively on the specific implementation or version we have chosen but that's a wider Homebrew implementation detail. We used to have things that allowed random formulae to depend on any of multiple dependencies to fulfil the dependency (Requirement
s) but this was extremely error-prone and we're unable to support it adequately so we don't. We are currently moving Java formula to use this openjdk
dependency from the JavaRequirement
which causes us issues with maintaining our CI environment and requires users to install a cask (or equivalent) for a formula dependency. We are likely to do this for X11 if possible in future, too.
Hey @MikeMcQuaid and others, thanks for engaging productively.
Clojure does not actively test against "the latest version of Java", but rather tries to focus on the LTS releases (currently Java 8 or 11, soon 14). We do try to make sure it works on the interim ones, but that's not part of our current test matrix, and not part of what we recommend to users. Also, we try to be JVM-vendor-agnostic so that users on Oracle or other OpenJDK-based distributions should all work.
I don't know enough about what is possible or allowed with Homebrew to have a good handle on the space of options. Could we remove the Java dependency entirely and have the tools check and tell you what to do if it's not present?
With respect to our own tap, there are other possible benefits there but it would be actively harmful to leave an old formula here without updates and only update in a secondary tap. I am the only person building the tar files and publishing updates to the formula here. If I switch to doing that on a Clojure-specific tap, could we blacklist Clojure here with a pointer to the Clojure tap? Otherwise, this will become the default place to look with an increasingly out of date thing to find.
While you might be the first person to currently suggest updates that doesn't necessarily mean that the formula will be outdated when you no longer suggest them. If it's consistently outdated we can always consider a deletion though.
I’m part of the Clojure team, and the one doing 99% of the dev and release work on these tools. I’m going to start pushing updates only to a Clojure tap, so it will be consistently old.
For Clojure users, the best and most up to date source for these updates will be the Clojure tap. It seems like it can only be worse for users to get an old version from core without knowing it’s old. I don’t see how it benefits anyone to not point to the Clojure tap from the start?
@puredanger, to get back to the original issue that the “dependency is wrong”, would depending on the openjdk@11
formula be the “right” dependency?
I think we are talking past each other. What exactly is the problem?
If the issue is that users are getting a copy of openjdk
that they don't technically need, and the proposed solution is to change the dependency back to depends_on :java
, that's not going to happen. See https://docs.brew.sh/Building-Against-Non-Homebrew-Dependencies for why this is the case.
The fact that depends_on :java
existed for so long is largely a due to a lack of maintainer time and is undesired behavior. It causes many problems for our CI and increases our maintenance burden. Migrating to a brewed openjdk
has been in the works for a long time. The linked document also provides an alternative if you don't want a brewed jdk (i.e., maintain your own tap). Note that the clojure
formula specifically permits setting your own JAVA_HOME
which will permit you to use whatever Java you want.
If the issue is that clojure
only works with LTS releases of Java, then https://github.com/Homebrew/homebrew-core/issues/50536#issuecomment-589896834 appears to be the right solution, so please open a pull request for that.
@reitermarkus Yes, if brew must fetch openjdk, let it fetch the LTS release (currently 11, soon 14, basically every 3rd release).
Although, as mentioned before, some projects have a hard dependency on JDK 8 as well, since Java 9 removed a few capabilities.
if brew must fetch openjdk
The point of a package manager is that after installing a package, it works, so yes, we must.
some projects have a hard dependency on JDK 8 as well
That is the reason we still allow overriding JAVA_HOME
for formulae related to Java development. Still, the default will always be to depend on the latest version possible.
@MikeMcQuaid
If you do not wish to use the installed
openjdk
formula: simply setJAVA_HOME
and ignore it.
In my eyes, the problem with that suggestion is, how does one do that universally, on one's machine? How do I ensure that any invocation of java
on my machine, from any environment (that doesn't have a prescribed java version, such as redefining JAVA_HOME
, obviously), such as app runs, sudo runs, any process calls from any programming language, etc, uses that version? The same problem occurs with JAVA_OPTIONS
and is entirely not homebrew's fault - but it still an issue with that suggestion (IMHO).
@Hindol
let it fetch the LTS release (currently 11, soon 14, basically every 3rd release).
(14 isn't an LTS, btw. It's every 3 years not 3rd release, so the next LTS is Java 17, in Sept 2021.)
@reitermarkus
Still, the default will always be to depend on the latest version possible.
In https://github.com/Homebrew/homebrew-core/commit/f51759139edadc7e52d33b924c1b26ea058148f1 @SMillerDev also mentioned "This allows us to test tools with updates". I don't understand exactly how much this change is motivated by testing (there are a few mentions of CI troubles in the thread). But would it be acceptable if, as a default, the dependency resulted in the latest being used, but users had the option to satisfy the constraint of "need java" by their own selection of JDK variant/version?
Unfortunately, the break-neck speed of JDK releases (every 6 months, with only a 1 month overlap) is quite the imposition on the ecosystem, and honouring the 3 year LTS release cadence is more feasible and seems to have become fairly common. Much like the Clojure community, the Scala community (whom I'm a part of) tries to spend time on the non-LTS releases, but we're more certain of the software on the LTS releases. So, in some way, this isn't about "use the latest version" it's more like "use the latest pre-release".
Lastly, much ❤️ to the maintainers of the Homebrew ecosystem. I'm always grateful for all the work that goes in.
In my eyes, the problem with that suggestion is, how does one do that universally, on one's machine?
Set it in bashrc or equivalent and configure your IDE to also honor it, that covers all cases AFAIK.
But would it be acceptable if, as a default, the dependency resulted in the latest being used, but users had the option to satisfy the constraint of "need java" by their own selection of JDK variant/version?
That's the current situation that you can set with JAVA_HOME
.
I don’t see how it benefits anyone to not point to the Clojure tap from the start?
People will not start looking for the clojure tap by default and will install clojure in weird unexpected ways if it doesn't just work with a default install. Your solution only works if people go through your documents, find your tap and then try installing it.
In my eyes, the problem with that suggestion is, how does one do that universally, on one's machine?
Set it in bashrc or equivalent and configure your IDE to also honor it, that covers all cases AFAIK.
Wait, how does that cover all cases? How does it even cover sudo
? How does it cover a python script calling subprocess.run
(https://stackoverflow.com/a/54123892/463761). In my experience, it just doesn't.
Sudo has some options to retain environment variables and you can always set the variable for the target user as well. All of this is something that Linux users have used for years though. It's a proven concept and Homebrew is just joining everyone else here.
I'm not saying you can't seek and find workarounds at every usage site to try and make it use the right version of Java. But I hope you can see how "Oh, why don't you just set JAVA_HOME instead" isn't so straightforward in truth.
It's a proven concept and Homebrew is just joining everyone else here.
In a sense, so is macOS's java
and /usr/libexec/java_home
and the new setup actively subverts all that. I understand there's a good reason for it, but the "just set JAVA_HOME" alternative isn't great. (But I thank you for your answers.)
In my experience, it just doesn't.
@dwijnand I'm afraid it's not our job to explain to you how to configure your software and environment so they work how you want.
I don’t see how it benefits anyone to not point to the Clojure tap from the start?
People will not start looking for the clojure tap by default and will install clojure in weird unexpected ways if it doesn't just work with a default install. Your solution only works if people go through your documents, find your tap and then try installing it.
Additionally: https://formulae.brew.sh and similar tooling does not exist for your tap. People expect brew install clojure
to work and we're going to continue to ensure that it does. What is likely to happen is that other users or maintainers will step up and update clojure
if you do not wish to do so.
I think we are talking past each other. What exactly is the problem?
This is still happening. No-one can explain to me why this dependency is "wrong" and provide a test-case demonstrating this. Therefore I'm forced to draw the conclusion that there isn't one and this is an upstream preference and what they "support" rather than a hard requirement. Please prove me wrong.
Unless the conversation turns fairly rapidly in this direction this thread is going to get locked; it doesn't seem to be a productive use of anyone's time at this point.
Hi, for a little context I've given non-trivial economical support to Clojure and Homebrew alike. I see how both parts here are defending a reasonable technical approach, pursuing well-intentioned goals.
Also I see some flaws in how this issue was opened, and also in how it was immediately closed. Both facts can skew the whole discussion.
So it might be desirable to start a new issue from scratch, trying to stick to the facts and keep it technical.
As much as I admire Alex Miller (known within Clojure folks for his expertise in all JVM matters), clojure formula dependency is wrong
is not a problem statement - it's a perceived symptom.
Likewise, the JAVA_HOME
suggestion seems reasonable, and only one person (not OP) has tried to explain what might be wrong with it.
provide a test-case demonstrating this
This also seems necessary.
So it might be desirable to start a new issue from scratch, trying to stick to the facts and keep it technical.
I would rather we continue discussion on the closed issue. This is how Homebrew uses our issue tracker: things that are not immediately/obviously actionable are closed and may be reopened when they converge on a solution (or a new issue opened).
Otherwise: I agree completely, thanks @vemv.
Catching up (thanks @MikeMcQuaid @SMillerDev @reitermarkus). There's really two things going on here and I want to explicitly separate them if I can.
1 - I filed the original issue because the change in dependency was incorrect from the point of view of Clojure. Clojure depends on any Java, >= 1.8 so changing it to be "latest version of adoptopenjdk" was not that. However, I understand better now that the policy in homebrew is "use latest deps built from source", so the dependency change is entirely in keeping with homebrew's policy and is not wrong in that regard. I have not disagreed with that explanation or decision to close this issue on those points. (And indeed, post install, setting JAVA_HOME puts the user in full control of which Java is actually used, so that all works as expected.)
I do think it would be helpful to use openjdk@11
if that's an ok thing to do. It certainly matches better with clojure testing and release strategy. That was suggested above but it's hard for me to tell who is on homebrew team recommending things and who is just chipping in, so I'm not sure if that was a good suggestion or not. If so, I will open a PR for it.
2 - The second thing here is that from the Clojure team's point of view someone externally changed our dependency without any input from us. I understand that's how things work here, but it's disconcerting. As the creators of a thing, we want to have some level of control over how these decisions are made and presented to users. Homebrew is providing an incredibly valuable service to us as the favored package manager on our most popular platform, and I have a ton of love and respect for y'all doing that. What I'm trying to figure out is the best way that we can live in the homebrew ecosystem while also doing great by our users.
It seems like external taps are one big way that could be done, and it might be useful for a variety of reasons - control over the formula, control over releases (without needing to gate on the homebrew team to approve prs), providing tagged versions (@ etc), and external brew commands.
I've done a fair amount of googling and reading of brew docs and looking at what other communities are doing and certainly there seem to be many cases of organizations moving formulas both in and out of core for their own external taps but it's hard to determine the reasons in most of those cases. I would love to have more guidance on what you think about that, how to make that decision and how to work cooperatively so that our users have a clear path to getting the best possible version, with the minimum of confusion. (And I also understand if the answer is, "not our problem". I'm not asking you to solve it, just looking for best guidance.)
Thanks.
Thanks for the detailed response @puredanger :heart:.
And indeed, post install, setting JAVA_HOME puts the user in full control of which Java is actually used, so that all works as expected.
Glad to hear it 👍
I do think it would be helpful to use
openjdk@11
if that's an ok thing to do. It certainly matches better with clojure testing and release strategy. That was suggested above but it's hard for me to tell who is on homebrew team recommending things and who is just chipping in, so I'm not sure if that was a good suggestion or not. If so, I will open a PR for it.
I would rather we left it as-is unless there is known breakage.
I would love to have more guidance on what you think about that, how to make that decision and how to work cooperatively so that our users have a clear path to getting the best possible version, with the minimum of confusion.
From our perspective: once something is in Homebrew/core unless it breaks or we're unable to run CI on it for some other reason (e.g. depends on some systemwide change we're unwilling or unable to make to our CI system) we will keep it there indefinitely. We have the technical ability to migrate formulae to taps but this is not something we're willing to do for Homebrew/core essentially for security reasons.
The best way for upstream to ensure that the software is packaged the way they would like is to have the test do
block reflect necessary functionality and provide some basic regression testing. This stops us from deliberately or accidentally breaking the functionality.
In this case, if there were issues with the latest Java, changing the dependency and changing the test do
such that it produces an error with the "wrong" Java version would avoid us accidentally updating it to the wrong version in future.
I understand that's how things work here, but it's disconcerting. As the creators of a thing, we want to have some level of control over how these decisions are made and presented to users.
As a more meta-point: this isn't something Homebrew does but something most systemwide package managers do. We're responsible not just for packaging your software but ensuring that all software package operates as sensibly and well-integrated as possible. This results in us making decisions for our users on e.g. dependencies to minimise the amount of dependencies they need to install and generally be in keeping with the ethos of our package manager.
If it would be good to have explicit documentation "homebrew for upstreams" or similar then I can look at writing it in the nearish future.
@MikeMcQuaid we really want someone to manage their Java installation independently from Clojure. Could we remove the Java dependency entirely?
Could we remove the Java dependency entirely?
No, that would make the default install unusable.
we really want someone to manage their Java installation independently from Clojure.
They can and will by setting JAVA_HOME
What @SMillerDev said. Clojure does depend on Java (hence the dependency as it cannot run without it) but there's nothing to stop users from ignoring the openjdk
formula entirely and using their own Java with JAVA_HOME.
I was very surprised when I saw Homebrew unexpectedly installing openjdk
while upgrading Clojure to its latest release.
I have JAVA_HOME set to a Java 11 (adoptopenjdk), but Homebrew instead installs openjdk 13.
I ran the following for troubleshooting in brew irb
:
irb(main):001:0> JavaRequirement.new(%w[11+]).send("possible_javas")
=> [nil, #<Pathname:/Users/adamf/.asdf/shims/java>]
irb(main):002:0> JavaRequirement.new(%w[11+]).send("preferred_java")
=> #<Pathname:/Users/adamf/.asdf/shims/java>
irb(main):003:0> JavaRequirement.new(%w[11+]).send("java_home_cmd")
=> nil
irb(main):004:0> JavaRequirement.new(%w[11+]).send("satisfied?")
=> true
@adamfeldman Please read the above. Your contribution doesn't add anything new.
Please note we will close your issue without comment if you delete, do not read or do not fill out the issue checklist below and provide ALL the requested information. If you repeatedly fail to use the issue template, we will block you from ever submitting issues to Homebrew again.
brew update
and can still reproduce the problem?brew doctor
, fixed all issues and can still reproduce the problem?brew gist-logs <formula>
(where<formula>
is the name of the formula that failed) and included the output link? https://gist.github.com/puredanger/bf1597cf60ed911fd70348b9953c18f9brew gist-logs
didn't work: ranbrew config
andbrew doctor
and included their output with your issue?What you were trying to do (and why)
Install Clojure and use my existing Java installation.
What happened (include command output)
brew install clojure
Command output
What you expected to happen
Clojure formula should be installed but I already have a valid Java set up and I did not want openjdk installed. The clojure formula was recently changed incorrectly from Java >= 1.8 to adoptopenjdk.
I am on the Clojure core team and wrote the clojure brew formula and installer.
Clojure does not require adoptopenjdk. Some users use it with the Oracle JDK or other vendor JDKs. The requirement from a language perspective is any valid Java, >= 1.8, so the previous dependency was correct. Many Clojure users prefer to have control over their JDK version and most users are still using Java 8 as the module system introduced in Java 9 may require changes before migration.
Without the opportunity to do that migration, auto-installing adoptopenjdk at the latest version can cause their application to throw warnings or fail (due to modules that have been broken out of the JDK and are no longer automatically provided). Here is an example issue of how this can manifest to users: https://github.com/clojure-emacs/orchard/issues/20 (this particular issue was not caused by this brew change, but is exactly the experience users with a forced Java update may encounter).
This commit should be reverted.
Step-by-step reproduction instructions (by running
brew install
commands)brew install clojure