Move Linuxbrew to separate organisation

[MikeMcQuaid]

This is partly because it's confusing, partly because Linuxbrew is still alpha/beta compared to the stable state of other repos and because Travis CI handles builds per-organisation. It'd also allow you to have more control over managing teams.

Any objections? CC @Homebrew/linux @sjackman

[sjackman]

My initial reaction is that I'd prefer to keep Linxubrew within the Homebrew organization. Linuxbrew may not be as stable as Mac Homebrew, but it gets better every day and has over 1,700 :star: now. We now have five people volunteering to help out with Linuxbrew, which is a drastic improvement from just me. I'm afraid that I don't understand the benefits with respect to Travis; perhaps you can elaborate if it's an important point. Now that I can add people to the @homebrew/linux team, I have all the control over teams that I need.

I leave traveling for two months this Friday, and changing the organization is going to require some code changes to Linuxbrew and changing URLs and what not. Probably not a lot, but I'd rather not have to tackle that particular job until I get back. I'd appreciate your holding off on this change until the new year.

I'd appreciate hearing the thoughts of @xu-cheng, @DomT4 and @tdsmith.

[DomT4]

My gut feeling is that at the moment removing Linuxbrew from the Homebrew organisation doesn't solve an immediately glaring problem and to some degree would appear to peel back Linuxbrew's legitimacy; In many ways if we wanted Linuxbrew to be a separate organisation the time for that would have been at creation or very shortly after that point.

The situation obviously isn't perfect, we need firmer rules on what stays in Linuxbrew etc and what should be sent upstream, and we need to deal with whatever the Travis situation happens to be, but I'm inclined to try steps to resolve that and see how it pans out before we think big.

I agree Linuxbrew is still more of a WIP than Homebrew, but that's easily enough explained in the README if there's people actively confused over the situation. Curious to know more on Travis.

[MikeMcQuaid]

Note we can redirect the organisation and all repo clones and API calls will be automatically updated to the new location.

[xu-cheng]

Personally, I kinda opposite to move Linuxbrew into a new organization. I agree with DomT4 that moving wouldn't solve any immediate problem. The Travis part isn't a problem for Linuxbrew for now as it doesn't use any. And we can use Travis on fork when it's required(for example, creating a bot account for Linuxbrew).

Beside, it may cause unnecessary confusion to the existed Linuxbrew users. If helps, README should be enough to clarify for new users.

Last but not least, to improve Linuxbrew in long term will require some work on Homebrew's end. For example, we should add better os abstraction other than relying on OS.mac?/linux?.

[MikeMcQuaid]

Beside, it may cause unnecessary confusion to the existed Linuxbrew users. If helps, README should be enough to clarify for new users.

I don't know how it'll cause any confusion at all if there's an automatic redirect. I think it's more confusing currently.

Linuxbrew and Homebrew have completely different maintainers and focuses. I think having Linuxbrew in the Homebrew organisation encourages the OS.linux? stuff which is a bad code smell.

[ahundt]

I think it would make more sense to keep it in the same organization. In fact the compatibility is the whole reason I started using linuxbrew! What's wrong with formulas supporting both operating systems?

[ahundt]

Actually, I've always been confused why they aren't even just the same project. That would be even more ideal from my perspective.

[MikeMcQuaid]

They aren't the same project because it adds a huge overhead on Homebrew maintainers to verify things work on Linux and the use-case on Linux is a lot more niche (as it already provides good, system package managers).

[ahundt]

@mikemcquaid that does seem to fit well with the minimalist selection of features you guys prefer, though I'd still appreciate if the separate project can at least remain under the same organization since there are obvious ties between the two.

[tseemann]

Have you considered how splitting off Linuxbrew would affect the other taps?

Homebrew-science for example is heavily Linux based, and most of my time is spent (possibly wasted) on getting formulae to build on OS X, as most of the development is Linux based. The

But perhaps once again this is desirable, as those taps are not core system packages.

[MikeMcQuaid]

To repeat: I don't believe splitting Linuxbrew will affect users or development in any way other than it being less affiliated with Homebrew.

[DomT4]

Have you considered how splitting off Linuxbrew would affect the other taps?

Extensively.

[tseemann]

@DomT4 Is that extensive consideration publicly viewable? I don't mind if it isn't, I was just interested in what was topics were covered, and what conclusions you came to. "Extensively" doesn't tell us much :)

[DomT4]

I'm not even sure it's privately viewable any more to be honest; was in the team chat and it only logs so many messages and I never shut up to the point that I'm often only talking to myself.

There was a thorough discussion about taps. It is perhaps worth noting that I don't think there would be much code upheaval; what works now should continue working. My concerns as stated further up in this thread were more the messaging implications in how it would look.

Either way, nothing's going to happen until Shaun is back and further discussions as needed can happen.

[MikeMcQuaid]

This comment is an(other) example of why I'd like to see more distance between the projects: https://github.com/Homebrew/homebrew/commit/278337dcaa346911306d5e63b72522ad0d4570d8#commitcomment-14391539

[ahundt]

Sorry to belabor the point but from another perspective that's a perfect example of why they should be merged!

[MikeMcQuaid]

@ahundt To ask them to be merged is to ask the people who spend significant amounts of time working on Homebrew to spend more time testing all their changes on Linux for a project with (at most) 1/10th of the users for an OS they don't use which already provides a system package manager.

[ahundt]

@mikemcquaid I understand, I say this from a hopeful perspective rather than a demanding one. Thanks for taking such good care of homebrew. :+1:

[ahundt]

I have an idea I'll throw against the wall to see if it sticks:

What about minimal, separate core homebrew/linuxbrew repositories, then split the formula folder in both into its own separate default tap? Then it would be less troublesome for linuxbrew and homebrew to remain forked, and formula updates that aren't major breaking changes will simply be automatic for both.

There may be something I don't forsee but I figure it is worth mentioning the idea.

[MikeMcQuaid]

That's been a plan for years but it'll take a long time to get there and still won't address the issue that Linuxbrew has large numbers of formulae that are broken on Linux.

[MikeMcQuaid]

@sjackman Please shout when you are no longer travelling.

[retokromer]

In my personal opinion, it would make more sense to keep Linxubrew within the Homebrew organisation.

[sjackman]

Hi, Mike. I'll be back in the new year, some time after Jan 3.

[MikeMcQuaid]

@sjackman Can you give your thoughts on the above? Thanks.

[sjackman]

My preference is to keep Linuxbrew in the Homebrew organization. It shows that Linuxbrew is a part of the Homebrew ecosystem, and not simply a similarly named but unrelated project. I don't see a particular technical advantage to moving Linuxbrew to its own organization.

[MikeMcQuaid]

@sjackman We keep getting Linuxbrew issues filed on Homebrew, it doesn't have enough maintainers or a focus on quality (e.g. you fork Homebrew's formulae many of which don't work). This all reflects badly on Homebrew and, honestly, Linuxbrew maybe benefits from being part of the Homebrew ecosystem but it now detracts from Homebrew. If it had it's own organisation it can sink or swim as an independent project and you can have less oversight. If not, there's a bunch of short-term changes I'd like to see relating to the above for it to stay here. I was the person who migrated Homebrew to the organisation from mxcl/homebrew and it's completely transparent to users. That said, I'd be willing to do the work to ensure it's the same for Linuxbrew.

[sjackman]

I don't think that moving Linuxbrew to its own organization is going to resolve the matter of users submitting issues to the wrong project. I am keen to hear what short-term changes you think could improve the situation.

I'm sorry to hear that you think Linuxbrew detracts from Homebrew. I don't believe that's a fair assessment. Linuxbrew has just about two thousand stars on GitHub. That's a significant number of people who have found the project interesting or useful. I personally find it very useful to be able to use the same package manager on both Mac and Linux. That's a feature that's quite unique to Homebrew/Linuxbrew.

I have a keen interest in quality, and I do as best as I can within my time constraints. I'm a father of two young kids and a PhD student. I agree that Linuxbrew does not have enough maintainers. I need more help.

[MikeMcQuaid]

I don't think that moving Linuxbrew to its own organization is going to resolve the matter of users submitting issues to the wrong project. I am keen to hear what short-term changes you think could improve the situation.

I think it will. There's been confusion on Twitter about it being an "official" Homebrew project and the URL points to that being the case.

I'm sorry to hear that you think Linuxbrew detracts from Homebrew. I don't believe that's a fair assessment. Linuxbrew has just about two thousand stars on GitHub. That's a significant number of people who have found the project interesting or useful.

I don't think GitHub stars is an objective measurement. If someone found it useful and then stopped using it do you think they would go back and unstar the project?

I personally find it very useful to be able to use the same package manager on both Mac and Linux. That's a feature that's quite unique to Homebrew/Linuxbrew.

This is exactly my point: it's not the same package manager. It's an attempt to port Homebrew to OS X.

I have a keen interest in quality, and I do as best as I can within my time constraints. I'm a father of two young kids and a PhD student. I agree that Linuxbrew does not have enough maintainers. I need more help.

I appreciate that; we're all time constrained. That said, I really think Linuxbrew has been around long enough that it should sink or swim on it's own merits in its own organisation. As to quality: you should whitelist rather than blacklist formulae that don't work, move away from OS.linux?/OS.mac? checks in both core code and formulae code and work towards better abstractions.

I don't think we're going to come to mutual agreement here and it's obviously I feel fairly strongly about this; after all I moved it into the Homebrew organisation originally so I feel this was ultimately my mistake and want to rectify it by helping the transition. Are there any @Homebrew/maintainers who would actively block my moving Linuxbrew into a separate organisation?

[bfontaine]

I won’t actively block; I don’t think it’ll harm Linuxbrew to move under its own organization. Homebrew-Cask is under its own and AFAIK they’re doing great. You’ll have more control and will be able to create taps dedicated to Linuxbrew; something that’s hardly possible under the current org without causing confusion. If you want to use Travis you’ll also get more workers because right now you have to share them with all Homebrew repos.

[retokromer]

I second @mikemcquaid's remark regarding a white-list rather than a black-list for formulae.

On the one hand, as said, I would personally prefer having linuxbrew inside homebrew. On the other hand, frankly speaking, I have the impression -- I really hope my impression is wrong! -- that @sjackman is willing to run an one-man-show, despite his calls for help. I fixed myself the "things" I need on the company's computers and render-farm, but I gave up to submit the patches to linuxbrew. Sorry!

[UniqMartin]

I'm alright with Linuxbrew leaving the Homebrew umbrella. I think it will reduce confusion and will bring more freedom to Linuxbrew, maybe even more visibility and thus new maintainers to the project, without denying its origins and its close ties to Homebrew (that is still evident from its name).

[DomT4]

I guess I should say something, being both a Homebrew maintainer and the accidental maintainer-face of Linuxbrew for the last few months. I was also only beaten in commits across the entire Homebrew organisation last year by BrewTestBot (which isn't a fair fight), so I see a lot of what goes on.

Linuxbrew is a useful project. It fills the niche, and there is one, of wanting the very latest and greatest software on Linux but not wanting to run an unstable distro, or not wanting to manually wget and ./configure every project on the planet which however much regular Linux users insist they love doing I'm convinced is just not true :smile_cat:.

I think Linuxbrew has some real structural issues which need addressing, and I'm not sure how practical it is for Shaun to single-handedly manage that given has his own things in life which have to take priority over an open-source project such as family and education. I am highly sympathetic towards how much work Linuxbrew is, even without starting work on the kind of core changes the project needs going forwards.

Some of these structural issues are small, like brew config being 50% useless on Linux, which is something that wouldn't necessarily be difficult to fix but nonetheless remains very OS X oriented and looks fairly bad in terms of UX. The more fundamental problem is the amount of code that is wrapped in if OS.mac? or if OS.linux? - Linuxbrew really needs its own core rather than trying to bend Homebrew's core to awkwardly fit, or we need to start working on core PRs to make it a better cross-platform experience by creating or switching to methods that work on both platforms, but without significantly worsening the experience for OS X users who are the primary use-case for Homebrew.

This leads me onto my next point, which is that there seems to be a lot of general interest in Linuxbrew but not a lot of interest actually helping maintain this ship and let it rig its own sails. To my knowledge, of everyone who offered to help, nobody who wasn't a core Homebrew maintainer or Dana contributed much in Shaun's absence. As far as I saw, 95% of the time everything was handled by @dunn, myself or Dana. I can understand maintainers elsewhere not having the time to contribute to Linuxbrew, and consequently think very strongly Linuxbrew needs its own team that just does Linuxbrew.

There is a lot of confusion as to who is responsible for this project. It does seem to be broadly believed that the core Homebrew team also manages the Linuxbrew project, and we also get a lot of Linux-exclusive issues reported to the core. We also get issues reported to the taps that we don't have the knowledge to fix or the fix is too hacky to merit doing, and it either sits broken or we nudge Shaun to help.

I agree per the suggestions above it would perhaps be good for Linuxbrew to essentially start with a formula whitelist and slowly re-add formulae that are known to work. I also think it could be an exciting opportunity for Linuxbrew if moved to its own organisation to choose to stop supporting some Homebrew taps and merge that functionality into Linuxbrew core - It doesn't make sense that Linuxbrew needs a fair bit from homebrew/dupes because those things aren't dupes on Linux. That's another minor UX thing that feels unfinished and rough.

I think any damage to Linuxbrew by moving it would be relatively minimal. There's an automatic redirect courtesy of Github, and we can update the links in the Homebrew core documentation that point to it. There's certainly no suggestion of booting Shaun himself out of the Homebrew organisation via his involvement in Science. I think there's a lot of opportunity here for Linuxbrew, as explored in detail above.

I understand the desire to remain within Homebrew, but I don't think moving is the worst thing in the world and gives you a lot more freedom in a way to tear up the playbook and not have so many of these external pressures on you that come about as being part of "official" Homebrew. The concept of "official" Homebrew is a little outdated in the world where taps are so prominent and can be CI tested via Travis using the same brew test-bot command the core uses, to be honest.

I wrote this over the space of the day on and off, so the tense and "audience" is all over the place. Hopefully it somewhat makes sense. I probably won't comment a huge amount beyond this.

[MikeMcQuaid]

Thanks @DomT4, said it better than I could.

[tdsmith]

I'm +0 on moving; I don't know that I think any of the pro or con arguments are very urgent.

I think Linuxbrew and homebrew-science together meet a unique need for home-directory scientific software management on Linux.

I don't know that arguments about branding matter deeply to users; I do wish that Linuxbrew was more reliable but I doubt that Linuxbrew significantly contributes to Homebrew's reputation (for better or for worse) and I don't think that people adopt Linuxbrew because it is "a Homebrew project." They adopt Linuxbrew because they wanna run a newer version of blast on their cluster. :) And if they're using Apple laptops and are already familiar with Homebrew's syntax and conventions, so much the better. None of that changes based on the repository URL. Linuxbrew will still be brew for Linux.

I don't think that having Linuxbrew in the Homebrew organization contributes significantly to the Homebrew maintainership burden; I don't think the occasional misfiled ticket is especially annoying to handle. (I'm aware that I haven't exactly been putting my money where my mouth is wrt maintenance duties lately, so weigh my statements accordingly.)

The only reason that would be important to me to migrate Linuxbrew out of the Homebrew organization is the potential for more aggressive CI on Travis. I have found it frustrating to support Linuxbrew because the supported environments aren't well-described or well-tested and I end up breaking things on versions of CentOS I have never once cared about. Since the number of simultaneously active Travis workers is capped per organization, if Linuxbrew starts using more Travis workers to cover more platforms through clever (ab)use of docker, that would degrade the Homebrew contribution and maintainership experience unless we complete this migration. (It's possible I could be wrong because e.g. maybe OS X and Linux workers are tallied independently, or because we might be able to get Travis support to smile upon the Homebrew organization.)

[MikeMcQuaid]

if Linuxbrew starts using more Travis workers to cover more platforms through clever (ab)use of docker, that would degrade the Homebrew contribution and maintainership experience unless we complete this migration. (It's possible I could be wrong because e.g. maybe OS X and Linux workers are tallied independently

From what I can tell: yes, this is the case and I think it's a good enough reason to move to a new org by itself.

or because we might be able to get Travis support to smile upon the Homebrew organization.

I tried, failed here.

[drbenmorgan]

I think Linuxbrew and homebrew-science together meet a unique need for home-directory scientific software management on Linux.

I don't know that arguments about branding matter deeply to users; I do wish that Linuxbrew was more reliable but I doubt that Linuxbrew significantly contributes to Homebrew's reputation (for better or for worse) and I don't think that people adopt Linuxbrew because it is "a Homebrew project." They adopt Linuxbrew because they wanna run a newer version of blast on their cluster. :) And if they're using Apple laptops and are already familiar with Homebrew's syntax and conventions, so much the better. None of that changes based on the repository URL. Linuxbrew will still be brew for Linux.

Pretty much this from my own use cases as well - Homebrew/Linuxbrew have been incredibly helpful (thanks to the team and contributors!) for creating/deploying scientific software stacks across laptops, desktops and clusters with a common system (brew, the Formula DSL, taps, and no root permission requirement). So from this user's perspective, as long as these conventions remain it makes no difference to usability if linuxbrew moves in the short term.

Some of these structural issues are small, like brew config being 50% useless on Linux, which is something that wouldn't necessarily be difficult to fix but nonetheless remains very OS X oriented and looks fairly bad in terms of UX. The more fundamental problem is the amount of code that is wrapped in if OS.mac? or if OS.linux? - Linuxbrew really needs its own core rather than trying to bend Homebrew's core to awkwardly fit, or we need to start working on core PRs to make it a better cross-platform experience by creating or switching to methods that work on both platforms, but without significantly worsening the experience for OS X users who are the primary use-case for Homebrew.

This leads me onto my next point, which is that there seems to be a lot of general interest in Linuxbrew but not a lot of interest actually helping maintain this ship and let it rig its own sails. To my knowledge, of everyone who offered to help, nobody who wasn't a core Homebrew maintainer or Dana contributed much in Shaun's absence. As far as I saw, 95% of the time everything was handled by @dunn, myself or Dana. I can understand maintainers elsewhere not having the time to contribute to Linuxbrew, and consequently think very strongly Linuxbrew needs its own team that just does Linuxbrew.

Guilty as charged... What's the feeling, if any, on Linuxbrew needing its own core versus providing more OS abstraction in Homebrew (in other words, where can contribution be most useful to both projects)? The same goes for the (desired/planned?) feature of the Library/Formula folder becoming a tap - it's noted as a lot of work, but seems useful in the context of maintaining Formulae for both platforms.

[sjackman]

Thanks to everyone who has taken the time to compose their thoughts. I'm following this thread, and I'll contribute my detailed thoughts later today. This conversation has become quite constructive. I'm looking forward to discussing more how we can improve Linuxbrew.

[ahundt]

I agree with @DomT4 @tdsmith on nearly everything they said, though it would be nice if linuxbrew could stay under homebrew. I'd like to note that I'm a PhD student myself, and thus I fit squarely under the scientific use cases mentioned by @tdsmith .

@mikemcquaid wrote: @sjackman We keep getting Linuxbrew issues filed on Homebrew, it doesn't have enough maintainers or a focus on quality (e.g. you fork Homebrew's formulae many of which don't work). This all reflects badly on Homebrew and, honestly, Linuxbrew maybe benefits from being part of the Homebrew ecosystem but it now detracts from Homebrew.

Sorry about the cases where I was the culprit on this! I'm just a user and as such lend more weight to those doing the heavy lifting, particularly the great work of @mikemcquaid on homebrew. I do differ from him in that I find the disinterest in growing homebrew to be cross platform to detract from the project much more than the quality of linuxbrew detracts from homebrew. I'm primarily a C++ developer and nearly all my code is cross platform. I've found the extra burden of setup and maintenance to be very small, perhaps 1-9% additional work. In fact I've patched many linux only projects to support OS X and I find it helps bring in users and eliminate bugs that wouldn't turn up as easily on a single platform. Admittedly, the whole point of both linuxbrew and homebrew is to help users deal with package management so the overhead % could be much higher for this specific project. Nonetheless, that's why my own perspective differs from @mikemcquaid.

@DomT4 wrote: This leads me onto my next point, which is that there seems to be a lot of general interest in Linuxbrew but not a lot of interest actually helping maintain this ship and let it rig its own sails.

I've tried to submit patches when I know enough to do so but either sufficient documentation or someone who knows the code base needs to be available to get new developers up to speed so they can create genuinely useful contributions. I've patched a number of formulas on homebrew/*, but I don't really know how to fix up some of the lower layers of code that make things tick without spending a very long time learning the intricacies of not just the technical details but also matching the vision of what the most involved developers prefer and do/don't want to maintain.

My goal for linuxbrew was reusing my OS X setup process for my project to create one that is easy for others to install on ubuntu 14.04. I didn't know enough to make useful contributions within my time constraints without some support from a more expert developer/contributor, which happened to coincide when @sjackman wasn't available, particularly at the very beginning. The solution I've currently deployed implemented for ubuntu 14.04 doesn't involve Linuxbrew since others following my instructions (and sometimes myself) would often get stuck on a compiler error they can't fix because they don't have the necessary experience with brew.

As a side note, I think some sort of working continuous integration would need to be running on a selected set of platforms to verify merges don't break things, and perhaps the whitelist that was mentioned. Hopefully this hasn't strayed too far off topic and is useful feedback. Thanks again to everyone for your great work on both projects!

[sjackman]

Though I would prefer to keep Linuxbrew within the Homebrew organization, I'm okay with moving Linuxbrew to its own organization if you feel strongly about it, Mike. I've registered http://linuxbrew.sh to give Linuxbrew its own wee space on the web.

The whitelist/blacklist has been mentioned a few times. It's a good idea. I will start blacklisting software that is known not to work on Linux and which requires work. To develop a whitelist of software that's known to work, I'd like to attempt building every formula with Linuxbrew and tabulate the results. What system is in place for creating all the bottles for a new release of Mac OS? Perhaps I can use that system for this purpose. If Travis is up to this grand task, then perhaps moving Linuxbrew to its own organization with its own pool of workers would be helpful. If not Travis, I can try using the computer cluster at work, but using publicly accessible tools would be preferable.

I'd appreciate the help of anyone, particularly those familiar with the Homebrew Travis build system, that would like to help out with this project.

[MikeMcQuaid]

@sjackman Thanks Sean, I appreciate it. Can you give me owner rights to https://github.com/linuxbrew and I'll start doing some migration tests.

To be clear (as I perhaps wasn't before): Homebrew (and me particularly) are working on various things without Linuxbrew specifically in mind but with cross-platform compatibility. In 2016 we hope to have (some) bottles for OS X generated and uploaded on Travis CI and core code and formulae separated into separate repositories. When we've done that separation it'll mean it'll be much easier to have working and keep working on Linux (and perhaps other platforms) and move towards stable core releases rather than a rolling release process. This will make it easy for Linuxbrew to either just be a whitelisted Linux tap and installer for Homebrew which more-or-less matches the current maintenance patterns but without the overhead of running a full fork.

[sjackman]

I've invited you to the Linuxbrew org, Mike. The portability improvements planned for Homebrew and bottle support for Travis are exciting.

[sjackman]

I'm visiting family in Montréal from Jan 14–18 and will have limited access to the Internet. Shall we hold the migration until after I return?

[MikeMcQuaid]

@sjackman No worries, shout when you're back.

[sjackman]

I fly back tonight, so go ahead with the migration.

[MikeMcQuaid]

@sjackman. Done! I added @tseemann to the team, gave the @Linuxbrew/core team admin access and ensured the redirects were setup and working properly. Thanks for your patience and support here.

[sjackman]

Great! I'll check it out tonight and do a bit of work on my side… Change Homebrew/linuxbrew to Linuxbrew/linuxbrew in the code base. Redirect http://linuxbrew.sh to the new org. http://brew.sh/linuxbrew/ is giving a 404 error now. :cry: Set up Travis CI. This last one is clearly a bigger project, but a fantastic one when it's finally working.

[sjackman]

Mike, could you redirect http://brew.sh/linuxbrew to http://linuxbrew.sh ?

[MikeMcQuaid]

@sjackman Done!

[sjackman]

Thanks for the speedy response, Mike!