Formulae names & dependencies

[luispuerto]

Here are some ideas we should keep in mind about the renaming of the formulae and if we should bring dependencies into the tap. This issue is highly related to #769.

Any opinion is highly welcomed. @fjperini and specially @miguelangelperg that seems that it's suffering from "name issues".

---

• We need to keep the number of formulae in the tap as low as possible.
  More formulae → more to maintain → more work → more difficulty to keep everything in shape.
• We should try to rely as much as possible into core formulae.
  • We don't have to maintain it.
  • Core formulae is more wide available to other people so it's easier to detect bugs.
• We are the Hombrew tap of the OSGeo project, so should try to host just the formulae related to OSGeo projects. In the same way, shouldn't we host all the OSGeo projects related formulae instead of host them in core?
• Any other package in the tap should fall into these two categories:
  • It's a dependency or it enhances the software in some way and it isn't in the core.
  • Can we add it to Homebrew core?
  • Do we really need to bring that formula to our tap?
  • It's a dependency or it enhances the software in some way and it is in the core, but it doesn't fulfill our needs.
  • Do we really need to bring that formula to our tap?
  • Can't we really use core formula as it's?
  • Can we propose a PR over that formulae that will fulfill our necessities while while keeping the current requirements of Homebrew core?
• There are currently some OSGeo projects that currently have formulae in the core tap. For example gdal, pdal, postgis... We have some of those formulae cloned in this tap with a different name so it doesn't shadowed the core formulae. What we should do?
  • Does it have any sense that there are two formulas that do the same?
  • Since it's OSGeo software, shouldn't we ask for deletion the formula in core and keep it here instead?
  • Can we propose a PR over that formulae that will fulfill our necessities while while keeping the current requirements of Homebrew core?
• Whether it's a OSGeo project package formulae or not, we need to keep in mind the following if we decide to duplicate:
  • Core is going to have always preference, since depends_on: in any core or other tap formulae unless otherwise is specify is going to use core formulae. Probably they are going to be linked too first.
  • They are thinking to deprecate pinned taps.
  • We probably need to keep our formula keg only so it doesn't interfere with other formulae that could use the core version, but this is something we need to decide.
• Name of formulae should be consistent:
  • If we duplicate a core formulae we should name the formula something like osgeo-<formula_name>.
  • Try to avoid numbers, even less if they are related to the version. The formula always is the last version of the software, any other version is <formula-name>@<version-number>
• We probably can't figure part of this on our own and we need to contact people on Homebrew and in OSGeo.
  • In Homebrew: to know their opinion about all of these and how they can help, and if they are willing to.
  • In OSGeo: I really don't know who has create gdal, pdal and other formulae relate to OSGeo projects in core, but I guess OSGeo project have some opinion about if formulae for their project should be in core or in osgeo4mac taps.

Thoughts?

[kevinstadler]

I just updated to 3.6.3 as well and cloned the profiletool plugin into the same directory as you, but I can still only see core plugins in the plugin manager.

Just to check, does the "Settings" tab also look like this for you?

During installation I also get the following message, maybe this has something to do with it?

==> Pouring osgeo-qgis-3.6.3.high_sierra.bottle.tar.gz
Warning: osgeo/osgeo4mac/osgeo-qgis dependency gcc was built with a different C++ standard
library (libstdc++ from clang). This may cause problems at runtime.
GRASS 7 GrassUtils.py already updated
Whitebox Tools whiteboxProvider.py already updated
TauDEM taudemProvider.py already updated
LAStools LAStoolsProvider.py already updated
Warning: The post-install step did not complete successfully
You can try again using `brew postinstall osgeo/osgeo4mac/osgeo-qgis`

Even with --verbose I don't get any more information about why exactly the step "did not complete successfully"

[fjperini]

@kevinstadler The message is just a message, it is not an error. As for the plugins window, try to delete the QGIS configuration ~/Library/Application\ Support/QGIS, it may be due to that.

If you can share the result of brew doctor to check if you have a problem with any dependency.

[kevinstadler]

Nevermind, I think I got it:

../src/app/qgisapp.cpp:10483 : (loadPythonSupport) [101ms] load library /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/qgispython (3.6.3)
../src/core/qgsmessagelog.cpp:29 : (logMessage) [10ms] 2019-05-23T19:47:40 [1] Couldn't load Python support library: Cannot load library /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/qgispython: (dlopen(/usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/libqgispython.dylib, 10): Library not loaded: /usr/local/opt/python/Frameworks/Python.framework/Versions/3.7/Python
  Referenced from: /usr/local/opt/osgeo-qgis/QGIS.app/Contents/MacOS/lib/libqgispython.dylib
  Reason: image not found)

With so many different Pythons installed I was sure there would be a 3.7 among them but apparently not (and it doesn't seem to be among the osgeo-qgis dependencies?). I'm just running brew install python37 right now, I hope that will do the trick!

Edit: I actually had a 3.7 installed in /usr/local/Cellar/python3/3.7.3 but for some reason this wasn't linked anywhere, the hard-coded /usr/local/opt/python/Frameworks/Python.framework/Versions/ path only had a 3.6 subdirectory.

[fjperini]

@kevinstadler Yes, you need to have linked python.

brew link python --force

[zbeekman]

👋 Howdy! Homebrew maintainer here, also in the process of reviewing a submission to JOSS. I haven't read this entire thread, but just wanted to weigh RE: depending on packages in core, duplicating packages, option removal.

Dependencies in homebrew-core

I would highly recommend you try to depend on formulae in home-brew core since it's very well tested, predictable, and plays nice with other brewed software. In addition, when a dependency of a package is updated, all the formula that use it are tested and the linkage is tested.

Option removal

I understand the desire to provide flexibility, but keep in mind that there is a combinatorial explosion of possible builds if a package has options and it's dependencies have options, and their dependencies, etc. Since this change, the rate of errors reported by users has significantly dropped, and the vast majority of software gets installed from a bottle which has been tested in a very controlled and reproducible environment.

If you see a package in homebrew core where you wish there was an option that was previously removed, then there are two possible recourses for regaining the missing functionality:

1. If the missing option does not conflict with the current default build and simply adds additional functionality, you are welcome and encouraged to submit a pull request making it enabled by default. Unless it breaks something or adds a very deep dependency tree it should get accepted
2. For cases where an option would conflict with a default or where it doesn't simply add features but changes the existing functionality, homebrew-core will consider formulae duplicated but built with other options, if they were popular in the past

The ultimate goal is to provide the most functionality possible, while minimizing problems.

Duplicating packages

You're always welcome to duplicate packages in your tap. However, if software is licensed with an open-source license, there's little recourse for removing packages from homebrew-core. The arguments above, IMO, favor trying to contribute as much as you can upstream to core for better testing and leveraging existing infrastructure and expertise.

I definitely don't want to re-litigate the option remove discussion, but wanted to share my $0.02 here, while reporting an issue I encountered.

[grischard]

@zbeekman as far as I understand, the beginning of all this was that a .app could only be installed as a cask, and a cask couldn't have tons of dependencies like osgeo-qgis has. Is our understanding of homebrew-core's policies correct?

[luispuerto]

@zbeekman thanks a lot for your contribution to the discussion, we really appreciate!! However, as @grischard has said, QGIS is a a little bit special —as almost anything in GIS— so it's a little bit more complicated.

We really have in mind to try to make as reliant in Core as possible, mainly for the reasons you stated, but things take time and we at this moment decided to head in this direction because we thought was going to be quicker.

[zbeekman]

Hi all, sorry for my slow response, I've been underwater.

@grischard

@zbeekman as far as I understand, the beginning of all this was that a .app could only be installed as a cask, and a cask couldn't have tons of dependencies like osgeo-qgis has. Is our understanding of homebrew-core's policies correct?

Yes, this is correct. However, the whole point of an app bundle to bundle the required dynamic libraries and other assets within the app bundle. CMake has fairly good and extensive support for doing this. Apple has the install_name_tool that lets you copy dylibs into your app bundle and re-write paths in executables and dylibs to find the bundled libraries. Furthermore, tools exist out there to "freeze" python applications and bundle them and their dependencies as apps. There was a discussion on discourse.brew.sh here about this: https://discourse.brew.sh/t/python-gui-app-formula-core-or-cask/3863. In particular https://github.com/pyinstaller/pyinstaller looks like a promising tool to help with this. Given the rather extreme complexity of the OSGeo codebase/toolchain, a standalone app bundle really makes the most sense to me here. I don't know of any other .app that has external dependencies.

@luispuerto

We really have in mind to try to make as reliant in Core as possible, mainly for the reasons you stated, but things take time and we at this moment decided to head in this direction because we thought was going to be quicker.

Sure, I understand sometimes time pressures require you to implement something as a temporary work around. I would just like to point out that while your implementation does an OK job of not shadowing core formulae, there are a number of problems, like #1198. Opening and parsing every single formula at once is not a good idea, and will cause breakage for many users when they hit the upper limit on max open files. In addition, it's super slow, so just processing the Formula takes a long time, nevermind waiting for a long list of from-source installs.

So my (completely unsolicited) advice is:

1. Do everything in your power to "freeze" and bundle qgis as a stand alone app. (And anything else with a crazy amount of complexity and a GUI.)
2. Try to reduce the number of options in osgeo formulae as much as possible by enabling sane defaults that are as broadly useful as possible, and shipping multiple versions of the same formula, where needed, if there are conflicting options.
3. Try to run brew audit --strict from time to time to attempt to keep up, where possible, with upstream Homebrew best practices and current, non-deprecated or removed functionality.
4. Enabled by 2 and 3, where possible open PRs to port changes in shadowed formulae or alternate versions of formulae back upstream to homebrew-core. I realize this can be difficult because homebrew-core is opinionated about style & maintainability. But where you're able to do this, you shift much of the maintenance burden of formulae to the greater community and the homebrew maintainers, and you're able to leverage the robust Homebrew CI environment and ship bottled, tested binaries to users.

You're welcome to take it or leave it. But, I'm a Homebrew maintainer and if I'm running into problems trying to install your software (see #1198) then users most certainly are. See, also, for example, https://twitter.com/manlius84/status/1139490041791733765. It looks like the default homebrew-core gdal was not useful for the user, but in general from source builds, and builds with options trigger their dependencies to be built from source, and lead to severe system breakage and misconfiguration for some users. And then they often go blaming homebrew.

At any rate, I'm not trying to argue here. I just wanted to point out that there may be ways to improve user experience and your experience maintaining osgeo if you're able to upstream stuff, eliminate those "what if", or "just because it's nice" options, and in general work on reducing complexity if not conforming with typical homebrew-core practices for formulae.

You're welcome to completely ignore my advice, of course, I won't take it personally.

Thanks for your hard work maintaining osgeo, and for taking the time to read my suggestions.

[fjperini]

@zbeekman @grischard @luispuerto The issue goes through experience: use and building of QGIS. As it is currently it solves most of the problems and you should only have a few things in consideration.

I agree to have a smaller version for those who require it.

We could start with this.

[grischard]

I'm very happy with the constructive way this discussion is going. Thank you all!

There are some core casks that do have dependencies - for example mactex or ibm-cloud-cli - but nowhere near as complex as this here.

I did try to look at the differences between the formulas to see how much work @zbeekman's step 4 would potentially be, but it looks like most are quite significant rewrites. @fjperini did you write down somewhere why you forked a particular formula?

[grischard]

http://blog.qgis.org/2019/07/29/introducing-new-qgis-macos-packages/ is pretty big, no?

[Emic37]

@grischard The last brew upgrade broke again my qgis installation. Since a few months, I encountered a lot of problems with these formula, despite all the efforts of @fjperini. So, when I saw that, I immediatly tried. All functionnalities I use works perfectly and the size of the app is around 1,9 Go !

So, I'm happy ! I save about 10 Go of free space using this package instead of the osgeo formulae ! And these official packages should be follow up for the update.