OSGeo / homebrew-osgeo4mac

Mac homebrew tap for maintaining a stable work environment for the OSGeo.org geospatial toolset
https://git.io/fhh3X
BSD 3-Clause "New" or "Revised" License
363 stars 111 forks source link

Formulae names & dependencies #827

Open luispuerto opened 5 years ago

luispuerto commented 5 years ago

Here are some ideas we should keep in mind about the renaming of the formulae and if we should bring dependencies into the tap. This issue is highly related to #769.

Any opinion is highly welcomed. @fjperini and specially @miguelangelperg that seems that it's suffering from "name issues".


Thoughts?

fjperini commented 5 years ago

To take into consideration: https://github.com/Homebrew/homebrew-core/issues/31510

luispuerto commented 5 years ago

I think we should also check what are they take into duplicate formulae and so. Either we ask in the issue tracker of core or here: https://discourse.brew.sh

miguelangelperg commented 5 years ago

Very timely reflection.

Beyond the difficulties involved, I especially agree with

Now, if this is the official tap of the Osgeo project, this tap must be the host of the formulas related to it

On the duplicity of formulas.

What options are not available in the core formulas? Why are they not available? What advantages do these options offer? It is worth having the formulas dulicadas to have these options, or can we wait for the problems that now prevent them to be active in the nucleus to be solved?

fjperini commented 5 years ago

Sorry for the inconvenience. But know understand that we decided to rename formulas to make other things simpler. The process took time due to the restrictions in CI and the delay of Homebrew Core in updating many of the dependencies, for example Qt and PROJ (a few hours ago). This leads to rebuild all the bottles. For this reason and another is that we were working on optimizing the tap for the future.

p/d: I will be rebuilding everything for PROJ 6.

fjperini commented 5 years ago

we are working to migrate the bottles to osgeo, bintray already caused many problems.

grischard commented 5 years ago

We need to keep the number of formulae in the tap as low as possible.

We should try to rely as much as possible into core formulae.

Yes please! I currently use osgeo4mac to install qgis. Its 294 dependencies include:

osgeo-gdal osgeo-gdal-python osgeo-hdf4 osgeo-hexer osgeo-laszip@2 osgeo-laz-perf osgeo-libgeotiff osgeo-libkml osgeo-liblas osgeo-libspatialite osgeo-matplotlib osgeo-netcdf osgeo-opencollada osgeo-openscenegraph osgeo-pcl osgeo-pdal osgeo-postgresql osgeo-proj osgeo-pyqt osgeo-qt-webkit osgeo-sip osgeo-vtk

This is quite a lot. Why does qgis want to overwrite the core postgresql, and why does qgis have a hard requirement on postgresql anyway?

Installing qgis also installs lighttpd, mysql, openldap, gtk, unbound and e2fsprogs. I don't understand why, especially the last one.

fjperini commented 5 years ago

@grischard For several reasons. Mainly to not have as many flags in the formulas and to provide more supports. In this way the user will not have to build on his machine if he requires optional supports (but required by many).

Also to have more control of when to update, there are many dependencies that are updated very late and inopportunely. And that they do not provide all their functions.

For example, for PostgreSQL the versions of homebrew-core do not work for a migration process.

We are having problems with Bintray, that's why the work stopped. Tonight we will remate everything.

p/d: yes, e2fsprogs can be removed, since it is for linux.

grischard commented 5 years ago

I'm really thankful for the qgis packages, but I feel misunderstood as an user. I don't get control of when to update, I'm married to osgeo4mac's postgresql and proj and gdal and so on. I'm currently sitting on a broken osgeo4mac qgis that can't find libproj.13.dylib in osgeo4mac's proj, and wondering if installing all these osgeo formulae won't just break something else.

Yeah, I got bitten by that postgresql migration thing myself too, but isn't the right way to go about this to fix the core?

Can qgis be built with, e.g., mysql support but not include the whole server as a dependency?

fjperini commented 5 years ago

All secondary dependencies will be on the tap. For this reason, everything will work correctly.

I'm already working to load everything again http://bottle.download.osgeo.org/

Once updated you will realize that everything works very well.

fjperini commented 5 years ago

to temporarily fix proj: ln -s libproj.15.dylib libproj.13.dylib

grischard commented 5 years ago

Yup, that gets qgis to load, thanks.

I see what you're trying to do. But the way I had to symlink libproj.15.dylib to libproj.13.dylib, the way PyQt5.QtCore seems to be missing as a dependency which produces an error when opening, the way python produces a second error when opening because it can't find the module called 'qgis', scares me.

fjperini commented 5 years ago

Just wait, it will be a few hours and everything will return to normal.

I really regret the inconvenience, but we had problems with Bintray in the middle of the tap update process.

grischard commented 5 years ago

I'll be patient then. Thank you!

fjperini commented 5 years ago

I have finished configuring everything! I will start to migrate the bottles from Bintray and update the missing ones.

luispuerto commented 5 years ago

@grischard I really don't know what is your specific problem, but as thing are right now, you can make it work qgis with a little bit of tweaking. At least it's working for me.

Anyhow.... wait a little bit and probably in the next hours everything is going to back to normal.

fjperini commented 5 years ago

During the morning I will transfer the bottles that are in bintray to osgeo.org.

After eating I will sit down to complete the updates.

Everything will return to normal, in the future we will not have these problems.

fjperini commented 5 years ago

I have already built qgis on my machine, I just need to fix something in orfeo. Later I apply all the changes. Captura de Pantalla 2019-03-31 a la(s) 12 16 04

fjperini commented 5 years ago

already building qgis! #1063

fjperini commented 5 years ago

the bottles for qgis are ready!

chinigo commented 5 years ago

Thank you so much for your hard work!

I just nuked my previous installation of qgis + dependencies and reinstalled from scratch, everything looks good.

grischard commented 5 years ago

Thank you. So I can now nuke any unlinked foo package where qgis required I install osgeo-foo?

fjperini commented 5 years ago

@grischard yes, we will use our versions for some dependencies.

luispuerto commented 5 years ago

Yeah!!! @fjperini thanks a lot for all the work these last weeks, we have a really great version of QGIS up and running thanks to you!

grischard commented 5 years ago

I'm not sure how things are supposed to be installed cleanly now. QGIS depends on ffmpeg (why?) which depends on openjpeg, which conflicts with osgeo-insighttoolkit on /usr/local/lib/pkgconfig/libopenjp2.pc. My osm2pgsql requires, directly or indirectly, postgres, gdal and proj, and qgis installs its own version. All this for QGIS. Hmm.

Is the long-term plan to have qgis in the main homebrew repository?

luispuerto commented 5 years ago

Is the long-term plan to have qgis in the main homebrew repository?

No, this is in anyway possible now with the current Homebrew core policy... QGIS is an app so there is no place for it in Core. It has to be build and has several dependencies... so it doesn't fit either in Cask.

I know that it's annoying but it's what it's

I'm not sure how things are supposed to be installed cleanly now. QGIS depends on ffmpeg (why?) which depends on openjpeg, which conflicts with osgeo-insighttoolkit on /usr/local/lib/pkgconfig/libopenjp2.pc. My osm2pgsql requires, directly or indirectly, postgres, gdal and proj, and qgis installs its own version. All this for QGIS. Hmm.

I think, @fjperini can give you a more comprehensive answer about it since he has engineered and knitted the formulae... However, I can give you some insights:

We are really open so suggestions, feedback and even more to people to participate in the tap. In other words, help is wanted!

luispuerto commented 5 years ago

PS/ in the opinion of some of us, we have to best QGIS in years using Homebrew, and I would say that it's even better than the one you can install from the official installer. Thanks to @fjperini we don't have to build anymore in our machines to have most the capabilities. This doesn't mean that it's perfect, just better. We want to make it even better 😉 and easier, but QGIS is complicated.

grischard commented 5 years ago

It's definitely a lot better than all the other methods we had before!

fjperini commented 5 years ago

Hi @grischard! The idea is to have a QGIS as complete as possible. For this reason so many dependencies are installed, ffmpeg corresponds to osgeo-grass.

Having other used formulas in the repository allows us to correct conflicts between dependencies and other problems. We could add osm2pgsql to the repository.

grischard commented 5 years ago

I just had a look at the casks policy, and couldn't find anything. It seems to be the opposite actually: mactex happily has depends_on formula: 'ghostscript'.

@fjperini yeah but do you really want to be maintaining all these formulas for just a few dependencies?

I'd be interested to know how you're handling dependencies on your installations. Currently, I can't link netcdf because it conflicts with osgeo-netcdf, and I can't remove netcdf either because it is required by osgeo-taudem, which I'll never use but qgis requires. Scratching my chin looking at my terminal.

fjperini commented 5 years ago

@grischard In my installation I have not had any problems with the dependencies. I could help you if you have problems with osgeo-netcdf.

We could choose to have a smaller version and a more complete version, something like a osgeo-qgis-suite.

Having a complete qgis is necessary, a large part of the community uses many of the tools and building everything requires time and knowledge about the problems that may arise.

luispuerto commented 5 years ago

I just had a look at the casks policy, and couldn't find anything. It seems to be the opposite actually: mactex happily has depends_on formula: 'ghostscript'.

The problem isn't exactly that... although you are going to hardly find a formula there with the amount of dependencies QGIS has. The problem is, you need to have the build done beforehand since cask isn't building just installing. In other words, the cask tap is for example for the install file you can find in the QGIS webside.

Why isn't there then a cask with that QGIS install? Well... there was for a while I think, or they try do have one, but I think the QGIS installer is too complicated for casks. (yes it's complicated)

You can know more here: https://github.com/Homebrew/homebrew-cask/pull/41883

QGIS is a conundrum, mainly because dependencies.

LeonardAukea commented 5 years ago

I've tried to fix my qgis installation. basically what I wanted to do was remove it all. Since it messes with my brew I think part of the problem is that I have postgressql & gdal for other things as well.

this is the current output of brew doctor (It looked a lot worse before):

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  proj
  libgeotiff
  libspatialite
  netcdf
  open-scene-graph

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install osgeo-pyqt osgeo-sip

Run `brew missing` for more details.

~ » brew missing                                                                                 
osgeo-pyqt-webkit: osgeo-sip osgeo-pyqt
osgeo-qgis-res: osgeo-sip osgeo-pyqt

Do you have any kind of guide as to how I would fix this due to your renamings and whatnot. What should II merge with what etc and is it meant to be done on all formulas independently?

Thank you in advance!

fjperini commented 5 years ago

@LeonardAukea you can start by uninstalling these dependencies:

  proj
  libgeotiff
  libspatialite
  netcdf
  open-scene-graph

and install these:

 osgeo-pyqt-webkit
 osgeo-sip
 osgeo-pyqt
LeonardAukea commented 5 years ago

@fjperini Ok, thanks. this is what sort of happens:

~ » brew uninstall proj                                                                                   00:03:24

Error: Refusing to uninstall /usr/local/Cellar/proj/6.0.0
because it is required by gdal, libgeotiff, liblas, libspatialite, osgeo-gdal-python, osgeo-grass, osgeo-liblas, osgeo-orfeo, osgeo-ossim, osgeo-pyspatialite, osgeo-qgis-res, osgeo-saga-lts, osgeo-taudem and pdal, which are currently installed.
You can override this and force removal with:
  brew uninstall --ignore-dependencies proj
~ »                                                                                                       00:03:36
~ » brew uninstall --ignore-dependencies proj                                                             00:03:48
Uninstalling /usr/local/Cellar/proj/6.0.0... (56 files, 13MB)
~ » brew uninstall --ignore-dependencies libgeotiff                                                       00:04:01
Uninstalling /usr/local/Cellar/libgeotiff/1.5.1... (39 files, 576.9KB)
~ » brew uninstall --ignore-dependencies libspatialite                                                    00:04:11
Uninstalling /usr/local/Cellar/libspatialite/4.3.0a_7... (31 files, 18.6MB)
~ » brew uninstall --ignore-dependencies netcdf                                                           00:04:23
Uninstalling /usr/local/Cellar/netcdf/4.6.3... (85 files, 6.2MB)
~ » brew uninstall --ignore-dependencies open-scene-graph                                                 00:04:49
Uninstalling /usr/local/Cellar/open-scene-graph/3.6.3... (8,893 files, 229.2MB)
~ » brew install osgeo-pyqt-webkit                                                                        00:06:20
Updating Homebrew...
==> Auto-updated Homebrew!
Updated 1 tap (brewsci/bio).
==> Updated Formulae
brewsci/bio/minimap2

Warning: osgeo/osgeo4mac/osgeo-pyqt-webkit 5.12.1 is already installed and up-to-date
To reinstall 5.12.1, run `brew reinstall osgeo-pyqt-webkit`
~ » brew reinstall osgeo-pyqt-webkit                                                                      00:06:56
==> Reinstalling osgeo/osgeo4mac/osgeo-pyqt-webkit
==> Installing dependencies for osgeo/osgeo4mac/osgeo-pyqt-webkit: osgeo-sip and osgeo-pyqt
==> Installing osgeo/osgeo4mac/osgeo-pyqt-webkit dependency: osgeo-sip
==> Downloading https://bottle.download.osgeo.org/osgeo-sip-4.19.15.mojave.bottle.1.tar.gz

curl: (22) The requested URL returned error: 502 Bad Gateway
Error: Failed to download resource "osgeo-sip"
Download failed: https://bottle.download.osgeo.org/osgeo-sip-4.19.15.mojave.bottle.1.tar.gz
Warning: Bottle installation failed: building from source.
==> Downloading https://www.riverbankcomputing.com/static/Downloads/sip/sip-4.19.15.tar.gz

curl: (22) The requested URL returned error: 404 Not Found
Error: An exception occurred within a child process:
  DownloadError: Failed to download resource "osgeo-sip"
Download failed: https://www.riverbankcomputing.com/static/Downloads/sip/sip-4.19.15.tar.gz

I've got similar url error codes earlier.

fjperini commented 5 years ago

@LeonardAukea solved the problem with osgeo-sip #1102

LeonardAukea commented 5 years ago

@fjperini Thanks. but something seems to be messy still. Looks like the packages without the osgeo-prefix are still a dependency to the new naming convention. Or am I mistaken?

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  osgeo-sip
  osgeo-pyqt

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libgeotiff libspatialite netcdf open-scene-graph proj

Run `brew missing` for more details.
~/ » brew missing                                                                                                                                                                                                23:11:47
gdal: proj libgeotiff libspatialite netcdf
liblas: proj libgeotiff libspatialite netcdf
osgeo-gdal-python: proj libgeotiff libspatialite netcdf
osgeo-grass: proj libgeotiff netcdf libspatialite
osgeo-liblas: proj libgeotiff libspatialite netcdf
osgeo-orfeo: netcdf proj libgeotiff open-scene-graph libspatialite
osgeo-ossim: open-scene-graph proj libgeotiff libspatialite netcdf
osgeo-pyspatialite: proj libspatialite
osgeo-qgis-res: proj libgeotiff libspatialite netcdf
osgeo-saga-lts: proj netcdf libgeotiff libspatialite
osgeo-taudem: proj libgeotiff libspatialite netcdf
pcl: netcdf
pdal: proj libgeotiff libspatialite netcdf
vtk: netcdf

Linking the packages leads to error sins e.g sip is present: should I overwrite?

And if I want to e.g reinstall osgeo-grass:

~/ » brew reinstall osgeo-grass                                                                                                                                                                                  23:18:27
osgeo-postgresql: You have other linked versions!

Unlink with brew unlink postgresql or remove with brew uninstall --ignore-dependencies postgresql

osgeo-gdal: You have other linked versions!

Unlink with brew unlink gdal or remove with brew uninstall --ignore-dependencies gdal

Error: Unsatisfied requirements failed this build.
luispuerto commented 5 years ago

@LeonardAukea

@fjperini Thanks. but something seems to be messy still. Looks like the packages without the osgeo-prefix are still a dependency to the new naming convention. Or am I mistaken?

there are tons of packages without the osgeo-* prefix that are still a dependency. We just brought to the tap the most problematic and important ones.

Warning: You have unlinked kegs in your Cellar.
Leaving kegs unlinked can lead to build-trouble and cause brews that depend on
those kegs to fail to run properly once built. Run `brew link` on these:
  osgeo-sip
  osgeo-pyqt

Warning: Some installed formulae are missing dependencies.
You should `brew install` the missing dependencies:
  brew install libgeotiff libspatialite netcdf open-scene-graph proj

Run `brew missing` for more details.
~/ » brew missing                                                                                                                                                                                                23:11:47
gdal: proj libgeotiff libspatialite netcdf
liblas: proj libgeotiff libspatialite netcdf
osgeo-gdal-python: proj libgeotiff libspatialite netcdf
osgeo-grass: proj libgeotiff netcdf libspatialite
osgeo-liblas: proj libgeotiff libspatialite netcdf
osgeo-orfeo: netcdf proj libgeotiff open-scene-graph libspatialite
osgeo-ossim: open-scene-graph proj libgeotiff libspatialite netcdf
osgeo-pyspatialite: proj libspatialite
osgeo-qgis-res: proj libgeotiff libspatialite netcdf
osgeo-saga-lts: proj netcdf libgeotiff libspatialite
osgeo-taudem: proj libgeotiff libspatialite netcdf
pcl: netcdf
pdal: proj libgeotiff libspatialite netcdf
vtk: netcdf

Linking the packages leads to error sins e.g sip is present: should I overwrite?

You can have both sips if you want, but our recommendation is to have just one... of course you can have linked just one of them and if you update osgeo-qgis is going to install, link and so on and so forth the one from this tap.

If you want to have both sips you are probably going to need to juggle a little bit with them, in other words decide which one you one linked and/or over the other and cope with the consequences.

I'm doing the same with r since I don't have the homebrew-core install on my computer. So each time I reinstall/update osgeo-qgis I have to rename that keg so osgeo-qgis install r from core, and then delete that keg and replace with the one I like.

And if I want to e.g reinstall osgeo-grass:

~/ » brew reinstall osgeo-grass                                                                                                                                                                                  23:18:27
osgeo-postgresql: You have other linked versions!

Unlink with brew unlink postgresql or remove with brew uninstall --ignore-dependencies postgresql

osgeo-gdal: You have other linked versions!

Unlink with brew unlink gdal or remove with brew uninstall --ignore-dependencies gdal

Error: Unsatisfied requirements failed this build.

With this is the same... you still have the other versions installed...

If you are using those formulae just with QGIS the recommendation is to uninstall all the other and let osgeo-qgis do it's business during the install.

Emic37 commented 5 years ago

The idea is to have a QGIS as complete as possible. For this reason so many dependencies are installed, ffmpeg corresponds to osgeo-grass.

@fjperini, a bit like @grischard I don't understand what are all these dependencies for ? I understand the idea but maybe that these formulae install a Qgis version which is too complete ? On my computer, qgis-ltr and his dependencies use 11 Go of disk space! It's considerably more than the 2Go of data installed by official advanced installer for a base qgis-ltr installation on windows computer.

Of course, it's a base installation without many add-ons but that's enough for me. For me, less is better. I would prefer can choose to install or not osgeo-saga-lts, osgeo-grass, osgeo-lastools, osgeo-taudem, r, and surely more.

Maybe formulae should be base on the official installer with similar options for more simplicity ?

And if the advanced user want use a specific add-on, he can install it using an option or simply install "optional" package.

fjperini commented 5 years ago

Ok! In the course of the day I will be doing some tests.

fjperini commented 5 years ago

We will be working, some inconveniences with proj6 and gdal3 arose. Do not update at the moment.

fjperini commented 5 years ago

@grischard is almost decided, we will use gdal 2.4.1 and proj 5.0.0. Just wait until I apply all the changes.

fjperini commented 5 years ago

Keep in mind that for osgeo-proj, osgeo-gdal, osgeo-gdal-* and osgeo-libgeotiff you will have to uninstall and reinstall. And then continue with the other updates.

In case homebrew does not update these packages

kevinstadler commented 5 years ago

While I really appreciate the ability to install QGIS from Homebrew and all the work that must have gone into this, I just want to add an enormous +1 to the request for cutting down the gigantic dependencies tree.

I understand that all the dependencies are required for building the QGIS Bottle with support for all the modules on the server, but is it really necessary to specify them as dependencies when installing the Formula on the user side? Even if QGIS was built with support for certain modules, it's still possible to run the build without most of the current dependencies (as I'm currently doing), and it would only throw up appropriate error messages once the users attempts to use a functionality that requires other libraries. You say that a large part of the community uses many of the tools, but I doubt that even a power user ever makes use of even half of those dependencies. Things like the full postgresql and mysql servers aren't actually required by anyone who doesn't know how to install them themselves (apart from the fact that the dependency should really only be some (client) header files and not the full server formula, no?).

I gave up full installation about half way through the dependencies, then just installed with --ignore-dependencies and now have a seemingly fully functional QGIS running. If you want I can try to figure out where in the dependency tree I aborted the installs so that other users can also make use of a much smaller installation this way.

fjperini commented 5 years ago

@kevinstadler we will create an option for those that decide a simpler installation, first we are solving other problems with proj and gdal.

Just wait for the process to complete.

grischard commented 5 years ago

Hurray to the option for a simpler installation with less dependencies. Thank you @fjperini for all your hard work!

kevinstadler commented 5 years ago

Thanks so much! Just one more slightly unrelated question, is there a specific formula for installing QGIS with Python plugin support? For me the plugin menu just says "no Python support detected" even though I'm pretty sure I have all necessary dependencies, and I've also manually put a plugin in the profile/default/python/plugins folder but it's not showing up. I'm using the osgeo-qgis (not LTR) formula right now.

fjperini commented 5 years ago

@kevinstadler It should work. You could share the plugin that you put so I try it.

kevinstadler commented 5 years ago

@fjperini thanks for your help! It's the profiletool plugin: https://plugins.qgis.org/plugins/profiletool/ I've put the extracted profiletool/ folder into what I think is the appropriate subfolder in the active QGIS profile: /Users/kevin/Library/Application Support/QGIS/QGIS3/profiles/default/python/plugins/

The python/ folder already existed but only had an expressions/ subdirectory, so I had to create the plugins/ one myself.

fjperini commented 5 years ago

@kevinstadler After updating QGIS I will check this.

fjperini commented 5 years ago

@kevinstadler I have checked this and the plugin works correctly.

cd /usr/local/Cellar/osgeo-qgis/3.6.3/QGIS.app/Contents/Resources/python/plugins/

git clone https://github.com/PANOimagen/profiletool
Captura de Pantalla 2019-05-23 a la(s) 07 11 19