`brew install gdal` succeeds but does not install all expected modules, as of a recent version update

[sheffe]

brew gist-logs <formula> link OR brew config AND brew doctor output

`brew config`: 
HOMEBREW_VERSION: 4.0.9
ORIGIN: https://github.com/Homebrew/brew
HEAD: 584588afb4ab557bcc0536f2794b4d043c333bb9
Last commit: 5 days ago
Core tap JSON: 23 Mar 19:48 UTC
HOMEBREW_PREFIX: /opt/homebrew
HOMEBREW_CASK_OPTS: []
HOMEBREW_DISPLAY: /private/tmp/com.apple.launchd.BF1aI64Q5Y/org.macosforge.xquartz:0
HOMEBREW_MAKE_JOBS: 10
Homebrew Ruby: 2.6.10 => /System/Library/Frameworks/Ruby.framework/Versions/2.6/usr/bin/ruby
CPU: 10-core 64-bit arm_firestorm_icestorm
Clang: 13.1.6 build 1316
Git: 2.32.1 => /Applications/Xcode.app/Contents/Developer/usr/bin/git
Curl: 7.79.1 => /usr/bin/curl
macOS: 12.6.3-arm64
CLT: 14.2.0.0.1.1668646533
Xcode: 13.4
Rosetta 2: false

`brew doctor`:
Your system is ready to brew.

Verification

• [X] My "brew doctor output" says Your system is ready to brew. and am still able to reproduce my issue.
• [X] I ran brew update and am still able to reproduce my issue.
• [X] I have resolved all warnings from brew doctor and that did not fix my problem.

What were you trying to do (and why)?

Trying to run brew install gdal -- gdal is a foundational toolkit for geospatial data work. brew install gdal for GDAL version 3.6.0 produced the expected installation result, but 3.6.2 and 3.6.3 do not. No errors occur, but the more recent installations are missing a significant number of tools in /opt/homebrew/bin after the installation.

The problem occurs in the installation of GDAL-provided python scripts like "gdal_proximity.py" that should be installed to /opt/homebrew/bin; this did occur as of 3.6.0 but no longer happens as of 3.6.2 and 3.6.3. GDAL's lead developer indicates that this is not a deliberate change by GDAL, and may be rooted in how brew inspects and uses GDAL's swig/python/CMakeLists.txt.

I have reproduced this on my own machine with a clean installation of homebrew / gdal, and verified the behavior change occurs after GDAL 3.6.0 by reproducing this on several friends' machines.

What happened (include all command output)?

"What happened?" -- I see a perfectly normal installation with no errors from the Homebrew side, with command output below. However, GDAL tools that were installed to /opt/homebrew/bin as of 3.6.0 are not available after the installation of 3.6.2 or 3.6.3. (All command output below is with respect to 3.6.3.)

➜ ~ brew install gdal ==> Fetching gdal ==> Downloading https://ghcr.io/v2/homebrew/core/gdal/manifests/3.6.3 Already downloaded: /Users/jsheffield/Library/Caches/Homebrew/downloads/2f89eeb3321573c695f20c415bee88ccb479a55e9f806b1d7ddb779852f43d6d--gdal-3.6.3.bottle_manifest.json ==> Downloading https://ghcr.io/v2/homebrew/core/gdal/blobs/sha256:ffeaa3544f3412c5c9b3419bc345f1b51b622e861705eccabbeba67767f43821 Already downloaded: /Users/jsheffield/Library/Caches/Homebrew/downloads/32889f5a302ecbb98716bdaef063082eefd7a5297499c3fbd7eb8b63f9fb6e96--gdal--3.6.3.arm64_monterey.bottle.tar.gz ==> Pouring gdal--3.6.3.arm64_monterey.bottle.tar.gz 🍺 /opt/homebrew/Cellar/gdal/3.6.3: 448 files, 26.9MB ==> Running brew cleanup gdal... Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP. Hide these hints with HOMEBREW_NO_ENV_HINTS (see man brew).

When I run ls /opt/homebrew/bin | grep gdal after the successful 3.6.3 installation, I see the following: gdal-config gdal_contour gdal_create gdal_grid gdal_rasterize gdal_translate gdal_viewshed gdaladdo gdalbuildvrt gdaldem gdalenhance gdalinfo gdallocationinfo gdalmanage gdalmdiminfo gdalmdimtranslate gdalsrsinfo gdaltindex gdaltransform gdalwarp

What did you expect to happen?

After a new installation of GDAL, below is the correct output of ls /opt/homebrew/bin | grep gdal that I would expect (using 3.6.0). All scripts ending ".py" are missing after brew install gdal for 3.6.3 but available for 3.6.0.

gdal-config gdal2tiles.py gdal2xyz.py gdal_calc.py gdal_contour gdal_create gdal_edit.py gdal_fillnodata.py gdal_grid gdal_merge.py gdal_pansharpen.py gdal_polygonize.py gdal_proximity.py gdal_rasterize gdal_retile.py gdal_sieve.py gdal_translate gdal_viewshed gdaladdo gdalattachpct.py gdalbuildvrt gdalcompare.py gdaldem gdalenhance gdalinfo gdallocationinfo gdalmanage gdalmdiminfo gdalmdimtranslate gdalmove.py gdalsrsinfo gdaltindex gdaltransform gdalwarp

Step-by-step reproduction instructions (by running brew commands)

brew install gdal #installs 3.6.3 successfully but without any of the Python tools that should be installed.

[carlocab]

Not sure how that happened, since this is all that changed in the update to 3.6.3: https://github.com/Homebrew/homebrew-core/commit/2c120e249d79576f2d7ea47f3482cdc5a9730805

[sheffe]

Thanks Carlo. I could verify that GDAL was fine in 3.6.0 but the problem exists in both 3.6.2 and 3.6.3. Digging with your link as example, I think it's more likely the issue occurred starting in 3.6.1: I think I've found the right PR #118716 here (but not sure). The PR for 3.6.2 looks minimal/much like your example for 3.6.3.

(With apologies, I'm not sure how to reproduce this myself on the older versions to verify.)

[carlocab]

Can you try this patch do see if it fixes things?

diff --git a/Formula/gdal.rb b/Formula/gdal.rb
index 0c1b9df6a29..fca2cf8e8c4 100644
--- a/Formula/gdal.rb
+++ b/Formula/gdal.rb
@@ -79,18 +79,21 @@ class Gdal < Formula
   end

   def install
+    site_packages = prefix/Language::Python.site_packages(python3)
     # Work around Homebrew's "prefix scheme" patch which causes non-pip installs
     # to incorrectly try to write into HOMEBREW_PREFIX/lib since Python 3.10.
     inreplace "swig/python/CMakeLists.txt",
-              /(set\(INSTALL_ARGS "--single-version-externally-managed --record=record.txt")\)/,
-              "\\1 --install-lib=#{prefix/Language::Python.site_packages(python3)})"
+              'set(INSTALL_ARGS "--single-version-externally-managed --record=record.txt',
+              "\\0 --install-lib=#{site_packages} --install-scripts=#{bin}"

+    osgeo_ext = site_packages/"osgeo"
     # keep C++ standard in sync with `abseil.rb`
     args = %W[
       -DENABLE_PAM=ON
       -DBUILD_PYTHON_BINDINGS=ON
-      -DCMAKE_INSTALL_RPATH=#{lib}
+      -DCMAKE_INSTALL_RPATH=#{rpath};#{rpath(source: osgeo_ext)}
       -DPython_EXECUTABLE=#{which(python3)}
+      -DGDAL_PYTHON_INSTALL_LIB=#{site_packages}
       -DCMAKE_CXX_STANDARD=17
     ]

@@ -100,6 +103,8 @@ class Gdal < Formula
     system "cmake", "-S", ".", "-B", "build", *args, *std_cmake_args
     system "cmake", "--build", "build"
     system "cmake", "--install", "build"
+
+    bash_completion.install (share/"bash-completion/completions").children
   end

   test do

You'll need to

brew tap homebrew/core
cd "$(brew --repository homebrew/core)"
# Apply patch. e.g. Copy the diff above and do `pbpaste | patch -p1`
HOMEBREW_NO_INSTALL_FROM_API=1 brew reinstall --build-from-source gdal

Once you've done that, you can

brew untap homebrew/core

Note that the scripts you're looking for appear to still be available; they're just not in the bin directory. See, e.g.,

find "$(brew --prefix gdal)" -name "gdal_*.py"

I didn't check all of the missing ones, however.

[carlocab]

Note: there was a small issue with the patch above. I've now updated it.

[carlocab]

Oops, edited it again. I think that should work now.

[carlocab]

Should be fixed by #126704.

[carlocab]

Should be fixed now. Do

brew update && brew reinstall gdal

in about half an hour.

[sheffe]

Thank you @carlocab , confirmed to work on my machine this morning. Appreciate the help.