Anaconda, mod_wsgi and arm64 architecture

[genghisken]

Hi Graham

I fear this is a very frequently asked question and with some repetition of #824. I have to switch to the M3 Arm-64 platform (Sonoma) and I cannot get mod_wsgi installed and running. The install looks like it worked but I get the dreaded message below. If this is in an FAQ, please feel free to direct me towards it!

[Thu Aug 08 00:03:08.987260 2024] [so:error] [pid 290:tid 8508444480] AH06665: No code signing authority for module at /Users/kws/anaconda3/envs/psat_panstarrs38/lib/libpython3.8.dylib specified in LoadModule directive.
httpd: Syntax error on line 160 of /tmp/panstarrs/httpd.conf: Code signing absent - not loading module at: /Users/kws/anaconda3/envs/psat_panstarrs38/lib/libpython3.8.dylib

I've tried signing the code (which complains that it's already signed - hence removed the existing signature and added my own), but specifying the certificate after the LoadModule statement in the generated httpd.conf achieves nothing. It still complains that the module is unsigned.

So, then I uninstalled it and tried to installed mod_wsgi-httpd or mod_wsgi-standalone. Now I get the same compilation errors. If only I could see where the code lives, I could probably fix the cast error for pid_t, etc.

      In file included from encoding/apr_encode.c:28:
      In file included from ./include/apr_encode.h:24:
      ./include/apr.h:561:2: error: Can not determine the proper size for pid_t
      #error Can not determine the proper size for pid_t
       ^
      In file included from encoding/apr_encode.c:30:
      In file included from ./include/apr_strings.h:52:
      ./include/apr_want.h:94:8: error: redefinition of 'iovec'
      struct iovec
             ^
      /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/sys/_types/_iovec_t.h:31:8: note: previous definition is here
      struct iovec {
             ^
      2 errors generated.
      make[1]: *** [encoding/apr_encode.lo] Error 1
      make: *** [all-recursive] Error 1
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-install-ejrobamv/mod-wsgi-httpd_c6473afbb1af42769b3f92f2d2719b44/setup.py", line 75, in <module>
          raise RuntimeError('Failed to build APR.')
      RuntimeError: Failed to build APR.
      [end of output]

So I'm stuck from here. Is mod_wsgi simply now not compatible with Mac / Arm-64? Do I need to move my development platform off Sonoma?

[GrahamDumpleton]

On macOS you cannot use the system supplied Apache httpd, or even build Apache httpd yourself from ASF (Apache Software Foundation) source code, as the APR libraries distributed by the ASF don't support the last few macOS versions and have not done so for quite a while.

Your only option on macOS is to use Apache httpd from HomeBrew and use gcc from HomeBrew as well, ignoring the system macOS versions.

[GrahamDumpleton]

On top of that, you also can't use macOS supplied version of Apache httpd due to the code signing issues where they require Apache modules to be signed. I don't know what is required to sign the module once compiled and no one has been able to explain it to me either.

[genghisken]

Yeah, many thanks. I've gone through several tutorials re: signing modules, but modifying the generated httpd.conf to add the certificate name after the module in the LoadModule command is simply not recognised. (I get the same not signed error.)

I'm also curious to know where my original mod_wsgi arm64 python wheel came from in the first place. If I try to reinstall with --no-cache-dir I get the following...

(psat_panstarrs38) kws::MacBook-Pro { /tmp/panstarrs }-> pip install mod_wsgi --no-cache-dir
Collecting mod_wsgi
  Downloading mod_wsgi-5.0.0.tar.gz (497 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 497.7/497.7 kB 7.9 MB/s eta 0:00:00
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [57 lines of output]

      WARNING: The Python installation you are using does not appear to have
      been installed with a shared library, or in the case of MacOS X, as a
      framework. Where these are not present, the compilation of mod_wsgi may
      fail, or if it does succeed, will result in extra memory being used by
      all processes at run time as a result of the static library needing to
      be loaded in its entirety to every process. It is highly recommended
      that you reinstall the Python installation being used from source code,
      supplying the '--enable-shared' option to the 'configure' script when
      configuring the source code prior to building and installing it.

      running egg_info
      creating /private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info
      writing /private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/PKG-INFO
      writing dependency_links to /private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/dependency_links.txt
      writing entry points to /private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/entry_points.txt
      writing top-level names to /private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/top_level.txt
      writing manifest file '/private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/SOURCES.txt'
      reading manifest file '/private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-pip-egg-info-l_xq431h/mod_wsgi.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no files found matching 'docs/_build/html/_static/*'
      warning: no files found matching 'docs/_build/html/_static/css/*'
      warning: no files found matching 'docs/_build/html/_static/font/*'
      warning: no files found matching 'docs/_build/html/_static/js/*'
      warning: no files found matching 'docs/_build/html/_sources/*'
      warning: no files found matching 'pyproject.toml'
      adding license file 'LICENSE'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-install-tano55ib/mod-wsgi_4f4f421b6c4c4f5eacefc609d691fae2/setup.py", line 416, in <module>
          setup(name = package_name,
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 310, in run
          self.find_sources()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 318, in find_sources
          mm.run()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 544, in run
          self.prune_file_list()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 610, in prune_file_list
          base_dir = self.distribution.get_fullname()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
          return _distribution_fullname(self.get_name(), self.get_version())
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
          canonicalize_version(version, strip_trailing_zero=False),
      TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'

[GrahamDumpleton]

If you can, avoid using Anaconda Python. They keep breaking the ability to embed Python in other applications and even if you can get it to build, not guaranteed it will work. Thus also suggest using Python from HomeBrew.

Am not really sure why getting the error with Anaconda Python, but do be aware you are using a quite old version of it, ie., Python 3.8. Are you able to at least update to newer Anaconda Python?

[genghisken]

Incidentally mod_wsgi works fine on my old Intel Mac, which is also running MacOS Sonoma!

Is it possible to get my Mac to defer to using my homebrew install of httpd and gcc when installing mod_wsgi-httpd or mod-wsgi-standalone. Or do I have to revert to a manual config? Do I need to set CFLAGS or LDFLAGS, etc to force the issue?

[genghisken]

Yes, I can update - I realise it's a very old python. I was trying to replicate as near as possible my Mac intel dev environment first, but I'll try upgrading. Everything else works - mod_wsgi is my only remaining module to fix. (I use python 3.11 everywhere else, so I'll try that.)

Sorry - Anaconda is our default dev python. I'll not be switching unless there's a major showstopper in the near future. I'll switch to using nginx if I have to!

Many thanks for the speedy reponses!

[GrahamDumpleton]

When using mod_wsgi-httpd or mod-wsgi-standalone it it is relying on building from Apache httpd source code, which as already mentioned will fail on macOS due to APR not being updated by the ASF. Although that said, it seems there was a release of APR back at April 16, 2023. It has been so long since I have done anything with mod_wsgi and checked whether they had fixed it is possible they may have as mod_wsgi-httpd is still using api-1.7.0, where as latest is now 1.7.4. Let me see if I can get it working with the later version.

[genghisken]

Thanks again for your engagement with this. mod_wsgi has been the solid backbone of our webserver code and I'll be gutted if I need to move my dev platform to a remote Ubuntu server without at least making an effort to resolve the ongoing issues. Not sure what time zone you're in, but I need to go offline for about 6 hours now! (I'm on UK time.)

[GrahamDumpleton]

Am in Australia, so morning, but also got a lot of other work have to do. :-(

Isn't a straight forward change to mod_wsgi-httpd due to demise of setuptools. So have to update setup.py file to whatever supposed to use now. Will see if can still sort something out.

[GrahamDumpleton]

Okay, from Python 3.12 you have to install setuptools yourself.

[GrahamDumpleton]

Seems later versions do work and can get mod_wsgi-httpd working again on macOS. Only took them a couple of years to fix APR. 🤷‍♂️

[GrahamDumpleton]

My internet connection had a lot of unexpected problems today due to a botched service transfer, so haven't managed to update mod_wsgi-standalone package yet as need to also roll some other mod_wsgi changes in as well. What I have been able to do though is update mod_wsgi-httpd, so try the following:

pip uninstall mod_wsgi-standalone
pip uninstall mod_wsgi
pip uninstall mod_wsgi-httpd

Keep running until it says none of the packages are installed.

Then (preferably using a virtual environment), run:

pip install mod_wsgi-httpd

Then run:

pip install mod_wsgi

When installing mod_wsgi package it should detect that mod_wsgi-httpd was installed and use it.

This is the long handed way that works instead of installing mod_wsgi-standalone package.

[genghisken]

Still persisting with my ancient python 3.8 install the pip install mod_wsgi-httpd step fell over at this stage:

      adding license file 'LICENSE'
      Traceback (most recent call last):
        File "<string>", line 2, in <module>
        File "<pip-setuptools-caller>", line 34, in <module>
        File "/private/var/folders/b5/cnlc_xr1187_pb1q0xrg5qxc0000gp/T/pip-install-798v6fxw/mod-wsgi-httpd_cdf813f943834c4198bf97a325c479ed/setup.py", line 148, in <module>
          setup(name = 'mod_wsgi-httpd',
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/__init__.py", line 108, in setup
          return distutils.core.setup(**attrs)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 970, in run_commands
          self.run_command(cmd)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/dist.py", line 945, in run_command
          super().run_command(command)
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_distutils/dist.py", line 989, in run_command
          cmd_obj.run()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 310, in run
          self.find_sources()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 318, in find_sources
          mm.run()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 544, in run
          self.prune_file_list()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/command/egg_info.py", line 610, in prune_file_list
          base_dir = self.distribution.get_fullname()
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_core_metadata.py", line 266, in get_fullname
          return _distribution_fullname(self.get_name(), self.get_version())
        File "/Users/kws/anaconda3/envs/psat_panstarrs38/lib/python3.8/site-packages/setuptools/_core_metadata.py", line 284, in _distribution_fullname
          canonicalize_version(version, strip_trailing_zero=False),
      TypeError: canonicalize_version() got an unexpected keyword argument 'strip_trailing_zero'

Hang on while I upgrade my whole environment to python 3.11, which I should have done in the first place. (Installation of mod_wsgi seemed to work, though I haven't tested it yet.) I have some reliance on setuptools for my own package, so I'll delay upgrading to 3.12 until I've got that fixed!

[GrahamDumpleton]

Need to just search around to try and find a solution.

• https://github.com/pypa/setuptools/issues/4496

What version of setuptools are you using? Is it the latest version or an older version?

[GrahamDumpleton]

Can you pin setuptools to be <71.

[genghisken]

Amazing. Pinning setuptools to 70 works fine! Many thanks for your help!! My python 3.8 environment is now functional.

And also - many apologies for wasting your time. It turns out that upgrading to python 3.11, mod_wsgi (no standalone or httpd required) and Anaconda work together out of the box. No code signing, etc. All a good exercise for me, but not sure it's been useful for you.

[GrahamDumpleton]

Not a waste of time as I have been sitting on needing to make some mod_wsgi changes for a while. Wasn't the best day for me to decide to do something about it with some big internet access issues and also needing to finish a release of a different project at same time but has worked out okay so far. :-)