Missing _scproxy module from macOS?

[kellyjonbrazil]

Hi there!

I have been using pyoxidizer on macOS, linux, and Windows for a couple years now with no major issues. I recently tried to build a new version of my software on macOS and the binary compiles, but I get the following error when running:

Traceback (most recent call last):
  File "jc.parsers.xml", line 77, in <module>
  File "xmltodict", line 8, in <module>
  File "xml.sax.saxutils", line 6, in <module>
  File "urllib.request", line 2627, in <module>
ModuleNotFoundError: No module named '_scproxy'

It seems that the xmltodict library depends on urllib, which depends on _scproxy which can't be found. I have not run into this before even though I've been using xmltodict for years. This also still works fine on linux and windows.

I thought maybe there was some macOS update that broke my xcode environment so I upgraded macOS and xcode, but I still get the same message.

Could this be a problem with the provided python interpreter bundles for macOS? I'm not sure how to troubleshoot this any further.

I'm on PyOxidizer 0.23.0 but this also happened on PyOxidizer 0.22.0. Again, the weird thing is that this worked just fine not long ago on v0.22.0 and below.

% pyoxidizer -V
PyOxidizer 0.23.0

p.s. are any of these lines of any interest in the compilation output?

target Python distribution for x86_64-apple-darwin resolves to: https://github.com/indygreg/python-build-standalone/releases/download/20221106/cpython-3.10.8%2B20221106-x86_64-apple-darwin-pgo-full.tar.zst (sha256=c56a4728bdf498e3a4351906474811eafe0da29d811d0f5097d1f01e1fa7017c)
...
locating Apple SDK macosx11.1+ supporting macosx10.9
using MacOSX (version: 11.1) SDK at /Library/Developer/CommandLineTools/SDKs/MacOSX11.1.sdk targeting macosx10.9

[kellyjonbrazil]

I took a peek inside the python distribution downloaded by PyOxidizer (cpython-3.10.8+20221106-x86_64-apple-darwin-pgo-full.tar) and was able to find _scproxy.o in the build/Modules directory:

python/build/Modules/_scproxy.o

But I don't see any _scproxy module under python/install. Is that a clue or am I barking up the wrong tree?

[kellyjonbrazil]

I'm noticing that even older builds of my application are breaking on the missing _scproxy module, but it wasn't as apparent before without specific testing. A change in my application made the issue more visible.

I'll keep checking to see if I can find a build that works and if I can find any correlation with the python distribution.

[kellyjonbrazil]

For reference, here are _scproxy references inside the PYTHON.json file:

# json.build_info.extensions._scproxy = []
"_scproxy": [
        {
          "in_core": false,
          "init_fn": "PyInit__scproxy",
          "links": [
            {
              "framework": true,
              "name": "CoreFoundation"
            },
            {
              "framework": true,
              "name": "SystemConfiguration"
            }
          ],
          "objs": [
            "build/Modules/_scproxy.o"
          ],
          "required": false,
          "variant": "default"
        }
      ],

# json.python_config_vars
MODBUILT_NAMES = "array  cmath  math  _contextvars  _struct  _weakref  _testinternalcapi  _random  _pickle  _datetime  _zoneinfo  _bisect  _heapq  _asyncio  _statistics  unicodedata  fcntl  grp  select  mmap  _csv  _socket  termios  resource  _posixsubprocess  audioop  _md5  _sha1  _sha256  _sha512  _sha3  _blake2  syslog  binascii  zlib  _multibytecodec  _codecs_cn  _codecs_hk  _codecs_iso2022  _codecs_jp  _codecs_kr  _codecs_tw  _bz2  _crypt  _curses  _curses_panel  _ctypes  _ctypes_test  _decimal  _dbm  _elementtree  _hashlib  _json  _lsprof  _lzma  _multiprocessing  _opcode  _posixshmem  _queue  _scproxy  _sqlite3  _ssl  _testbuffer  _testimportmultiple  _testinternalcapi  _testmultiphase  _tkinter  _uuid  _xxsubinterpreters  _xxtestfuzz  pyexpat  readline  posix  errno  pwd  _sre  _codecs  _weakref  _functools  _operator  _collections  _abc  itertools  atexit  _signal  _stat  time  _thread  _locale  _io  faulthandler  _tracemalloc  _symtable  xxsubtype"
MODOBJS = "Modules/_abc.o Modules/_asynciomodule.o Modules/_bisectmodule.o Modules/_bz2module.o Modules/_codecs_cn.o Modules/_codecs_hk.o Modules/_codecs_iso2022.o Modules/_codecs_jp.o Modules/_codecs_kr.o Modules/_codecs_tw.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/_contextvarsmodule.o Modules/_cryptmodule.o Modules/_csv.o Modules/_ctypes.o Modules/_ctypes_test.o Modules/_curses_panel.o Modules/_cursesmodule.o Modules/_datetimemodule.o Modules/_dbmmodule.o Modules/_decimal.o Modules/_elementtree.o Modules/_functoolsmodule.o Modules/_hashopenssl.o Modules/_heapqmodule.o Modules/_iomodule.o Modules/_json.o Modules/_localemodule.o Modules/_lsprof.o Modules/_lzmamodule.o Modules/_math.o Modules/_opcode.o Modules/_operator.o Modules/_pickle.o Modules/_posixsubprocess.o Modules/_queuemodule.o Modules/_randommodule.o Modules/_scproxy.o Modules/_sre.o Modules/_ssl.o Modules/_stat.o Modules/_statisticsmodule.o Modules/_struct.o Modules/_testbuffer.o Modules/_testimportmultiple.o Modules/_testinternalcapi.o Modules/_testmultiphase.o Modules/_threadmodule.o Modules/_tkinter.o Modules/_tracemalloc.o Modules/_uuidmodule.o Modules/_weakref.o Modules/_xxsubinterpretersmodule.o Modules/_xxtestfuzz.o Modules/_zoneinfo.o Modules/arraymodule.o Modules/atexitmodule.o Modules/audioop.o Modules/basearith.o Modules/binascii.o Modules/blake2b_impl.o Modules/blake2module.o Modules/blake2s_impl.o Modules/bufferedio.o Modules/bytesio.o Modules/cache.o Modules/callbacks.o Modules/callproc.o Modules/cfield.o Modules/cmathmodule.o Modules/connection.o Modules/constants.o Modules/context.o Modules/convolute.o Modules/crt.o Modules/cursor.o Modules/difradix2.o Modules/dlfcn_simple.o Modules/errnomodule.o Modules/faulthandler.o Modules/fcntlmodule.o Modules/fileio.o Modules/fnt.o Modules/fourstep.o Modules/fuzzer.o Modules/grpmodule.o Modules/io.o Modules/iobase.o Modules/itertoolsmodule.o Modules/malloc_closure.o Modules/mathmodule.o Modules/md5module.o Modules/microprotocols.o Modules/mmapmodule.o Modules/module.o Modules/mpalloc.o Modules/mpdecimal.o Modules/multibytecodec.o Modules/multiprocessing.o Modules/numbertheory.o Modules/posixmodule.o Modules/posixshmem.o Modules/prepare_protocol.o Modules/pwdmodule.o Modules/pyexpat.o Modules/readline.o Modules/resource.o Modules/rotatingtree.o Modules/row.o Modules/selectmodule.o Modules/semaphore.o Modules/sha1module.o Modules/sha256module.o Modules/sha3module.o Modules/sha512module.o Modules/signalmodule.o Modules/sixstep.o Modules/socketmodule.o Modules/statement.o Modules/stgdict.o Modules/stringio.o Modules/symtablemodule.o Modules/syslogmodule.o Modules/termios.o Modules/textio.o Modules/timemodule.o Modules/tkappinit.o Modules/transpose.o Modules/unicodedata.o Modules/util.o Modules/xmlparse.o Modules/xmlrole.o Modules/xmltok.o Modules/xxsubtype.o Modules/zlibmodule.o"

[kellyjonbrazil]

And here is my pyoxidizer.bzl file. I notice that I'm usingpolicy.extension_module_filter = "no-libraries" to keep the package size down. I'll see if changing the value to "all" fixes the issue.

def make_dist():
    return default_python_distribution()

def resource_callback(policy, resource):
    if type(resource) == "PythonModuleSource":
        resource.add_include = True

def make_exe(dist):
    policy = dist.make_python_packaging_policy()

    # use the callback function to filter out source code
    # see: https://pyoxidizer.readthedocs.io/en/v0.9.0/packaging_resources.html#using-callbacks-to-influence-resource-attributes
    # and: https://github.com/indygreg/PyOxidizer/issues/312
    policy.register_resource_callback(resource_callback)

    # Package all available Python extensions in the distribution.
    # policy.extension_module_filter = "all"
    policy.extension_module_filter = "no-libraries"

    # Use in-memory location for adding resources by default.
    policy.resources_location = "in-memory"

    # Attempt to add resources relative to the built binary when
    # `resources_location` fails.
    policy.resources_location_fallback = "filesystem-relative:jclib"

    # Remove source files from dist to reduce executable size (keep docstrings)
    policy.include_distribution_sources = False
    policy.include_non_distribution_sources = False
    policy.bytecode_optimize_level_zero = False
    policy.bytecode_optimize_level_one = True
    policy.bytecode_optimize_level_two = False

    # This variable defines the configuration of the embedded Python
    # interpreter. By default, the interpreter will run a Python REPL
    # using settings that are appropriate for an "isolated" run-time
    # environment.
    #
    # The configuration of the embedded Python interpreter can be modified
    # by setting attributes on the instance. Some of these are
    # documented below.
    python_config = dist.make_python_interpreter_config()

    # Remove source for smaller executable (keep docstrings)
    python_config.optimization_level = 1

    # Enable the standard path-based importer which attempts to load
    # modules from the filesystem.
    python_config.filesystem_importer = True

    # Evaluate a string as Python code when the interpreter starts.
    python_config.run_command = "import jc.cli; jc.cli.main()"

    # Produce a PythonExecutable from a Python distribution, embedded
    # resources, and other options. The returned object represents the
    # standalone executable that will be built.
    exe = dist.to_python_executable(
        name="jc",
        packaging_policy=policy,
        config=python_config,
    )

    # Invoke `pip install` using a requirements file and add the collected resources
    # to our binary.
    exe.add_python_resources(exe.pip_install(["-r", "requirements.txt"]))

    # Return our `PythonExecutable` instance so it can be built and
    # referenced by other consumers of this target.
    return exe

def make_install(exe):
    # Create an object that represents our installed application file layout.
    files = FileManifest()

    # Add the generated executable to our install layout in the root directory.
    files.add_python_resource(".", exe)

    return files

# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_dist)
register_target("exe", make_exe, depends=["dist"])
register_target("install", make_install, depends=["exe"], default=True)

# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()

# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.

PYOXIDIZER_VERSION = "0.16.0"
PYOXIDIZER_COMMIT = "4053178f2ba11d29f497d171289cb847cd07ed77"

[kellyjonbrazil]

Cool - changing policy.extension_module_filter from "no-libraries" to "all" seems to have fixed the issue.

It looks like this was only needed on macOS and not linux or Windows since the _scproxy package is a macOS-specific library.