Closed kellyjonbrazil closed 1 year ago
I took a peek inside the python distribution downloaded by PyOxidizer (cpython-3.10.8+20221106-x86_64-apple-darwin-pgo-full.tar
) and was able to find _scproxy.o
in the build/Modules
directory:
python/build/Modules/_scproxy.o
But I don't see any _scproxy
module under python/install
. Is that a clue or am I barking up the wrong tree?
I'm noticing that even older builds of my application are breaking on the missing _scproxy
module, but it wasn't as apparent before without specific testing. A change in my application made the issue more visible.
I'll keep checking to see if I can find a build that works and if I can find any correlation with the python distribution.
For reference, here are _scproxy
references inside the PYTHON.json
file:
# json.build_info.extensions._scproxy = []
"_scproxy": [
{
"in_core": false,
"init_fn": "PyInit__scproxy",
"links": [
{
"framework": true,
"name": "CoreFoundation"
},
{
"framework": true,
"name": "SystemConfiguration"
}
],
"objs": [
"build/Modules/_scproxy.o"
],
"required": false,
"variant": "default"
}
],
# json.python_config_vars
MODBUILT_NAMES = "array cmath math _contextvars _struct _weakref _testinternalcapi _random _pickle _datetime _zoneinfo _bisect _heapq _asyncio _statistics unicodedata fcntl grp select mmap _csv _socket termios resource _posixsubprocess audioop _md5 _sha1 _sha256 _sha512 _sha3 _blake2 syslog binascii zlib _multibytecodec _codecs_cn _codecs_hk _codecs_iso2022 _codecs_jp _codecs_kr _codecs_tw _bz2 _crypt _curses _curses_panel _ctypes _ctypes_test _decimal _dbm _elementtree _hashlib _json _lsprof _lzma _multiprocessing _opcode _posixshmem _queue _scproxy _sqlite3 _ssl _testbuffer _testimportmultiple _testinternalcapi _testmultiphase _tkinter _uuid _xxsubinterpreters _xxtestfuzz pyexpat readline posix errno pwd _sre _codecs _weakref _functools _operator _collections _abc itertools atexit _signal _stat time _thread _locale _io faulthandler _tracemalloc _symtable xxsubtype"
MODOBJS = "Modules/_abc.o Modules/_asynciomodule.o Modules/_bisectmodule.o Modules/_bz2module.o Modules/_codecs_cn.o Modules/_codecs_hk.o Modules/_codecs_iso2022.o Modules/_codecs_jp.o Modules/_codecs_kr.o Modules/_codecs_tw.o Modules/_codecsmodule.o Modules/_collectionsmodule.o Modules/_contextvarsmodule.o Modules/_cryptmodule.o Modules/_csv.o Modules/_ctypes.o Modules/_ctypes_test.o Modules/_curses_panel.o Modules/_cursesmodule.o Modules/_datetimemodule.o Modules/_dbmmodule.o Modules/_decimal.o Modules/_elementtree.o Modules/_functoolsmodule.o Modules/_hashopenssl.o Modules/_heapqmodule.o Modules/_iomodule.o Modules/_json.o Modules/_localemodule.o Modules/_lsprof.o Modules/_lzmamodule.o Modules/_math.o Modules/_opcode.o Modules/_operator.o Modules/_pickle.o Modules/_posixsubprocess.o Modules/_queuemodule.o Modules/_randommodule.o Modules/_scproxy.o Modules/_sre.o Modules/_ssl.o Modules/_stat.o Modules/_statisticsmodule.o Modules/_struct.o Modules/_testbuffer.o Modules/_testimportmultiple.o Modules/_testinternalcapi.o Modules/_testmultiphase.o Modules/_threadmodule.o Modules/_tkinter.o Modules/_tracemalloc.o Modules/_uuidmodule.o Modules/_weakref.o Modules/_xxsubinterpretersmodule.o Modules/_xxtestfuzz.o Modules/_zoneinfo.o Modules/arraymodule.o Modules/atexitmodule.o Modules/audioop.o Modules/basearith.o Modules/binascii.o Modules/blake2b_impl.o Modules/blake2module.o Modules/blake2s_impl.o Modules/bufferedio.o Modules/bytesio.o Modules/cache.o Modules/callbacks.o Modules/callproc.o Modules/cfield.o Modules/cmathmodule.o Modules/connection.o Modules/constants.o Modules/context.o Modules/convolute.o Modules/crt.o Modules/cursor.o Modules/difradix2.o Modules/dlfcn_simple.o Modules/errnomodule.o Modules/faulthandler.o Modules/fcntlmodule.o Modules/fileio.o Modules/fnt.o Modules/fourstep.o Modules/fuzzer.o Modules/grpmodule.o Modules/io.o Modules/iobase.o Modules/itertoolsmodule.o Modules/malloc_closure.o Modules/mathmodule.o Modules/md5module.o Modules/microprotocols.o Modules/mmapmodule.o Modules/module.o Modules/mpalloc.o Modules/mpdecimal.o Modules/multibytecodec.o Modules/multiprocessing.o Modules/numbertheory.o Modules/posixmodule.o Modules/posixshmem.o Modules/prepare_protocol.o Modules/pwdmodule.o Modules/pyexpat.o Modules/readline.o Modules/resource.o Modules/rotatingtree.o Modules/row.o Modules/selectmodule.o Modules/semaphore.o Modules/sha1module.o Modules/sha256module.o Modules/sha3module.o Modules/sha512module.o Modules/signalmodule.o Modules/sixstep.o Modules/socketmodule.o Modules/statement.o Modules/stgdict.o Modules/stringio.o Modules/symtablemodule.o Modules/syslogmodule.o Modules/termios.o Modules/textio.o Modules/timemodule.o Modules/tkappinit.o Modules/transpose.o Modules/unicodedata.o Modules/util.o Modules/xmlparse.o Modules/xmlrole.o Modules/xmltok.o Modules/xxsubtype.o Modules/zlibmodule.o"
And here is my pyoxidizer.bzl
file. I notice that I'm usingpolicy.extension_module_filter = "no-libraries"
to keep the package size down. I'll see if changing the value to "all"
fixes the issue.
def make_dist():
return default_python_distribution()
def resource_callback(policy, resource):
if type(resource) == "PythonModuleSource":
resource.add_include = True
def make_exe(dist):
policy = dist.make_python_packaging_policy()
# use the callback function to filter out source code
# see: https://pyoxidizer.readthedocs.io/en/v0.9.0/packaging_resources.html#using-callbacks-to-influence-resource-attributes
# and: https://github.com/indygreg/PyOxidizer/issues/312
policy.register_resource_callback(resource_callback)
# Package all available Python extensions in the distribution.
# policy.extension_module_filter = "all"
policy.extension_module_filter = "no-libraries"
# Use in-memory location for adding resources by default.
policy.resources_location = "in-memory"
# Attempt to add resources relative to the built binary when
# `resources_location` fails.
policy.resources_location_fallback = "filesystem-relative:jclib"
# Remove source files from dist to reduce executable size (keep docstrings)
policy.include_distribution_sources = False
policy.include_non_distribution_sources = False
policy.bytecode_optimize_level_zero = False
policy.bytecode_optimize_level_one = True
policy.bytecode_optimize_level_two = False
# This variable defines the configuration of the embedded Python
# interpreter. By default, the interpreter will run a Python REPL
# using settings that are appropriate for an "isolated" run-time
# environment.
#
# The configuration of the embedded Python interpreter can be modified
# by setting attributes on the instance. Some of these are
# documented below.
python_config = dist.make_python_interpreter_config()
# Remove source for smaller executable (keep docstrings)
python_config.optimization_level = 1
# Enable the standard path-based importer which attempts to load
# modules from the filesystem.
python_config.filesystem_importer = True
# Evaluate a string as Python code when the interpreter starts.
python_config.run_command = "import jc.cli; jc.cli.main()"
# Produce a PythonExecutable from a Python distribution, embedded
# resources, and other options. The returned object represents the
# standalone executable that will be built.
exe = dist.to_python_executable(
name="jc",
packaging_policy=policy,
config=python_config,
)
# Invoke `pip install` using a requirements file and add the collected resources
# to our binary.
exe.add_python_resources(exe.pip_install(["-r", "requirements.txt"]))
# Return our `PythonExecutable` instance so it can be built and
# referenced by other consumers of this target.
return exe
def make_install(exe):
# Create an object that represents our installed application file layout.
files = FileManifest()
# Add the generated executable to our install layout in the root directory.
files.add_python_resource(".", exe)
return files
# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_dist)
register_target("exe", make_exe, depends=["dist"])
register_target("install", make_install, depends=["exe"], default=True)
# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()
# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.
PYOXIDIZER_VERSION = "0.16.0"
PYOXIDIZER_COMMIT = "4053178f2ba11d29f497d171289cb847cd07ed77"
Cool - changing policy.extension_module_filter
from "no-libraries"
to "all"
seems to have fixed the issue.
It looks like this was only needed on macOS and not linux or Windows since the _scproxy
package is a macOS-specific library.
Hi there!
I have been using pyoxidizer on macOS, linux, and Windows for a couple years now with no major issues. I recently tried to build a new version of my software on macOS and the binary compiles, but I get the following error when running:
It seems that the
xmltodict
library depends onurllib
, which depends on_scproxy
which can't be found. I have not run into this before even though I've been usingxmltodict
for years. This also still works fine on linux and windows.I thought maybe there was some macOS update that broke my xcode environment so I upgraded macOS and xcode, but I still get the same message.
Could this be a problem with the provided python interpreter bundles for macOS? I'm not sure how to troubleshoot this any further.
I'm on PyOxidizer 0.23.0 but this also happened on PyOxidizer 0.22.0. Again, the weird thing is that this worked just fine not long ago on v0.22.0 and below.
p.s. are any of these lines of any interest in the compilation output?