Including itertools from cpython

[cloudultima3164]

Hi! First of all, I want to say thank you for developing this tool. I am a big fan of Rust, and this tool is probably the easiest packaging manager I've used so far (if I can call it that).

The issue: I'm still trying to figure out how to properly make a configuration file, so I may have just overlooked this in the documentation, but I can't seem to include itertools in the final build. My script runs fine if I build with policy.extension_module_filter = "all", but I would really rather use the minimal option and then fine-tune other included packages. I've tried a couple of different things to see if I could make it work, but to no avail.

My configuration file is basically the following:

def make_dist():
    return default_python_distribution()

def resource_callback(policy, resource):
    if resource.package == "itertools":
        resource.add_include = True
# ^^^^ tried this

def make_exe(dist):
    policy = dist.make_python_packaging_policy()
    policy.allow_in_memory_shared_library_loading = True
    policy.bytecode_optimize_level_two = True
    policy.file_scanner_classify_files = True
    policy.file_scanner_emit_files = False
    policy.include_classified_resources = True
    policy.extension_module_filter = "minimal"
    policy.resources_location = "in-memory"
    policy.resources_location_fallback = "filesystem-relative:lib"
    policy.register_resource_callback(resource_callback)

    python_config = dist.make_python_interpreter_config()
    python_config.filesystem_importer = True
    python_config.run_module = "my_script"

    exe = dist.to_python_executable(
        name="my_script",
        packaging_policy=policy,
        config=python_config,
    )
    exe.add_python_resources([
        PythonExtensionModule(
            name="itertools",
            is_stdlib=True,
            add_include=True,
            add_location="in_memory",
            add_location_fallback="filesystem-relative:lib",
            add_bytecod_optimization_level_two=True
        ),
       # ^^^^ also tried this

        exe.read_package_root(
            path="my_path",
            packages=["my_script"]
        ),
        exe.read_package_root(
            path="my_path/src/py",
            packages=["local_package1", "local_package2", "local_package3"]
        ),
        exe.pip_install(["-r", "req.txt"]
        )
    ])
    return exe

def make_embedded_resources(exe):
    return exe.to_embedded_resources()

def make_install(exe):
    # Create an object that represents our installed application file layout.
    files = FileManifest()

    # Add the generated executable to our install layout in the root directory.
    files.add_python_resource(".", exe)

    return files

# Tell PyOxidizer about the build targets defined above.
register_target("dist", make_dist)
register_target("exe", make_exe, depends=["dist"])
register_target("resources", make_embedded_resources, depends=["exe"], default_build_script=True)
register_target("install", make_install, depends=["exe"], default=True)

# Resolve whatever targets the invoker of this configuration file is requesting
# be resolved.
resolve_targets()

# END OF COMMON USER-ADJUSTED SETTINGS.
#
# Everything below this is typically managed by PyOxidizer and doesn't need
# to be updated by people.

PYOXIDIZER_VERSION = "0.16.2"
PYOXIDIZER_COMMIT = "e91995636f8deed0a7d8e1917f96a7dc17309b63"

However, I'm guessing neither of the things I tried in the above code are not the right approach, as the callback results in:

error[PYOXIDIZER_BUILD]: error converting PythonResource to Value: DiagnosedError(Diagnostic { level: Error, message: "Cannot .package on type PythonExtensionModule", code: Some("CV00"), spans: [SpanLabel { span: Span { low: Pos(138), high: Pos(154) }, label: Some(".package not supported for type PythonExtensionModule"), style: Primary }] })
  --> ./pyoxidizer.bzl:30:11
   |
30 |       exe = dist.to_python_executable(
   |  ___________^
31 | |         name="my_script",
32 | |         packaging_policy=policy,
33 | |         config=python_config,
34 | |     )
   | |_____^ to_python_executable()

error: error converting PythonResource to Value: DiagnosedError(Diagnostic { level: Error, message: "Cannot .package on type PythonExtensionModule", code: Some("CV00"), spans: [SpanLabel { span: Span { low: Pos(138), high: Pos(154) }, label: Some(".package not supported for type PythonExtensionModule"), style: Primary }] })

and the attempt to directly create a PythonExtensionModule results in:

error[CM01]: Variable 'PythonExtensionModule' not found
  --> ./pyoxidizer.bzl:31:9
   |
31 |         PythonExtensionModule(
   |         ^^^^^^^^^^^^^^^^^^^^^ Variable was not found

error: Variable 'PythonExtensionModule' not found

Sorry if this is just something I missed in the documentation, but can someone please help me understand the proper way to fine tune which packages are included in the build when using policy.extension_module_filter = "minimal"?

[indygreg]

There are a few issues with your config file.

In this code:

def resource_callback(policy, resource):
    if resource.package == "itertools":
        resource.add_include = True

The idea is sound. But PythonExtensionModule doesn't expose a package attribute. You'll want to do something like this:

def resource_callback(policy, resource):
    if type(resource) == "PythonExtensionModule":
        if resource.name == "itertools":
            resource.add_include = True

In this code:

exe.add_python_resources([
        PythonExtensionModule(
            name="itertools",
            is_stdlib=True,
            add_include=True,
            add_location="in_memory",
            add_location_fallback="filesystem-relative:lib",
            add_bytecod_optimization_level_two=True
        ),
       # ^^^^ also tried this

        exe.read_package_root(
            path="my_path",
            packages=["my_script"]
        ),
        exe.read_package_root(
            path="my_path/src/py",
            packages=["local_package1", "local_package2", "local_package3"]
        ),
        exe.pip_install(["-r", "req.txt"]
        )
    ])

You cannot construct instances of PythonExtensionModule, as there is no constructor function.

Also, this code effectively passes a list of non-uniform types to add_python_resources(). add_python_resources() wants a list of resource-like types. A list of lists won't work. (Although it could potentially be made to work.)

Let me know if you have better success with this feedback!