Certain PyObjC frameworks are not properly imported - Githubissues

ronaldoussoren / modulegraph2

Modulegraph2 is a library for creating and introspecting the dependency graph between Python modules.

MIT License

11 stars 8 forks source link

Certain PyObjC frameworks are not properly imported #22

Open maxbelanger opened 1 year ago

maxbelanger commented 1 year ago

We've been testing migrating from modulegraph to modulegraph2 for the build tool that produces our embedded Python package, and we stumbled across what seems to be a regression from modulegraph. It seems that in some cases, the modules within certain PyObjC frameworks are not fully discovered. This can be reproduced with the latest version of modulegraph2.

Here's an example with the Quartz framework:

Install pyobjc-framework-Quartz and modulegraph2.

In a Python shell, run the following:

import modulegraph2
g = modulegraph2.ModuleGraph()
g.add_module("Quartz")
g.report()

If you filter the output, there's only a single reference to Quartz:

Package         Quartz                    /opt/homebrew/lib/python3.10/site-packages/Quartz

There are no references to its source modules in the report, or of its extensions (e.g. Quartz/CoreGraphics/_inlines.abi3.so). This issue does not affect modulegraph.

ronaldoussoren commented 1 year ago

This might be related to PyObjC's lazy loader which does some things that can confuse import lib. I'll have to debug to know for sure.

maxbelanger commented 1 year ago

This does seem to correlate with use of ObjCLazyModule, yes.

ronaldoussoren commented 1 year ago

My current plan is to fix this on both ends (if possible):

Update PyObjC to copy the missing bits to the ObjCLazyModule instances
Teach modulegraph2 about this pattern

I'm saying this without having tried to reproduce this yet, should have some time tonight.

ronaldoussoren commented 1 year ago

This is what I get:

Package         Quartz                    /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz
Package         Quartz.CoreGraphics       /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics
ExtensionModule Quartz.CoreGraphics._callbacks /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_callbacks.cpython-310-darwin.so
SourceModule    Quartz.CoreGraphics._contextmanager /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_contextmanager.py
ExtensionModule Quartz.CoreGraphics._coregraphics /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_coregraphics.cpython-310-darwin.so
ExtensionModule Quartz.CoreGraphics._doubleindirect /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_doubleindirect.abi3.so
ExtensionModule Quartz.CoreGraphics._inlines /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_inlines.abi3.so
SourceModule    Quartz.CoreGraphics._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_metadata.py
ExtensionModule Quartz.CoreGraphics._sortandmap /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreGraphics/_sortandmap.abi3.so
Package         Quartz.CoreVideo          /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo
ExtensionModule Quartz.CoreVideo._CVPixelBuffer /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo/_CVPixelBuffer.abi3.so
SourceModule    Quartz.CoreVideo._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/CoreVideo/_metadata.py
Package         Quartz.ImageIO            /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageIO
SourceModule    Quartz.ImageIO._metadata  /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageIO/_metadata.py
Package         Quartz.ImageKit           /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit
ExtensionModule Quartz.ImageKit._imagekit /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit/_imagekit.abi3.so
SourceModule    Quartz.ImageKit._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/ImageKit/_metadata.py
Package         Quartz.PDFKit             /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit
ExtensionModule Quartz.PDFKit._PDFKit     /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit/_PDFKit.abi3.so
SourceModule    Quartz.PDFKit._metadata   /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/PDFKit/_metadata.py
Package         Quartz.QuartzComposer     /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzComposer
SourceModule    Quartz.QuartzComposer._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzComposer/_metadata.py
Package         Quartz.QuartzCore         /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore
SourceModule    Quartz.QuartzCore._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore/_metadata.py
ExtensionModule Quartz.QuartzCore._quartzcore /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzCore/_quartzcore.abi3.so
Package         Quartz.QuartzFilters      /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzFilters
SourceModule    Quartz.QuartzFilters._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuartzFilters/_metadata.py
Package         Quartz.QuickLookUI        /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI
ExtensionModule Quartz.QuickLookUI._QuickLookUI /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI/_QuickLookUI.abi3.so
SourceModule    Quartz.QuickLookUI._metadata /Users/ronald/Projects/modulegraph2/workenv/lib/python3.10/site-packages/Quartz/QuickLookUI/_metadata.py

This is with python 3.10 installed from python.org, modulegraph2 is the tip of the tree and pyobjc was installed using pip.

With python3.11 modulegraph2 hits a false assertion, haven't tried to debug that yet. UPDATE: just committed a fix for that, unrelated to this issue.

ronaldoussoren commented 1 year ago

Looks like homebrew is relevant here as well. Did you install pyobjc through homebrew as well?

maxbelanger commented 1 year ago

I did reproduce this using Homebrew, but this was also affecting an internal deployment of PyObjC that does not use Homebrew (something resembling a venv). I think the important factor is that a wheel is in use and on the Python path.

ronaldoussoren commented 1 year ago

I did reproduce this using Homebrew, but this was also affecting an internal deployment of PyObjC that does not use Homebrew (something resembling a venv). I think the important factor is that a wheel is in use and on the Python path.

What do you mean with "a wheel is in use and on the Python path"?

I finally got around to installing homebrew in a VM and cannot reproduce the issue there.

What I did:

VM running macOS 12
Install brew from brew.sh
brew install python3
Check that python3 now refers to the homebrew installation in /usr/local/bin
python3 -m pip install -U pip (necessary to be able to install modulegraph2 from the repository)
python3 -m pip install pyobjc
git clone https://github.com/ronaldoussoren/modulegraph2
cd modulegraph2 && pip install -e .
run the reproducer from the issue

I get the following output:

$ python3 t.py | grep Quartz
Package         Quartz                    /usr/local/lib/python3.11/site-packages/Quartz
Package         Quartz.CoreGraphics       /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics
ExtensionModule Quartz.CoreGraphics._callbacks /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_callbacks.cpython-311-darwin.so
SourceModule    Quartz.CoreGraphics._contextmanager /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_contextmanager.py
ExtensionModule Quartz.CoreGraphics._coregraphics /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_coregraphics.cpython-311-darwin.so
ExtensionModule Quartz.CoreGraphics._doubleindirect /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_doubleindirect.abi3.so
ExtensionModule Quartz.CoreGraphics._inlines /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_inlines.abi3.so
SourceModule    Quartz.CoreGraphics._metadata /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_metadata.py
ExtensionModule Quartz.CoreGraphics._sortandmap /usr/local/lib/python3.11/site-packages/Quartz/CoreGraphics/_sortandmap.abi3.so
Package         Quartz.CoreVideo          /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo
ExtensionModule Quartz.CoreVideo._CVPixelBuffer /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo/_CVPixelBuffer.abi3.so
SourceModule    Quartz.CoreVideo._metadata /usr/local/lib/python3.11/site-packages/Quartz/CoreVideo/_metadata.py
Package         Quartz.ImageIO            /usr/local/lib/python3.11/site-packages/Quartz/ImageIO
SourceModule    Quartz.ImageIO._metadata  /usr/local/lib/python3.11/site-packages/Quartz/ImageIO/_metadata.py
Package         Quartz.ImageKit           /usr/local/lib/python3.11/site-packages/Quartz/ImageKit
ExtensionModule Quartz.ImageKit._imagekit /usr/local/lib/python3.11/site-packages/Quartz/ImageKit/_imagekit.abi3.so
SourceModule    Quartz.ImageKit._metadata /usr/local/lib/python3.11/site-packages/Quartz/ImageKit/_metadata.py
Package         Quartz.PDFKit             /usr/local/lib/python3.11/site-packages/Quartz/PDFKit
ExtensionModule Quartz.PDFKit._PDFKit     /usr/local/lib/python3.11/site-packages/Quartz/PDFKit/_PDFKit.abi3.so
SourceModule    Quartz.PDFKit._metadata   /usr/local/lib/python3.11/site-packages/Quartz/PDFKit/_metadata.py
Package         Quartz.QuartzComposer     /usr/local/lib/python3.11/site-packages/Quartz/QuartzComposer
SourceModule    Quartz.QuartzComposer._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzComposer/_metadata.py
Package         Quartz.QuartzCore         /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore
SourceModule    Quartz.QuartzCore._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore/_metadata.py
ExtensionModule Quartz.QuartzCore._quartzcore /usr/local/lib/python3.11/site-packages/Quartz/QuartzCore/_quartzcore.abi3.so
Package         Quartz.QuartzFilters      /usr/local/lib/python3.11/site-packages/Quartz/QuartzFilters
SourceModule    Quartz.QuartzFilters._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuartzFilters/_metadata.py
Package         Quartz.QuickLookUI        /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI
ExtensionModule Quartz.QuickLookUI._QuickLookUI /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI/_QuickLookUI.abi3.so
SourceModule    Quartz.QuickLookUI._metadata /usr/local/lib/python3.11/site-packages/Quartz/QuickLookUI/_metadata.py

This looks correct to me.