search

beeware / Python-Apple-support

A meta-package for building a version of Python that can be embedded into a macOS, iOS, tvOS or watchOS project.
MIT License
1.12k stars 161 forks source link

Generate the exported symbols list #217

Closed Kentzo closed 4 months ago

Kentzo commented 5 months ago

What is the problem or limitation you are having?

Currently linking against the generated static library requires a unrestricted export all external symbols known by the target binary. Otherwise LTO will remove them and binary modules won't load.

Describe the solution you'd like

The tooling should produce the exported symbols list that can be set either in Xcode settings or ld's -exported_symbols_list.

Describe alternatives you've considered

N/A

Additional context

No response

freakboy3742 commented 4 months ago

Thanks for the suggestion.

The situation around linking has changed significantly in the last 6 months; we've just finished landing all the patches used by this repository upstream in CPython 3.13, and along the way, a couple of modifications have been made. In particular, extension binaries are now dynamically linked against libPython, negating the need for the use of the deprecated -undefined,dynamic_lookup linker flag. I expect that these changes will be backported into this support package later this year (before the formal release of CPython 3.13.0 in October).

Using the support package also requires that binary stripping is disabled - achieved by turning on the Enable Testability setting in your Xcode project. I believe this avoids the LTO issue you're referring to.

So: I don't believe there's a need for an exported symbols list. If nothing else, iOS would be the only CPython platform that uses such a linking option. Closing this ticket on that basis.

Kentzo commented 4 months ago

Was there any review of the symbols that remain exported by the libPython dynamic library? I'm worried that it ends up exporting stuff that's not part of Limited API and standard binary modules requirements. Please see this discussion https://discuss.python.org/t/embedding-python-via-static-libraries-list-of-exported-symbols/52835

freakboy3742 commented 4 months ago

... was there any iOS-specific review of the symbols of the libPython that is used on every platform? No.

I'm going to echo the comments that you've received on that discuss.python thread - it's not clear why what you're asking for is needed. You clearly have an eclectic mode of operation in mind; it's not the mode of operation that is documented, and talking about "static linking of iOS" with Python 3.13 is a moot point, because static linking is specifically disabled by the iOS build.

If you have something specific in mind that I'm not understanding, patches are welcome. I'm not guaranteeing that I'm going to merge an arbitrary patch, though - any patch should be accompanied with a detailed description of how whatever you're adding would actually be used and useful in practice.

Kentzo commented 4 months ago

If you look at Apple's own distribution of open source software, you can see that in many cases they utilize EXPORTED_SYMBOLS_FILE of generated Xcode projects to narrow public interfaces (symbols table) of the bundled libraries.

I'm considering a patch, but I'm not sure about the maintainer's attitude towards this problem. If such addition is to be made it will need to be universal for all supported platforms. I feel that my case is not strong enough.

freakboy3742 commented 4 months ago

Well, I am the maintainer of this project, and I'm on the CPython triage team. I can't speak for anyone else on the CPython team, but from my perspective, I can tell you that that no - you definitely haven't made a strong case yet.

As I see it: You're advocating for a build artefact that is, by all accounts, not needed on any other platform - and iOS seems to be quite happy to live without it as well. You mention an EXPORTED_SYMBOLS_FILE - I don't deny this file exists, but I've literally never seen it, or needed to use it, and I don't know how I would use it. That might be a gap in my own knowledge - if it is, I suspect it may be a gap that exists in a large portion of the rest of the CPython team's knowledge as well.

Kentzo commented 4 months ago

Right, I understand that both this project and the upstream is moving away from static libraries. However this "issue" is true for either artifact.

I think a good analogy is that: imagine that Python.h exposed a symbol that's neither a part of Limited API nor needed by standard binary modules. It's nothing critical but an annoyance nevertheless.

freakboy3742 commented 4 months ago

CPython is not moving away from static libraries, except in as much that static linking is a lot less popular in general than static linking. I have seen no evidence to suggest that CPython is going to stop generating libPython.a for Linux and macOS.

CPython on iOS is not using static linking because third party binary modules need to be able to dynamically load libPython, which means a statically linked iOS binary would need to ship a dynamic libPython anyway.

Worrying about Python.h exposing symbols that aren't part of the Limited API seems like worrying about an entirely hypothetical problem. CPython does a lot of work (and has a lot of tooling) to define and enforce its public API. Introducing a whole level of complexity "in case CPython makes a mistake" seems like solving the wrong problem to me.

Kentzo commented 4 months ago

Could you point me to the relevant tooling? I would like to use it in my project to generate the exports file rather than to rely on the hacking approach described in the linked post.

freakboy3742 commented 4 months ago

It's not an area I'm especially familiar with; but AIUI, this is what the "argument clinic" is for: https://devguide.python.org/development-tools/clinic/