provide pep425 platform tag for AIX

[aixtools]

Environment

• pip version: 19.2.2
• Python version: Any (testing on Python3.7.4)
• OS: AIX

Description pip._internal.pep425tags.get_platform() removes dependency from distutils.util.get_platform()

Just as for macos, the value returned by distutils is insufficient for portable packaging, as it lacks sufficient details.

Expected behavior When AIX is the platform, return a platform in the form: AIX_V_R_M_YYWW_bNN

where: V=version; R=release; M=TechLevel; YYWW represents the builddate (Year (19 for 2019); WW is week of year); bNN - where NN is 32|64 for 32 or 64 bit mode)

How to Reproduce

1. Get standard pip
2. Then run "pip debug'
3. AIX platform dependents tags use value from distutils

Output Without: pip version: pip 19.2.2 from /opt/lib/python3.7/site-packages/pip (python 3.7) sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C] sys.executable: /opt/bin/python3.7 sys.getdefaultencoding: utf-8 sys.getfilesystemencoding: iso8859-1 locale.getpreferredencoding: ISO8859-1 sys.platform: aix6 sys.implementation: name: cpython Compatible tags: 20 cp37-cp37m-aix_6_1 cp37-abi3-aix_6_1 cp37-none-aix_6_1 cp36-abi3-aix_6_1 cp35-abi3-aix_6_1 cp34-abi3-aix_6_1 cp33-abi3-aix_6_1 cp32-abi3-aix_6_1 py3-none-aix_6_1

With: (build) root@x066:[/home/root]pip debug WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice. pip version: pip 19.3.dev0 from /home/root/build/lib/python3.7/site-packages/pip-19.3.dev0-py3.7.egg/pip (python 3.7) sys.version: 3.7.4 (default, Aug 14 2019, 12:40:12) [C] sys.executable: /home/root/build/bin/python sys.getdefaultencoding: utf-8 sys.getfilesystemencoding: iso8859-1 locale.getpreferredencoding: ISO8859-1 sys.platform: aix6 sys.implementation: name: cpython Compatible tags: 20 cp37-cp37m-AIX_6_1_7_1415_b32 cp37-abi3-AIX_6_1_7_1415_b32 cp37-none-AIX_6_1_7_1415_b32 cp36-abi3-AIX_6_1_7_1415_b32 cp35-abi3-AIX_6_1_7_1415_b32 cp34-abi3-AIX_6_1_7_1415_b32 cp33-abi3-AIX_6_1_7_1415_b32 cp32-abi3-AIX_6_1_7_1415_b32 py3-none-AIX_6_1_7_1415_b32 cp37-none-any

[chrahunt]

This may be better in pypa/packaging as we're planning to replace pep425tags with its tags module as mentioned in #6908.

[aixtools]

OK. I forked packaging, and shall add a PR there as well.

But - how do I test this?

And, will pypa/packaging (do not know how to make it blue, etc.) also replace pep425tags in pypa/wheel (where I am also submitting a change asap).

If yes - wonderful - only one change (to make for the future) - but how to test now?

[cjerdonek]

But - how do I test this?

Are you asking how to write a test? Or ...

[aixtools]

No. not write a test. Not sure what I would test.

How to test using Tag, actually - although, perhaps I should/could ask the same there.

My struggle is that I also have this same 'chunk' added into pypa/wheel (about to post that PR as well).

To date, the testing I have been doing involves using pypa/pip and pypa/wheel, both patched, using the devpi (https://pypi.org/project/devpi/) platform to test upload and download of wheels.

I have this much working - before I opened a new issue/PR - to be sure I had the minimum working. A new piece I also want to see happening is when an AIX system, at a higher, better later, platform tag level than any published wheel - accepts the published wheel - as it should be binary compatible.

I have been doing this (providing pre-packaged modules and Python) - packaged on AIX 6.1 with "users" of the packaging running these, unmodified on AIX 7.1 and AIX 7.2.

I want, now that it is clearly doable (including virtualenv environments), to make packaging available using Python (and pypi repository) because it is superior to platform software management tools. (On AIX these are installp or RPM).

So, back to testing - how do I test using Tag() rather than pep425tags.get_platform() ?

[cjerdonek]

No. not write a test. Not sure what I would test.

Generally speaking, whenever there is a change to the behavior of a function, there should usually be a unit test testing that change IMO. And in cases where the behavior change depends on a particular platform, it's best if the test can be written to run on all platforms (especially if the platform isn't part of the CI grid). That may mean patching and/or mocking certain functions.

So, back to testing - how do I test using Tag() rather than pep425tags.get_platform() ?

Since pip isn't even using Tag yet, it's probably premature to be working on / thinking too hard about how to do integration testing of that with pip. If you're asking about unit tests for Tag, that would be better discussed in packaging's repo, since that's where that code lives.

[chrahunt]

May be related to #5453.

[aixtools]

Maybe I do not understand how you use the word behavior as in the context of change behavior. I would not see this as a behavior change - only more specific output string.

Without the change (and the accompanying one in pypa/wheel) I can create wheels - but the tag (coming from distutils.itil/get_platform(), or sys.get_platform() is insufficient.

Including AIX 6.1 TL7 - which is the version of AIX 6.1 I build on - there are three TL (technology levels) (7, 8 and 9) to be considered. For AIX 7.1 there are currently 6 TL0..TL5), and for AIX 7.2 that will soon be 4 (TL3 is soon to be released)).

So, back to simplicity - if I package on AIX 6.1 TL7 and load on AIX 6.1 TL9, both as 32 or 64 bit, the current tagging could work. However, if I packaged on AIX 6.1 TL9 that current tagging would try to load the tagged wheel, but it would not be binary compatible. AIX provides binary compatibility from old to new, not both directions.

A 'behavior change I would like to get included, but separate from this tag change, is that pypa tools would accept a properly tagged AIX 6.1TL7 wheel on e.g., AIX 61.TL9, or AIX 7.2 TL0, while and AIX 7.1 TL3 system would reject a wheel tagged as coming from AIX 7.1 TL4.

In short, the current proposal merely resolves the inadequacy of distutils.util.get_paltform() (and sys.get_platform()) to adequately tag wheels intended for AIX. A 'bug' that has been in the system since at least 2007 when IBM adopted the "TL-SP" methodology and also adequately labeled the filesets (using the M digit of VRMF) to identify the Technology Level.

[cjerdonek]

Maybe I do not understand how you use the word behavior as in the context of change behavior.

By "change in behavior," I was making a distinction between PR's that don't change anything for the end user (e.g. refactoring PR's) versus PR's that do change something (e.g. bug fixes, feature additions, etc). In the case of PR #6923 that you proposed, what it does is change the return value of get_platform(), so I was making the point that you can add a unit test to test that change. This was in reply to your comment that you weren't sure what you should test.

[aixtools]

Ah :bigsmile:, yes, something simple I might even be able to do.

FYI - in case you are wondering why I do not use "oslevel -s" to get the values. a) oslevel -s takes more time b) oslevel -s can be 'misleading', as it 'downgrades the level reported if an obscure fileset is behind, while bos.mp64 is guaranteed to be current with the key run-time files.

root@x064:[/home/root]oslevel -s
7100-04-01-1543
root@x064:[/home/root]oslevel -s -q | head
Known Service Packs
-------------------
7100-04-06-1806
7100-04-05-1720
7100-04-04-1717
7100-04-03-1642
7100-04-02-1614
7100-04-01-1543
7100-04-00-0000
7100-03-09-1717

On this server, the man pages, and X11 rte is behind (not the base kernel

 root@x064:[/home/root]oslevel -l 7100-04-06-1806 -s
Fileset                                 Actual Level       Service Pack Level
-----------------------------------------------------------------------------
X11.adt.lib                             7.1.2.15           7.1.4.30
X11.apps.rte                            7.1.4.0            7.1.4.30
X11.base.lib                            7.1.4.0            7.1.4.31
X11.base.rte                            7.1.4.0            7.1.4.30
X11.compat.lib.X11R6                    7.1.4.0            7.1.4.30
X11.compat.lib.X11R6_motif              7.1.3.0            7.1.4.31
X11.motif.lib                           7.1.3.0            7.1.4.31
infocenter.man.EN_US.commands           7.1.4.0            7.1.4.30
infocenter.man.EN_US.files              7.1.4.0            7.1.4.30
infocenter.man.EN_US.libs               7.1.4.0            7.1.4.30

[pradyunsg]

AIX provides binary compatibility from old to new

OK cool, so there's a forward compatible ABI.

inadequacy of distutils.util.get_paltform() (and sys.get_platform()) to adequately tag wheels intended for AIX.

I'm confused. IIUC, you're the primary distributor of AIX builds for Python, and have contributed compatibility patches upstream as well.

Is there some reason you can't patch the Python that you distribute, to return the appropriate value from distutils/sys? Or even contribute that patch upstream to CPython?

My main concern is that while you know AIX very well, I have no clue what that platform is or how it works and honestly, no interest in learning about it. Having more bits of code in pip, for which we have exactly 0 expertise or significant understanding among the maintainers, is not a scenario I'd want to be in.

To be clear, I'm not opposed to AIX support -- I have a really strong preference for not being responsible for things I don't understand. :)

[aixtools]

re: a test - I am not familiar with pytest. How do I get verbose output similar to python -m test -v test_xyz

I am also trying options python setup.py test using options -v, -m (argument) -s (argument)

python setup.py test seems equivalent with python setup.py -s tests

that seems to only run tests.unit.test_vcs - HOW to get more!

Michael

[aixtools]

AIX provides binary compatibility from old to new

OK cool, so there's a forward compatible ABI.

Yes, unofficial since 1998-1999, official since 2007 (and AIX 6.1).

inadequacy of distutils.util.get_paltform() (and sys.get_platform()) to adequately tag wheels intended for AIX.

We may be in Python4 before that would ever get added :smile: and :sigh:

I'm confused. IIUC, you're the primary distributor of AIX builds for Python, and have contributed compatibility patches upstream as well.

I am a primary packager. IBM itself prefers to package using gcc and RPM - and has shown no interest in the Python way.

Is there some reason you can't patch the Python that you distribute, to return the appropriate value from distutils/sys? Or even contribute that patch upstream to CPython?

While I could patch sys.get_platform() and distutils.util.get_platform() to get around the PR process in 4 repositories (cpython, pip, wheel, packaging) I do not feel it is the right way to be doing this.

My main concern is that while you know AIX very well, I have no clue what that platform is or how it works and honestly, no interest in learning about it. Having more bits of code in pip, for which we have exactly 0 expertise or significant understanding among the maintainers, is not a scenario I'd want to be in.

I understand. I do not like to flame anyone, but IBM should have made some kind of support for Python possible 'decades' ago. It breaks my heart to see it ignored, and as a consequence, the MANY hacks VARs and customers apply to get anything done. Worse, it could become a reason to leave AIX.

So, if you say no - I may just use the idea you gave me (patch Cpython outside of official sources).

What I did not mention above is that it seems to take forever and a day to get a PR for AIX considered. There are people, who when approached individually, look more closely - and some progress is made. Everyone is a volunteer - and I do not want to be pinging, ttt posts, etc..

As much as possible, for better and worse, I am trying to be part of the process - not outside of it.

To be clear, I'm not opposed to AIX support -- I have a really strong preference for not being responsible for things I don't understand. :)

Understood. As my understanding of Python grows I identify where AIX support is lacking. Wheels (and pip) are great. Off topic: one of the things I liked about perl is CPAN. Wheels are so much better. But it must be done right - or the forward compatible ABI cannot work properly.

p.s. Someone checked in PyPi for me - there are zero AIX wheels/eggs etc. in PyPi, so we have no pip/setuptools/wheel dependencies to worry about. There is light in the tunnel!

[pradyunsg]

While I could patch sys.get_platform() and distutils.util.get_platform() to get around the PR process in 4 repositories (cpython, pip, wheel, packaging) I do not feel it is the right way to be doing this.

Hmm... I feel that might actually be the right way. I'll optimistically ping @ncoghlan, @pfmoore and @dstufft since they all likely have a better understanding of this stuff than me.

[pradyunsg]

re: a test - I am not familiar with pytest. How do I get verbose output similar to python -m test -v test_xyz

I don't know if have already, but if you haven't, please look at https://pip.pypa.io/en/latest/development/getting-started/. That should contain the answer to your question.

---

To answer your question more directly, you can do:

tox -e py37 -- -k test_xyz -v

This will invoke pytest like:

pytest -k test_xyz -v

That should do what I think you want to do.

[pfmoore]

Hmm... I feel that might actually be the right way.

I agree. I also have zero knowledge of AIX, and no real interest in, or need for, knowing more. However, I do think the fact that you're patching multiple places to work around a limitation in the stdlib distutils function says something about where the fix should go.

Having said that, it will also make a difference what versions of Python you want to support here. There is a near-zero chance, I suspect, that a change like this would be viewed as a bugfix for CPython, so you'd be targeting Python 3.9+, which may limit your audience. So from a purely practical point of view, making the change in the packaging tools may be your only option.

However, I share @pradyunsg's reluctance to take OS-specific patches in pip. I've seen the discussions you have had in getting AIX fixes into CPython, and while I have sympathy with you over the difficulties you have encountered, I also understand why the core team don't want to be left supporting platform-specific code that only you understand. If you put yourself on the critical path for AIX on pip, setuptools and wheel as well, the problem gets even worse.

One possible workaround for the issue might be if you were to create a (pure Python) aix-support library, that contained helper utilities for adding AIX support to pip etc. Then you could maintain and support that library, and the packaging tools could depend on it. If there's any issue with AIX, we could direct the problem at your library, and all we'd need to do would be to update our dependencies once you release a fix. There would likely be an initial patch to the packaging tools to call your library, but ongoing support would be handled by you, not by pip. Also, that sort of approach could be used as a model for any future requests for support of less common platforms.

[chrahunt]

I can understand why the bitness should be in the ABI tag, but I'm not sure about the build week and TL number. Wouldn't that imply a specific package would need to be uploaded for each client, or that each client would need to include their latest and all prior versions in their list of applicable tags?

[aixtools]

On 26/08/2019 17:01, Paul Moore wrote:

Hmm... I feel that might actually be the right way. I agree. I also have zero knowledge of AIX, and no real interest in, or need for, knowing more. However, I do think the fact that you're patching multiple places to work around a limitation in the stdlib distutils function says something about where the fix should go.

Having said that, it will also make a difference what versions of Python you want to support here. There is a near-zero chance, I suspect, that a change like this would be viewed as a bugfix for CPython, so you'd be targeting Python 3.9+, which may limit your audience. So from a purely practical point of view, making the change in the packaging tools may be your only option.

Would be 'nice' if it was in 3.8 also officially - in distutils.util.get_platform() - perhaps calling a pure-python _aix.py (as I did also in Lib/ctypes).

Same for sys.get_platform() - perhaps the _aix.py could go in "sys" and distutils.util could import that - only code in one place.

As to Python3.7 - I could patch that manually before packaging - and refer to the official code - that it is there, etc..

However, I share @pradyunsg's reluctance to take OS-specific patches in pip. I've seen the discussions you have had in getting AIX fixes into CPython, and while I have sympathy with you over the difficulties you have encountered, I also understand why the core team don't want to be left supporting platform-specific code that only you understand. If you put yourself on the critical path for AIX on pip, setuptools and wheel as well, the problem gets even worse.

Maybe 'packaging' will fix more things. But I shall try patching cpython, installing, and testing with default pip and wheel. The strange issue with wheel is that it uses a special feature that apparantly calls dist.utils.get_platform() so that when the wheel pep425 code goes looking, it sees it is already set, so does not call the OS specific code. Again, if the cpython code was giving a good tag - that would not be needed either.

Please let me emphasize - with no patches - pip and wheel do what they are suppossed to do, but the tag is not sufficient.

I'll reply to your second mail re: technology and build_date. Those are vital to permitting forward compatible ABI.

One possible workaround for the issue might be if you were to create a (pure Python) aix-support library, that contained helper utilities for adding AIX support to pip etc. Then you could maintain and support that library, and the packaging tools could depend on it. If there's any issue with AIX, we could direct the problem at your library, and all we'd need to do would be to update our dependencies once you release a fix. There would likely be an initial patch to the packaging tools to call your library, but ongoing support would be handled by you, not by pip. Also, that sort of approach could be used as a model for any future requests for support of less common platforms.

[aixtools]

On 27/08/2019 02:26, Christopher Hunt wrote:

I can understand why the bitness should be in the ABI tag, but I'm not sure about the build week and TL number. Wouldn't that imply a specific package would need to be uploaded for each client, or that each client would need to include their latest and all prior versions in their list of applicable tags?

I either wrote earlier, or intended to send a larger text regarding VRMF and build-date. Here is an excerpt:

While one might think you can update (migrate) from any level of AIX 6.1 to any level of AIX 7.1 this is not the case. There is a final value, not usually mentioned, known as the "build-date". For an AIX system to update and maintain binary compatibility for applications the new OS level must satisfy all of the following AIX0=>AIX1 (V0 <= V1, R0 <= R1, M0 <= M1, Builddate0 <= Builddate1)

e.g., these two official AIX releases:

6100-07-01-1141 6100-06-12-1339

Although AIX 6.1 TL6 appears to be 'younger' that AIX 6.1 TL7 - which is true 'feature wise', that is what a TL specifies - the build-dates are not. the ABI forward compatibility is not supported. To update 6100-06-12-1339 I would need, at a minimum - 6100-07-08-1339. All VRM-BD numbers are increased or equal (the SP numbers, just as the F in VRMF) do not assist with maintaining ABI forward compatibility.

I hope this answers your question re: TL and builddate.

Note: because of the difference in features I do not want a (oslevel -s) tag of

6100-07-08-1339 and 6100-06-12-1339 to be compressed to 6100-1339

To compress the tag - rather than so many parts - just go with:

6106-1339 and 6107-1339, etc..

This is regarding 'tag' generation - and again, pip and wheel will work

• asis - with these tags. A feature change is adding the logic to see that a wheel packaged with tag 6107-1339 can be loaded by 6108-1339 (6100-08-03-1339).

Thanks for asking!

Michael

[pfmoore]

This is regarding 'tag' generation - and again, pip and wheel will work - asis - with these tags. A feature change is adding the logic to see that a wheel packaged with tag 6107-1339 can be loaded by 6108-1339 (6100-08-03-1339).

I really have no interest in understanding the complexities of AIX. But do you understand that if I run pip on a machine that is tag 6108-1339, in order to be able to load wheels tagged with 6107-1339, pip has to know that (a) that older tag exists, and (b) it's compatible with a 6108-1339 system?

See pep415tags.get_supported and all the complexity under arch.startswith('macosx') for a comparable example. Or for a simpler version, the manylinux2010 branch that adds manylinux1 to the list of supported tags. Basically, an installer has to look at the target architecture, and generate an exhaustive list of all possible tags that are compatible with that architecture - there's no concept of "plus all older tags".

I'd strongly recommend that you test any proposed changes by trying to install a wheel generated on an older version of AIX onto a system that runs a newer version. If that's something you already do, and you have this covered, then that's fine. I'm not looking at any code you've submitted, just going off your explanations (which I could easily have misunderstood, as they include a lot of things I'm not even trying to understand 🙂)

[chrahunt]

Basically, an installer has to look at the target architecture, and generate an exhaustive list of all possible tags that are compatible with that architecture - there's no concept of "plus all older tags".

This was what I was trying to say, thank you. Indicating that all the old build numbers are compatible means that list has to be kept somewhere. I can't see how it could be generated.

[aixtools]

See pep415tags.get_supported and all the complexity under arch.startswith('macosx') for a comparable example. Or for a simpler version, the manylinux2010 branch that adds manylinux1 to the list of supported tags. Basically, an installer has to look at the target architecture, and generate an exhaustive list of all possible tags that are compatible with that architecture - there's no concept of "plus all older tags".

I replied by e-mail, re: how to extract a list from AIX of potential tags (oslevel -s -q - with q for query).

I have not researched how .../+simple/package works. Being naive I thought it generated a query to ask what is there, and use what is available as it's list. I am taking no concept of older tags to no concept of other tags, only my current tag.

re: macos, if I understand the basics correctly, the build system has some defines in the build that it uses to determine it's oldest level possible (or is that required level, not sure if macOS is actually ABI forward compatible. My wife complains that she old software no longer works after an update - and I stay away from macOS because of similar concerns. And, likewise, I am no longer interested in figuring it out (did that back in macOS v2 through v7 days).

So, thanks for the hints and suggestions.

p.s., to someone else - yes, I am testing - incrementally. Now I'll go a step back to see if patches to pip and wheel can be skipped for the 'basics', and then look at writing an _aix.py module for the .get_supported() piece.

Note: if I slow in responses - it is because my real job is getting busy and time is limited. Clearly not because I am losing interest. I consider this quite important and I want to see it done correctly.

[aixtools]

As to the best start...

I have a potential PR for CPython that "fixes" sysconfig.get_platform() and distutils.util.get_platform().

With stock pip and wheel, pip debug gives expected output, as does pip wheel cffi

(build) root@x066:[/home/root]pip debug
WARNING: This command is only meant for debugging. Do not use this with automation for parsing and getting these details, since the output and options of this command may change without notice.
pip version: pip 19.2.3 from /home/root/build/lib/python3.9/site-packages/pip (python 3.9)
sys.version: 3.9.0a0 (heads/pep425-sysconf-dirty:cbdd55fdc0, Aug 28 2019, 16:08:08) [C]
sys.executable: /home/root/build/bin/python3.9
sys.getdefaultencoding: utf-8
sys.getfilesystemencoding: iso8859-1
locale.getpreferredencoding: ISO8859-1
sys.platform: aix
sys.implementation:
  name: cpython
Compatible tags: 24
  cp39-cp39-AIX_6107_1415_b32
  cp39-abi3-AIX_6107_1415_b32
  cp39-none-AIX_6107_1415_b32
  cp38-abi3-AIX_6107_1415_b32
  cp37-abi3-AIX_6107_1415_b32
  cp36-abi3-AIX_6107_1415_b32
  cp35-abi3-AIX_6107_1415_b32
  cp34-abi3-AIX_6107_1415_b32
  cp33-abi3-AIX_6107_1415_b32
  cp32-abi3-AIX_6107_1415_b32
  ...
  [First 10 tags shown. Pass --verbose to show all.]

(build) root@x066:[/home/root]pip wheel cffi
Looking in indexes: http://x064:8441/root/pypi/
Collecting cffi
  Downloading http://x064:8441/root/pypi/%2Bf/041/c81822e9f84b1/cffi-1.12.3.tar.gz (456kB)
     |################################| 460kB 26.5MB/s
Collecting pycparser (from cffi)
  File was already downloaded /home/root/pycparser-2.19-py2.py3-none-any.whl
Skipping pycparser, due to already being wheel.
Building wheels for collected packages: cffi
  Building wheel for cffi (setup.py) ... done
  Created wheel for cffi: filename=cffi-1.12.3-cp39-cp39-AIX_6107_1415_b32.whl size=198417 sha256=01d07f02f6cf5cda14aaf509c68d2636065d4745ee8a80c01dd5f74d737a1787
  Stored in directory: /home/root
Successfully built cffi
(build) root@x066:[/home/root]

[aixtools]

One possible workaround for the issue might be if you were to create a (pure Python) aix-support library, that contained helper utilities for adding AIX support to pip etc. Then you could maintain and support that library, and the packaging tools could depend on it. If there's any issue with AIX, we could direct the problem at your library, and all we'd need to do would be to update our dependencies once you release a fix. There would likely be an initial patch to the packaging tools to call your library, but ongoing support would be handled by you, not by pip. Also, that sort of approach could be used as a model for any future requests for support of less common platforms.

I'll also investigate this. Currently working with Cpython/Lib/_aix_support (using CPython/Lib/_osx_support as an example)

[ncoghlan]

Just noting that I'm actually the one that nudged @aixtools towards finding an approach that would allow the use of properly tagged AIX wheels on existing versions of Python, rather than relying solely on standard library updates that would only be available on Python 3.9+.

While it may end up making sense to structure the work as "That is the proper fix for Python 3.9+,this is a backport of equivalent behaviour to earlier versions", I'm not entirely convinced of that - platform tags should be able to naturally evolve based on platform update cycles, not standard library baseline API updates (for example, see the multitude of challenges around the various incarnations of the manylinux spec).

[pfmoore]

@ncoghlan Understood, but I still don't think it's a good idea to embed specialised platform knowledge for an obscure platform like AIX into tools like pip - hence my suggestion of some sort of aix-support PyPI project that centralised both the implementation and the support of such code.

[ncoghlan]

@pfmoore Yeah, I posted my comment before I had fully caught up on the rest of the thread. Your aix-support suggestion here has a fair bit in common with my manylinux-wheel-tags suggestion over at https://discuss.python.org/t/publishing-manylinux-install-tag-heuristics-as-a-dedicated-pypi-package/2039

So what I'm starting to wonder is whether there may be value in having some kind of naming convention for support libraries that adapt pip/packaging/etc to specific target platforms in a way that allows the packaging support for those platforms to improve without requiring a full pip update. For example, using packaging as a common suffix rather than as a prefix:

aix-packaging manylinux-packaging osx-packaging windows-packaging android-packaging ios-packaging wasm-packaging

Precedent: ntpath and posixpath in the standard library (the concrete modules backing the os.path alias)

[pradyunsg]

I'm starting to wonder is whether there may be value in having some kind of naming convention for support libraries that adapt pip/packaging/etc to specific target platforms

Yes, I think there is. This is a fairly interesting idea that I think I'll bring up on the packaging issue tracker. I'm thinking packaging.tags, and in general packaging, should be the location where these details are managed, not pip and other tooling.

[aixtools]

The PR I shall be submitting for consideration consists of an additional env variable (for the builddate of the built python - as this sets the base value for any packaging - same, in purpose, as MACOSX_DEPLOYMENT_TARGET. This would not actually be referenced by anything until someone gets around to writing aix_packaging.get_platforms().

The other change is Lib/_aix_support.py and changing the code for aix from/to:

         # fall through to standard osname-release-machine representation
     elif osname[:3] == "aix":
-        return "%s-%s.%s" % (osname, version, release)
+        from _aix_support import get_platform as aix_platform
+        return aix_platform()
     elif osname[:6] == "cygwin":

Should we 'switch' the main discussion of "pypa/packaging"?

@ncoghlan - does it make good sense to hurry with an issue and PR on CPython? I had investigated pypa/ first, to see also what could be done outside of Cpython. However, except for get_platform() - which needs some specifics to be able to support multiple tags (equality already works) - just having this patch in permits pip download and pip wheel* to work without modification to the pypa tools.

[ncoghlan]

@aixtools We've already missed the feature window for 3.8, but having a PR in review for 3.9 would likely help frame the backport discussions for the PyPA tooling.

Edit: I originally wrote "accepted for 3.9" above, but changed it as I think the PR will be valuable for discussion purposes even while it's in review, as it makes the scope of the platform support changes clear.

[aixtools]

I'll make a PR for 3.9 then. So wish it could have been seen as a fix rather than as a feature. It is something that has been overlooked since "forever". Having taken additional time I believe it is a better tag - at least I think it is easier to read. And I think I'll drop the b from b32|b64 so the fields just split into digits.

I'll add the PR reference here, when finished.

[aixtools]

PR posted: https://github.com/python/cpython/pull/15678

[aixtools]

FYI: Two months down the road. No action yet on the Phyton3-3.9 PR yet, but it does work. FYI2: I backported - manually - in my packaging for v3.6.9, v3.7.5 and v3.8.0 and I'll be testing (below) with those.

Roughly speaking - a wheel built with v3.6.9 needs to message that the wheel is not supported when the Python version changes (e.g. v3.7.5), but still be accepted if the only change is the level of AIX.

I am going to get started with the pep425tags.get_supported() part, and see where it leads from there.

There were several references to taking this to a different area. For the short term, I'll be busy here - as I understand the basics.

[aixtools]

Question for someone who understands the way 'packaging' thinks...

In ./src/pip/_internal/pep425tags.py.get_supported(), as suggested - I looked at the macosx block.

The first line for 'match' is some magic I do not readily follow (re.compile.match(arch) - abbreviated form) but assume it is looking for some generic starting point, and if found has something 'magical', if I understood the comments "get_platform()" was re-written in this file so that it returns the system running Python, rather than the system/platform that built Python. In short, it is taken as a given that 10.9 is supported by, or can use, "stuff" labeled as 10.6.

And a list is generated appending the actual arch (which I have understood to be a label for 32-bit or 64-bit.

So, it generates a list with "major == 10", and minor is the range[0..minor] and the processor name (ppc, ppc64, intel, x86_64, fat, etc) is taken care of.

So, is a list, such as:

build: 6106-1043-64 runas: 6107-1415-64 ['6107-1415-64', '6107-1341-64', '6107-1339-64', '6107-1316-64', '6107-1241-64', '6107-1228-64', '6107-1216-64', '6107-1207-64', '6107-1150-64', '6107-1141-64', '6106-1339-64', '6106-1316-64', '6106-1241-64', '6106-1228-64', '6106-1216-64', '6106-1207-64', '6106-1140-64', '6106-1115-64', '6106-1112-64', '6106-1048-64', '6106-1044-64', '6106-1043-64']

or

root@x068:[/data/prj/python/git/python3-3.6.9]python3 x1.py build: 6106-1043-64 runas: 7104-1845-64 ['7105-1914-64', '7105-1913-64', '7105-1846-64', '7105-1838-64', '7105-1837-64', '7105-1832-64', '7105-1810-64', '7105-1806-64', '7105-1731-64', '7104-1914-64', '7104-1845-64', '7104-1806-64', '7104-1720-64', '7104-1717-64', '7104-1642-64', '7104-1614-64', '7104-1543-64', '7103-1717-64', '7103-1642-64', '7103-1614-64', '7103-1543-64', '7103-1524-64', '7103-1441-64', '7103-1415-64', '7103-1412-64', '7103-1341-64', '7102-1524-64', '7102-1441-64', '7102-1415-64', '7102-1341-64', '7102-1334-64', '7102-1316-64', '7102-1245-64', '7101-1415-64', '7101-1341-64', '7101-1334-64', '7101-1316-64', '7101-1241-64', '7101-1228-64', '7101-1216-64', '7101-1207-64', '7101-1150-64', '7101-1141-64', '7100-1334-64', '7100-1316-64', '7100-1241-64', '7100-1228-64', '7100-1216-64', '7100-1207-64', '7100-1140-64', '7100-1115-64']

what the 'packaging' system is expecting?

As reference - see the "71XX" and the "61XX" as the major part, and 64 (in this case) as the arch part - and the numbers in between as minor - just not sequential. FYI: the numbers come from the system, not a hard-coded table.

Any feedback is welcome - before I try and splice this into pep425tags.

[aixtools]

renamed the issue - should this be a new issue instead?

Phase 1: is to get a proper tag for AIX - this is straightforward. If the tag returned by Cpython starts as "aix" it does not have PEP425 "smarts": THEN call _AIX_support.get_platform() for a suitable tag.

Phase 2: will be dealing with "platform" tags that AIX can provide. The goal is similar to the manylinux and macos multi-platform support (e.g., PEP513 for manylinux1, unsure if there is an INFO PEP for macos multi-platform support).

re: Phase1: I have a patched CPython I am using for Phase1 - and pip and wheel work asis for exact matches.

Re: Phase2: testing - to validate it will work - I rename a wheel e.g., mv cryptography-2.8-cp36-cp36m-AIX_5307_0747_32.whl cryptography-2.8-cp36-cp36m-AIX_7105_1731_32.whl. After this I am able to load and use the wheel that was built on AIX 5.3 even though the system loading it is from AIX 7.1. NOTE - please - the wheel is not changed, only it's file name. Note also: via my portals AIX admins are downloading, _mv_ing wheels and installing them successfully.

bdist - better wheels - are a tremendous boon to admins who either may not, or cannot, install and use a compiler.

I'll be starting a new PR.

I am ready to write some documentation or what's new - but would like some assistance with choosing a name for the News entry.

[pradyunsg]

would like some assistance with choosing a name for the News entry.

Use a .feature extension for that file. :)

[aixtools]

so news/6922.feature ?

I'll start with that in any case. Thx.

[aixtools]

OK. Not sure where the rock is, and where the hard place is - but I feel stuck between the two.

From my perspective, this discussion had the PR7375 as it's focus, but it was closed in preference of the PR in pypa/packaging https://github.com/pypa/packaging/pull/190. However, the related discussion https://github.com/pypa/packaging/issues/188 sends us back here.

I would like to see a discussion somewhere. Please specify!