Install Python 3 on all OpenJ9 build machines

[nbhuiyan]

eclipse/omr#3675 introduces a new options processing framework for the compiler. It includes the options-gen script run at build time to generate files that are then used to populate various data structures.

The script requires at minimum Python 3.2.

After eclipse/omr#3675 is merged, an OpenJ9 PR will be opened to use the new options processing framework in the OpenJ9 JIT. This would add Python 3 as a build requirement, and hence all OpenJ9 build machines would require Python 3.2 and up to build the JIT component.

[nbhuiyan]

@jdekonin fyi

[DanHeidinga]

I would strongly prefer not to grow the set of tools required to build OpenJ9 as it creates a barrier to entry and makes it more difficult to port OpenJ9 to new platforms.

There's recently been a lot of work to remove dependencies on particular perl modules by rewriting the functionality of testkitgen in Java (thanks @fjeremic & @renfeiw & others) as requiring more / varied tools makes it harder to build OpenJ9 on all platforms. (zOS was the issue here).

Are there other ways to achieve this functionality without introducing another tooling dependency?

[smlambert]

Presume this also implies that all AdoptOpenJDK build machines need this prereq too? if so, that is less than optimal...

[DanHeidinga]

FYI @0xdaryl @mstoodle @pshipton

It would be good to have a discussion on this item and see if we can make a plan for how to move it forward that addresses the benefits of the OMR PR without adding the new tooling dependencies.

Is there a time this week we can get on a call together? Offering my webex if that helps.

[mstoodle]

Happy to discuss. I'm off today (comment timing notwithstanding), but Thursday afternoon looks pretty clear at the moment with only a few taken slots, if that works for others.

imo, we need to make it easier for this kind of thing to go forward. In this modern age, pushing back on a Python build-time dependency feels a bit arbitrary when we just went through the process to add a much less widely available 3rd party dependency (CUDA). There are 1000+ lines of Python in the OMR project at this point, so pushing back would mean helping them to rewrite that code and its dependent Python modules in...C++, I suppose, would be the proposal?

[DanHeidinga]

@mstoodle I think there's a clear difference between an optional build dependency that is used if available and a required tool.

I don't know if you were aware of the recent pain related to perl modules that was blocking testing on zOS but a lot of effort went into removing dependencies on perl modules that don't work on non-X86 architectures. It's why testkitgen has been ported to Java.

Every tool & every dependency adds a non-zero cost to getting set up to build openj9, especially when porting to new architectures, and introduces additional work for the infra teams at openj9 and AdoptOpenJDK to manage versions and updates across the broad range of supported OSes.

Let's be careful not to frame this as pushing back on a Python. A more accurate framing is to limit the build time dependencies so it's easier to get started with OpenJ9, maintain the infrastructure, and to port the project.

There's lot of options here including rewriting in C++, generating the Options ahead of time and checking in the generated files, and likely others.

[DanHeidinga]

I've sent a meeting invite to @mstoodle @nbhuiyan @smlambert @pshipton & @0xdaryl. If there are others that want or should be included, please let me know

[mstoodle]

Let's be careful not to frame this as pushing back on a Python . A more accurate framing is to limit the build time dependencies so it's easier to get started with OpenJ9, maintain the infrastructure, and to port the project.

...except for the fact that Python is the precise question we are considering here.

I know there's been pain with other dependencies, and your concerns are certainly valid ones in general, but there is a specific request being asked and (imo) we should consider that request on its merits which may not (or may) have all the same characteristics as other similar requests.

[DanHeidinga]

With all due respect, that ignores the larger problems each new dependency adds. So yes, let's look at the merits of adding python, a new dependency, in light of both the benefits the project gets and in the costs it pays.

No dependency is free and we must be mindful that the cost of supporting all the dependencies required to build the project is certainly greater than the sum the parts.

[mstoodle]

I'm not suggesting we ignore anything in reaching a conclusion. I only want to make a specific decision (based on all the available and applicable information).

I was worried, given the original framing of the response, that part of the decision seemed to have been already made based on recent experience with perl, and it wasn't clear to me that the earlier experience was necessarily verbatim transferable. However, re-reading what you wrote, I may have interpreted more than I should have, so if I contributed to taking this conversation off the rails, I apologize. But I think we're all driven by trying to make the right decisions, so at least our hearts are in the right place even if my words are sometimes unintentionally fan flames.

[DanHeidinga]

Notes from yesterdays discussion:

• Python for command line options is the first step in a bigger plan to generate content at OMR.
• JITBuilder also has a python dependency.
• Python is already installed on most machines but is at the 2.7 level.
• Infrastructure / releng support for managing these tools is a significant concern as there are fewer infra / releng members across the projects than there are other developers.
• OpenJDK doesn't require the broad set of additional tools OpenJ9 does (or will in the near future): Nasm, OpenSSL, cmake, protobuf, perl, .... These are barriers to entry for new developers and complicate adoption.
• OpenJDK has standardized on autotools for build and java for anything needed to be generated. Why is requiring python more acceptable for OMR than using Java?

As a next step, Daryl will write up a more detailed plan of where OMR is looking to introduce Python dependencies at the OMR project and ensure that plan has broad support in that community.

OpenJ9 will engage in those discussions and determine next steps once the OMR decisions have been made.

These are my very rough notes from the call. If there's anything inaccurate or that you'd like to expand on, please do so.

[0xdaryl]

See https://github.com/eclipse/omr/pull/3675#issuecomment-530630546 for more context around the decision to leverage Python to facilitate integrating the OMR compiler with downstream projects.

[DanHeidinga]

For context, there was discussion on how to update OpenJ9's command line processing when the -XX:PrintFlags option was added - see detailed discussion starting in: https://github.com/eclipse/openj9/pull/9567#issuecomment-631125598

It would be good if any future options work - either in OMR or OpenJ9 - take that discussion and general design direction into account.