Open valen-mascarenhas14 opened 8 months ago
Our goal is to make workflow-controller, argocli, and argoexec images multi-arched.
We propose to add this platform: [ linux/amd64, linux/arm64, linux/ppc64le ] to the following line in the YAML file at link. for enhancing the image release pipeline to include ppc64le architecture support
The Argo image serves as a crucial dependency for facilitating seamless integration and testing of Kubeflow pipelines on ppc64le
Can you point me to where KFP relies on it?
Sure @terrytangyuan This link shows how do we setup our cluster for testing .
This link shows the steps to install Argo .
This is the yaml file that has both the dependencies (https://github.com/argoproj/argo-workflows/releases/download/v3.5.2/install.yaml)
I meant documentation around the requirement of ppc64le.
hey @terrytangyuan we are trying to enable ppc64le pipeline in our local prow cluster , details are here in this issue- https://github.com/GoogleCloudPlatform/oss-test-infra/issues/1972#issuecomment-1842798928
I see. If it's just as simple as adding one more platform in the workflow, then I don't see why not supporting it.
Would you like to make the change and test if it works in your fork before submitting the PR?
yes we could , requesting @valen-mascarenhas14 to try it once via fork and then submit PR here.
I see. If it's just as simple as adding one more platform in the workflow, then I don't see why not supporting it.
IMO, I would probably reject this for the same reasons as RISC-V in #12067 (i.e. lack of usage and required maintenance and build time, same reason why Argo CD is reticent to add more) and recommend an unofficial build as I did https://github.com/argoproj/argo-workflows/pull/12067#issuecomment-1783603520
hey @terrytangyuan we are trying to enable ppc64le pipeline in our local prow cluster , details are here in this issue- GoogleCloudPlatform/oss-test-infra#1972 (comment)
I'm a little confused here, that's not in k8s test-infra
, that's for GCP. Are Google and Kubeflow planning to support ppc64le officially? (in particular, GCP supporting IBM PowerPC seems odd) If so, that could shift my opinion, but that wasn't really clear to me from the issue
hi @agilgur5 - on below , there are multiple kubeflow components already supported on ppc64le , more details in this umbrella issue - https://github.com/kubeflow/kubeflow/issues/6684
I'm a little confused here, that's not in k8s test-infra, that's for GCP. Are Google and Kubeflow planning to support ppc64le officially? (in particular, GCP supporting IBM PowerPC seems odd) If so, that could shift my opinion, but that wasn't really clear to me from the issue
@terrytangyuan I've created a fork and trying building the ppc64le specific argo-workflow images . It successfully builds the images on ppc64le . Here's the workflow link .
I'll go ahead and raise a PR if all looks good
Let's get @agilgur5's agreement before submitting the PR.
Agree with @agilgur5
IMO, I would probably reject this for the same reasons as RISC-V in https://github.com/argoproj/argo-workflows/pull/12067 (i.e. lack of usage and required maintenance and build time, same reason why Argo CD is reticent to add more) and recommend an unofficial build as I did https://github.com/argoproj/argo-workflows/pull/12067#issuecomment-1783603520
IMO align with @agilgur5 If there is a significant number of Argo Workflow users requesting a certain architecture, we can add it to the Argo Workflow build process. If not, users can fork and build the image, then update the image path in an issue so that other users can use it.
If not, users can fork and build the image, then update the image path in an issue so that other users can use it.
Yea in https://github.com/argoproj/argo-workflows/pull/12067#issuecomment-1783603520 I even suggested that we could host such unofficial builds for less common architectures in argoproj-labs
as well.
Those can be automated to run on every release of Workflows. That doesn't necessarily even require a fork (per se), just a CI/GitHub Actions process
there are multiple kubeflow components already supported on ppc64le , more details in this umbrella issue - https://github.com/kubeflow/kubeflow/issues/6684
If I'm reading that issue correctly and some of the affiliations here and there, this sounds to be driven exclusively & entirely by IBM? RISC-V is more open than PowerPC (and both are RISC ISAs), so if we're going to support one, it would make sense to support both as experimental. There's still quite a bit more phases therein too and no real user surveys provided as supporting evidence. Some of the other upstream deps seem to have expressed the same concerns.
@agilgur5 thanks for looking into this.
IMO this is also good for widening the argo community to a wider market
I don't speak for or represent Argo (my words are my own), but to be fair, it is one of the largest CNCF projects already.
by not pushing those demands to tekton as alternative
Other projects in the ecosystem like Tekton supporting PowerPC and having user adoption of that are good arguments. For the latter, no data has been presented. The former took me a bit to find (there are no binaries in the GH releases), but Tekton does seem to make builds for PowerPC (but not RISC-V?). Although idk if all its components support PowerPC.
Related is that Kubeflow Pipelines is still on an old, unsupported version of Argo (https://github.com/kubeflow/pipelines/pull/9301, https://github.com/kubeflow/pipelines/issues/8935, https://github.com/kubeflow/pipelines/issues/8942, etc). Even if Argo were to start building for PowerPC, KFP still wouldn't be able to use those images as they'd only be for supported Argo versions. I'm unsure if the KFP Tekton fork is up-to-date on a version of Tekton that supports PowerPC.
Kubeflow also doesn't yet fully support arm64
(https://github.com/kubeflow/kubeflow/issues/2337) (which has been increasingly popular due to Apple silicon).
Is this what you are looking for?
No. I mentioned that issue myself in my previous comment. It does not have any user surveys. Upvotes are not particularly nuanced as a signal (also, there are issues in Argo with many more upvotes if that measure were to be exclusively used, this one only has 4 upvotes. the Kubeflow issue is also for all of Kubeflow, not KFP specifically either as KFP users are a subset of Kubeflow users -- I have seen many that don't use KFP).
As an example, Argo does have a roughly yearly survey that has size & scale of users mentioned
of non-IBM Kubeflow community members
As there is no survey data or similar, that statement is difficult to quantify. How many organizations are using or interested in PowerPC and KFP (or Argo in a different fashion) on PowerPC? There is no data on that and nearly all comments are from IBM.
Also there still hasn't been a counter-argument presented for why PowerPC and other less common architectures could not be hosted in an unofficial builds repo. As I wrote above, that could be hosted within argoproj-labs
. IBM could also host one for IBM-led archs (including s390x
)
@agilgur5 one comment you made about argo having good acceptance already and that being kind of a counter to the need for adding Power support, but the whole point of a graduated CNCF project is to make it accessible to all CNCF community members for use and for contribution. And, as with any open source project, different end users or middlemen in the overall process contribute based on their own needs. In IBMs case, the feature that we contribute and support is typically support for additional architectures like ppc64le and s390x. Is the argument here that supported architectures are not a feature of the project? Communities typically evolve to serve all members and there are a lot of features that IBM might not need or endorse, but generally we wouldn't block them simply because we didn't think they were necessary or have proof that they were "widely enough" used, whatever that litmus test might be.
And, you do point to one of the challenges that IBM does have with open source communities - our end users (typically customers) are not very interactive in open source communities and thus we wind up as their proxies, which is more painful than you might expect. ;) But part of the challenge here is that we have built an ecosystem of $xxxM of product into broader ecosystems that involve billions of dollars in things like banking and finance or health care, etc. etc. Odds are the users of Power systems include every open source developer with a bank account or credit card, because that's the type of workloads that IBM Power and IBM Z can often be found in, running the world's largest and often most secure installations.
So, IBM voted, on behalf of its customers, via CNCF support for ArgoCD either directly or with Red Hat as a partner, we voted with Red Hat when we jointly created and released GitOps for Power, we vote when we have multiple developers engaging with open source communities and spend our dollars because our customers tell us that they want these capabilities.
I'm honestly not sure how we can get you the information you requested on votes - ideally someone like IDC or Gartner would have those connections into customers, but we wouldn't be out here advocating for changes like this if they weren't requested by what will be your own end users. ;)
Finally, with your argument of hosting things for Power and Z in cloned repositories, consider that we've worked with some >40,000 open source communities on Power in the past year alone. Cloning all of those projects and maintaining them with IBMers would be totally ridiculous in terms of cost and effort and the complete and total opposite of the point of open source. As you are probably aware, IBM has contributed to open source since the 90's at least, and probably even since the 60's, and in a volume proportionally larger than most other large companies throughout the years. We believe in open source, we believe in collaborations like the Linux Foundation, Apache Foundation, CNCF, etc. etc. Trying to clone all of GitHub for ppc64le and for s390x would be a massive waste of world brainpower, disk space and compute power. ;) What seems hard at the beginning but is ultimately easier is finding the right way to collaborate and embrace the promise of open source.
Sorry for the soap-boxy response, but I wanted to provide a little bit of a view from "the other side." :)
thanks for listening!
Is the argument here that supported architectures are not a feature of the project?
No, the question is and has been (this issue was opened as a feature request and still is one) why should those architectures be officially supported as part of the core? Especially so when we don't have any tests on those architectures nor contributors actively running on those architectures (as another form of testing). We have either or both for the currently supported architectures. Again as was mentioned above, other projects said very similar things (e.g. https://github.com/pyca/cryptography/issues/7723)
As was already written, every feature incurs a maintenance burden, and Workflows already has some efforts ongoing to reduce that burden and move things into user-land (e.g. #6943, #12694) as well as get more active contributors (c.f. the Sustainability Effort). Every project has to make a decision about the trade-offs of and priority of a feature, and if it can be implemented easily in user-land, that substantially decreases any priority or rationale for it to be supported in the core. As I wrote there, CD is also reticent to add more architectures for a similar trade-off of lack of usage vs. required maintenance and build time.
Finally, with your argument of hosting things for Power and Z in cloned repositories
That is similarly not what was said here. A separate build repo was suggested, similar to Node.js's https://github.com/nodejs/unofficial-builds/, which is a significantly larger project than Argo.
would be a massive waste of world brainpower, disk space and compute power.
That suggestion would in fact waste less resources, compared to builds for every commit on main
for infrequently used architectures. Making all builds more complex and longer for infrequently used architectures is an argument against this feature.
Similar can be said about core maintenance and support for an untested build, as opposed to an explicit unofficial builds repo, which can also have separate maintainers etc.
I'm honestly not sure how we can get you the information you requested
As was already written, given that OSS projects, including CNCF projects like Argo and many others, are able to survey their users, I would think that IBM would definitely have the resources to do the same thing.
Sorry for the soap-boxy response
For context, I am a volunteer who has put thousands of hours of unpaid time into OSS communities, including Argo & CNCF (and all of that is readily available, publicly accessible information). A significant portion of OSS is run by passionate volunteers & hobbyists. And that's about all I'll say on that point, it is preferred to keep things not personal, on topic, and focused on problem-solving and concrete data. Marketing statements by corporations are also generally discouraged and CNCF requires vendor neutrality (parts of your comment are definitely pretty close to the line).
As I have done a few times now, I would ask that all questions asked and concerns raised be addressed. Multiple IBM employees have yet to do so.
@agilgur5 - thanks again; let's summarize where we are:
argoproj-labs
(as well as approve docs PRs to link to it).
would be a massive waste of world brainpower, disk space and compute power.
That suggestion would in fact waste less resources, compared to builds for every commit on
main
for infrequently used architectures. Making all builds more complex and longer for infrequently used architectures is an argument against this feature.
To put some concrete numbers to this, from a recent build on main
:
amd64
builds take between 2m - 4marm64
builds take between 18m - 35mSo cross-compilation for arm64
already takes around an order of magnitude longer. I imagine that less used architectures may take even longer (less cross-compilation optimizations).
Those are very real numbers that do non-trivially affect the length of our existing release process (and I've waited on the arm64
builds more than once; at least every time I release a patch version I notice this)
@lehrig and others from RH/IBM - Could you reach out to me via RH/IBM Slack?
Related is that Kubeflow Pipelines is still on an old, unsupported version of Argo (https://github.com/kubeflow/pipelines/pull/9301, https://github.com/kubeflow/pipelines/issues/8935, https://github.com/kubeflow/pipelines/issues/8942, etc). Even if Argo were to start building for PowerPC, KFP still wouldn't be able to use those images as they'd only be for supported Argo versions.
Quick note - this has been fixed and KFP upgraded to 3.4+ https://github.com/kubeflow/pipelines/pull/10568.
Summary
Requesting the argo team create official images of argo-workflows for ppc64le?
Use Cases
Our use case involves building Argo-workflow on a Kubernetes cluster with ppc64le architecture. The Argo image serves as a crucial dependency for facilitating seamless integration and testing of Kubeflow pipelines on ppc64le
Proposal
Change build infrastructure to build ppc64le variants of argo-workflows.
Message from the maintainers:
Love this enhancement proposal? Give it a 👍. We prioritise the proposals with the most 👍.