search

kubernetes-sigs / jobset

JobSet: a k8s native API for distributed ML training and HPC workloads
https://jobset.sigs.k8s.io/
Apache License 2.0
152 stars 47 forks source link

Python SDK: Install Python and Java from source rather than using package manager #190

Open danielvegamyhre opened 1 year ago

danielvegamyhre commented 1 year ago

What would you like to be added: We should install Python and Java from source rather than using an environment specific package manager (apt).

Why is this needed: To improve the developer experience SDK generation (ran by make generate) and testing scripts (ran by make test ) more platform agnostic.

danielvegamyhre commented 1 year ago

/good-first-issue

k8s-ci-robot commented 1 year ago

@danielvegamyhre: This request has been marked as suitable for new contributors.

Guidelines

Please ensure that the issue body includes answers to the following questions:

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-good-first-issue command.

In response to [this](https://github.com/kubernetes-sigs/jobset/issues/190): >/good-first-issue Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.
chansuke commented 1 year ago

/assign

kannon92 commented 1 year ago

Rather than building from source, should we just have all this run in a container? Building from source is going unlease a maintenance headache as you will have to support issues related to builds on all different kinds of Linux OSes.

I look forward to this issue as my dev environment is quite flaky now as I don't have apt.

kannon92 commented 1 year ago

Just an update on this issue, I had to change the python sdk to use a container. @danielvegamyhre and I think the way forward should be using containers to generate these clients rather than relying on package managers or building from source.

so the client go one should ideally also use a container to generate sdk

jedwins1998 commented 8 months ago

@chansuke are you still working on this? I will assign it to myself if not.

chansuke commented 8 months ago

@jedwins1998 I'm not working.You can take it🫡

jedwins1998 commented 8 months ago

/assign

epicseven-cup commented 2 months ago

/assign

epicseven-cup commented 2 months ago

The end of life for openjdk-11 will be October 2024 and openjdk-17 is superseded on openjdk's website https://jdk.java.net/17/.

Will grabbing openjdk-17 from other vendors such as Microsoft, Oracle.. be fine?

danielvegamyhre commented 2 months ago

The end of life for openjdk-11 will be October 2024 and openjdk-17 is superseded on openjdk's website https://jdk.java.net/17/.

Will grabbing openjdk-17 from other vendors such as Microsoft, Oracle.. be fine?

Yes that should be fine.

epicseven-cup commented 2 months ago

Also just a quick question since python sdk is being generated from openapi-generator-cli I'm assuming that we don't need to update that to install from source anymore, just want to double check if that is the correct

danielvegamyhre commented 2 months ago

Just an update on this issue, I had to change the python sdk to use a container. @danielvegamyhre and I think the way forward should be using containers to generate these clients rather than relying on package managers or building from source.

so the client go one should ideally also use a container to generate sdk

@epicseven-cup see this comment thread above, I think it would be better for portability if wecontainerize SDK code gen rather than building from source as the issue description states.