fix(backend): handle client side HTTP timeouts to fix crashes of metadata-writer. Fixes #8200

[OutSorcerer]

Description of your changes:

• Handles client side timeouts (urllib3.exceptions.ReadTimeoutError) of k8s_watch.stream to prevent crashes of metadata-writer pod. Without this metadata-writer pod fails in cases when a connection error causes a client timeout. This should fix #8200.
• Adjusts the client side timeout according to the recommendations of Kubernetes Python client authors.

  It is recommended to set this [server] timeout value to a higher number such as 3600 seconds (1 hour). It is recommended to set this [client] timeout value to a lower number (for eg. ~ maybe 60 seconds).

  • The benefit of this for the metadata-writer use case is that, currently, with a client side timeout of 2000 seconds (33.3 minutes) if a connection error happens during this period, metadata-writer stops doing its job for up to 33.3 minutes. After this change metadata-writer will only be disconneted for up to the new client timeout, 60 seconds.
  • Since the client timeout is now smaller than the server timeout, if no events occur within the client timeout period, a ReadTimeoutError is thrown and caught even in the absence of errors.

• Makes client side and server side timeout configurable via environment variables.
• Updates Kubernetes Python client to the latest version.

Checklist:

• [x] You have signed off your commits
• [x] The title for your pull request (PR) should follow our title convention. Learn more about the pull request title convention used in this repository.

[google-oss-prow[bot]]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Once this PR has been reviewed and has the lgtm label, please assign ark-kun for approval. For more information see the Kubernetes Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files: - **[backend/metadata_writer/OWNERS](https://github.com/kubeflow/pipelines/blob/master/backend/metadata_writer/OWNERS)** Approvers can indicate their approval by writing `/approve` in a comment Approvers can cancel approval by writing `/approve cancel` in a comment

[google-oss-prow[bot]]

Hi @OutSorcerer. Thanks for your PR.

I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.

[hbelmiro]

/ok-to-test

[ishaan-mehta]

Is there an estimate on when this could be approved and merged, and/or is there anything I can do to help? Just curious as my deployment is running into the same issue.

This is more severe than just a pod restarting repeatedly, as when the pod is down, Kubeflow is seemingly unable to properly authorize users for namespaces.

[thesuperzapper]

@kubeflow/pipelines maintainers can we get some eyes on this important PR (It needs some work, but is important as it fixes a critical issue that prevents KFP metadata-writer working on some Kubernetes distros).

For more context, please see my comment here:

• https://github.com/kubeflow/pipelines/issues/8200#issuecomment-2457945006

But the issue is simply that some TCP sockets are timing out when we are watching Kubernetes resources from python.