search

kubernetes / client-go

Go client for Kubernetes.
Apache License 2.0
8.79k stars 2.91k forks source link

Use stream decoding in List calls #1224

Closed tareksha closed 1 year ago

tareksha commented 1 year ago

Hi,

Current List functions in the clients use a bulk buffering: the whole payload is fetched into an an in-memory buffer and then the buffer is decoded into the result type. Example from pods client:

result = &v1.Pod{}
c.client.Get().
Namespace(c.ns).
Resource("pods").
Name(name).
VersionedParams(&options, scheme.ParameterCodec).
Do(ctx).
Into(result)

This is very inefficient if there are many object to return, even worse when the objects themselves are big.

The underlying RESTClient already provides Stream() that can be used for streaming the payload without buffering everything in-memory. If we combine this with json stream decoding functionality we can practically eliminate the infamous memory peaks that are observed when controllers start a standard list-and-watch sequence. The short-lived huge buffers won't be needed.

I would imagine the alternative approach to look like this:

stream, err = c.client.Get().
    Namespace(c.ns).
    Resource("pods").
    Name(name).
    VersionedParams(&options, scheme.ParameterCodec).
    Stream(ctx)
if err != nil {
    return nil, err
}
defer stream.Close()
result = &v1.Pod{}
jsonDecoder := json.NewDecoder(stream)
// .. some schema configurations for decoder
err = jsonDecoder.Decode(result)
if err != nil {
    return nil, err
}
return result, nil

This is a general problem in clients generated for custom resource types as well. The kubernetes clients as a whole are inefficient when it comes to listing data.

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

k8s-triage-robot commented 1 year ago

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

You can:

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

k8s-ci-robot commented 1 year ago

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to [this](https://github.com/kubernetes/client-go/issues/1224#issuecomment-1620309114): >The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs. > >This bot triages issues according to the following rules: >- After 90d of inactivity, `lifecycle/stale` is applied >- After 30d of inactivity since `lifecycle/stale` was applied, `lifecycle/rotten` is applied >- After 30d of inactivity since `lifecycle/rotten` was applied, the issue is closed > >You can: >- Reopen this issue with `/reopen` >- Mark this issue as fresh with `/remove-lifecycle rotten` >- Offer to help out with [Issue Triage][1] > >Please send feedback to sig-contributor-experience at [kubernetes/community](https://github.com/kubernetes/community). > >/close not-planned > >[1]: https://www.kubernetes.dev/docs/guide/issue-triage/ Instructions for interacting with me using PR comments are available [here](https://git.k8s.io/community/contributors/guide/pull-requests.md). If you have questions or suggestions related to my behavior, please file an issue against the [kubernetes/test-infra](https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:) repository.