kubescape / storage

Apache License 2.0
1 stars 6 forks source link

feat: increase max request size #5

Closed vladklokun closed 1 year ago

vladklokun commented 1 year ago

What this PR changes?

This commit increases the maximum Send and Recieve request sizes in the API server and the etcd client.

This allows to process and store objects larger than the the default K8s and etcd limits (in the ballpark of 2 MiB).

Since TransportConfig does not expose request limits for etcd, the easiest way to change the etcd client limits is vendoring the module and editing the etcd client factory function.

Notes for the Reviewer

The changes themselves are small, but this PR introduces vendoring, so the stats look enormous.

gitguardian[bot] commented 1 year ago

⚠️ GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
| GitGuardian id | Secret | Commit | Filename | | | -------------- | ------------------------- | ---------------- | --------------- | -------------------- | | [5955901](https://dashboard.gitguardian.com/incidents/5955901?occurrence=87130197) | Generic High Entropy Secret | ecd1d21cb256cd2feccc09ce826c4e4be635590b | vendor/k8s.io/apiserver/pkg/apis/config/v1/types.go | [View secret](https://github.com/kubescape/storage/commit/ecd1d21cb256cd2feccc09ce826c4e4be635590b#diff-7d17d38d8494d9cd2a2d44f21ee6c18f8806844a7f263362f96028a40c3dcbd5R42) | | [5955901](https://dashboard.gitguardian.com/incidents/5955901?occurrence=87130198) | Generic High Entropy Secret | ecd1d21cb256cd2feccc09ce826c4e4be635590b | vendor/k8s.io/apiserver/pkg/apis/config/types.go | [View secret](https://github.com/kubescape/storage/commit/ecd1d21cb256cd2feccc09ce826c4e4be635590b#diff-b07c9b71f21d991bd38ba799efcd47124cc832503e6d20e7f7bd258abb6da88fR42) |
🛠 Guidelines to remediate hardcoded secrets
1. Understand the implications of revoking this secret by investigating where it is used in your code. 2. Replace and store your secrets safely. [Learn here](https://blog.gitguardian.com/secrets-api-management?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) the best practices. 3. Revoke and [rotate these secrets](https://docs.gitguardian.com/secrets-detection/detectors/generics/generic_high_entropy_secret#revoke-the-secret?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). 4. If possible, [rewrite git history](https://blog.gitguardian.com/rewriting-git-history-cheatsheet?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment). Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data. To avoid such incidents in the future consider - following these [best practices](https://blog.gitguardian.com/secrets-api-management/?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) for managing and storing secrets including API keys and other credentials - install [secret detection on pre-commit](https://https://docs.gitguardian.com/ggshield-docs/integrations/git-hooks/pre-commit?utm_source=product&utm_medium=GitHub_checks&utm_campaign=check_run_comment) to catch secret before it leaves your machine and ease remediation.

🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

Our GitHub checks need improvements? Share your feedbacks!

vladklokun commented 1 year ago

warning GitGuardian has uncovered 2 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components. mag_right Detected hardcoded secrets in your pull request hammer_and_wrench Guidelines to remediate hardcoded secrets

owl GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.Our GitHub checks need improvements? Share your feedbacks!

This is a false positive in the vendored k8s.io/apiserver package. It contains an example of a secret in a structure’s doc comment.

vladklokun commented 1 year ago

@dwertent

I can’t add a comment using native Github tools, because the changed vendored file is in the latter part of the 5000+ changed files.

The third place where we change the limits is in vendor/k8s.io/apiserver/pkg/storage/storagebackend/factory/etcd3.go:228, which looks like this:

var newETCD3Client = func(c storagebackend.TransportConfig) (*clientv3.Client, error) {
    // Omitted for brevity...
    // clientv3 is the etcd client
    cfg := clientv3.Config{
        DialTimeout:          dialTimeout,
        DialKeepAliveTime:    keepaliveTime,
        DialKeepAliveTimeout: keepaliveTimeout,
        DialOptions:          dialOptions,
        Endpoints:            c.ServerList,
        TLS:                  tlsConfig,
        Logger:               etcd3ClientLogger,

        // The part that changed
        MaxCallRecvMsgSize: 5 * 1024 * 1024,
        MaxCallSendMsgSize: 5 * 1024 * 1024,
    }

    return clientv3.New(cfg)

The reason why we have to change the config of this factory function is because I could not find how the APIServer machinery exposes the limits as part of the backend storage configuration options. The data structures I have seen do not contain the fields that control the message size.