search

kubebb / core

A declarative component lifecycle management platform
https://kubebb.github.io/website
Apache License 2.0
8 stars 9 forks source link

🌱 Reduce memory and service pressure on core operator #373

Closed 0xff-dev closed 1 month ago

0xff-dev commented 9 months ago

Currently, we are parsing the chart package within the operator and have encountered the following problems.

  1. The operator occupies a substantial amount of memory. https://github.com/kubebb/core/pull/366#issuecomment-1751653987 We have restricted the parsing to a maximum of 5 chart packages concurrently. During testing with Bitnami's repository, we determined that it consumes roughly 1.5GB of memory.

  2. The overall operation time is excessively lengthy. Due to the extensive number of chart packages in the Bitnami repository, each with numerous versions, and the requirement to create over 6000 configmaps, the process took over an hour during testing on a local kind cluster. Moreover, the cluster experienced issues and ceased to function, making this problem even more critical.

  3. It is uncertain how large the values.yaml and README files are for each chart package, and there is a possibility that they might exceed the configmap limit.

For the time being, it is advisable to eliminate the service responsible for parsing the chart packages from the operator. Instead, we can handle this task either within the bff-server or by creating a separate service. This service would handle requests for specific chart packages and retrieve the relevant content. Implementing caching in the server can also enhance the processing speed of subsequent requests.

@bjwswang @Abirdcfly @Carrotzpc @dayuy @littsummer

bjwswang commented 9 months ago

Like @0xff-dev said, it's not reasonable to manage so many configmaps only for caching values.yaml and README.md(especially) which has unpredictable size. This may leads the core operator to crash which is happening right now.

@Carrotzpc any thoughts? bff-server or a seperate service (only for querying chart content)

Abirdcfly commented 9 months ago

inspired by #342, maybe we can fetch all helm charts to minio? Then bff-server read these file from minio.