Nephio is a Kubernetes-based automation platform for deploying and managing highly distributed, interconnected workloads such as 5G Network Functions, and the underlying infrastructure on which those workloads depend.
Apache License 2.0
93
stars
52
forks
source link
Porch limits package size to the Kubernetes (etcd) request limit because of its reliance on PackageRevisionResources #689
Original issue comments:
Comment user: https://github.com/mortent
Comment created at: 2023-02-19T23:44:54Z
Comment last updated at: 2023-02-19T23:44:54Z
Comment body: So Porch doesn't actually use CRDs for PackageRevisionResources and these resources are never stored in etcd. Instead, Porch uses an aggregated apiserver that stores the resources in a package directly in git or oci. But as your example demonstrates, there is still a limit to the size of the manifests within a package. I'm not sure if this limit can be easily changed, so we would have to look into that.
Do you have examples of real-life packages that that are larger than 3mb? That is a lot of yaml... We have discussed these limitations on the sizes of packages in the past and we have thought that packages would usually not become this big and if they do, they should probably be broken up into multiple packages.
Comment user: https://github.com/tliron
Comment created at: 2023-02-20T02:27:45Z
Comment last updated at: 2023-02-20T02:27:45Z
Comment body: > So Porch doesn't actually use CRDs for PackageRevisionResources and these resources are never stored in etcd.
Thanks, but I'm confused by this comment. I linked to the source code for the CRD PackageRevisionResources. In my use of Porch I see these CRs being generated for every package revision and each one contains YAML dumps of all resources in the package. And of course all Kubernetes resources are stored in etcd.
Comment user: https://github.com/johnbelamaric
Comment created at: 2023-02-20T05:02:08Z
Comment last updated at: 2023-02-20T05:02:08Z
Comment body: The schema may be there but it's backed by an aggregated API server, not a CR in etcd. The PackageRevision is a CR but not PackageRevisionReources.
The request limit is on the HTTP side. If the size is a real issue we likely can work around that with some fancy things input request paging if some sort. You'll just have to use kpt rpkg as the client not kubectl for those extra big packages.
Comment user: https://github.com/tliron
Comment created at: 2023-02-20T05:34:26Z
Comment last updated at: 2023-02-20T05:34:26Z
Comment body: Again, I'm confused. PackageRevisionResources is definitely registered as a CRD and there are CRs for it, one per package revision. Without kpt alpha rpkg I can access them via kubectl get packagerevisionresources. It's the creation of these CRs which results in the error above. It seems like both are you saying that this resource type is not necessary. Then perhaps it should be removed and then this problem would disappear?
Comment user: https://github.com/johnbelamaric
Comment created at: 2023-02-20T07:41:23Z
Comment last updated at: 2023-02-20T07:41:23Z
Comment body: Access via kubectl is expected, it's just not stored in etcd.
Comment user: https://github.com/tliron
Comment created at: 2023-02-20T08:02:52Z
Comment last updated at: 2023-02-20T08:02:52Z
Comment body: Oh, I see, I assumed that even with aggregation Kubernetes would still buffer the data on etcd as a CR.
The error is still there, though, indeed because of the request size limit. So, is this the best API for accessing package data? It's not very common to have 3MB packages, but there are big applications out there with a lot of data put into ConfigMaps, as well as elaborate CRDs. I don't think it's good to impose a size limit due to this limitation (which can differ per Kubernetes distribution). It seems a good candidate for a streaming transport, e.g. gRPC.
Original issue URL: https://github.com/kptdev/kpt/issues/3829 Original issue user: https://github.com/tliron Original issue created at: 2023-02-19T18:26:46Z Original issue last updated at: 2023-02-20T19:44:09Z Original issue body: 1. Porch requires that all resources in a package be stored as YAML manifests in PackageRevisionResources.
I am attaching a package that demonstrates this limitation.
network-function.zip
Original issue comments: Comment user: https://github.com/mortent Comment created at: 2023-02-19T23:44:54Z Comment last updated at: 2023-02-19T23:44:54Z Comment body: So Porch doesn't actually use CRDs for PackageRevisionResources and these resources are never stored in etcd. Instead, Porch uses an aggregated apiserver that stores the resources in a package directly in git or oci. But as your example demonstrates, there is still a limit to the size of the manifests within a package. I'm not sure if this limit can be easily changed, so we would have to look into that.
Do you have examples of real-life packages that that are larger than 3mb? That is a lot of yaml... We have discussed these limitations on the sizes of packages in the past and we have thought that packages would usually not become this big and if they do, they should probably be broken up into multiple packages.
Comment user: https://github.com/tliron Comment created at: 2023-02-20T02:27:45Z Comment last updated at: 2023-02-20T02:27:45Z Comment body: > So Porch doesn't actually use CRDs for PackageRevisionResources and these resources are never stored in etcd.
Thanks, but I'm confused by this comment. I linked to the source code for the CRD
PackageRevisionResources. In my use of Porch I see these CRs being generated for every package revision and each one contains YAML dumps of all resources in the package. And of course all Kubernetes resources are stored in etcd.Comment user: https://github.com/johnbelamaric Comment created at: 2023-02-20T05:02:08Z Comment last updated at: 2023-02-20T05:02:08Z Comment body: The schema may be there but it's backed by an aggregated API server, not a CR in etcd. The PackageRevision is a CR but not PackageRevisionReources.
The request limit is on the HTTP side. If the size is a real issue we likely can work around that with some fancy things input request paging if some sort. You'll just have to use kpt rpkg as the client not kubectl for those extra big packages.
Comment user: https://github.com/tliron Comment created at: 2023-02-20T05:34:26Z Comment last updated at: 2023-02-20T05:34:26Z Comment body: Again, I'm confused.
PackageRevisionResourcesis definitely registered as a CRD and there are CRs for it, one per package revision. Withoutkpt alpha rpkgI can access them viakubectl get packagerevisionresources. It's the creation of these CRs which results in the error above. It seems like both are you saying that this resource type is not necessary. Then perhaps it should be removed and then this problem would disappear?Comment user: https://github.com/johnbelamaric Comment created at: 2023-02-20T07:41:23Z Comment last updated at: 2023-02-20T07:41:23Z Comment body: Access via kubectl is expected, it's just not stored in etcd.
See https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/
Comment user: https://github.com/tliron Comment created at: 2023-02-20T08:02:52Z Comment last updated at: 2023-02-20T08:02:52Z Comment body: Oh, I see, I assumed that even with aggregation Kubernetes would still buffer the data on etcd as a CR.
The error is still there, though, indeed because of the request size limit. So, is this the best API for accessing package data? It's not very common to have 3MB packages, but there are big applications out there with a lot of data put into ConfigMaps, as well as elaborate CRDs. I don't think it's good to impose a size limit due to this limitation (which can differ per Kubernetes distribution). It seems a good candidate for a streaming transport, e.g. gRPC.