search

openyurtio / openyurt

OpenYurt - Extending your native Kubernetes to edge(project under CNCF)
https://openyurt.io
Apache License 2.0
1.7k stars 398 forks source link

[Question] Does yurthub support to local proxy for customresourcedefinitions (CRDs) ? #2121

Open obitoquilt opened 1 month ago

obitoquilt commented 1 month ago

What happened:

request for /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 in offline env, but got customresourcedefinitions.apiextensions.k8s.io "" not found error

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

others /kind question

rambohe-ch commented 1 month ago

@obitoquilt Thank you for raising the issue. I have verified the request /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 in my environment, and the crds have been cached on local disk.

I think the reason of this issue is that you have not configured the cache-agents field of yurthub. For cache configuration details, you can reference the yurthub faq No. 13: https://openyurt.io/docs/faq/yurthub/

obitoquilt commented 1 month ago

@obitoquilt Thank you for raising the issue. I have verified the request /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 in my environment, and the crds have been cached on local disk.

I think the reason of this issue is that you have not configured the cache-agents field of yurthub. For cache configuration details, you can reference the yurthub faq No. 13: https://openyurt.io/docs/faq/yurthub/

Thanks for your reply. the crds also have been cached on my local disk, but when i disconnect network on edge node (it cannot connect to api-server), re-request for /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 and got customresourcedefinitions.apiextensions.k8s.io "" not found error.

rambohe-ch commented 1 month ago

@obitoquilt Thank you for raising the issue. I have verified the request /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 in my environment, and the crds have been cached on local disk. I think the reason of this issue is that you have not configured the cache-agents field of yurthub. For cache configuration details, you can reference the yurthub faq No. 13: https://openyurt.io/docs/faq/yurthub/

Thanks for your reply. the crds also have been cached on my local disk, but when i disconnect network on edge node (it cannot connect to api-server), re-request for /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 and got customresourcedefinitions.apiextensions.k8s.io "" not found error.

@obitoquilt On my environment, crds can be listed when network offline. would you like to show me the details of crds(like crds filepath, etc.) caches on your local disk?

obitoquilt commented 1 month ago

@obitoquilt Thank you for raising the issue. I have verified the request /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 in my environment, and the crds have been cached on local disk. I think the reason of this issue is that you have not configured the cache-agents field of yurthub. For cache configuration details, you can reference the yurthub faq No. 13: https://openyurt.io/docs/faq/yurthub/

Thanks for your reply. the crds also have been cached on my local disk, but when i disconnect network on edge node (it cannot connect to api-server), re-request for /apis/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500 and got customresourcedefinitions.apiextensions.k8s.io "" not found error.

@obitoquilt On my environment, crds can be listed when network offline. would you like to show me the details of crds(like crds filepath, etc.) caches on your local disk?

/etc/kubernetes/cache/cilium-agent/:

the content of ciliumcidrgoups.cilium.io file is:

{"apiVersion": "apiextensions.k8s.io/v1", "kind": "PartialObjectMetadata", "metadata": {"creationTimestamp": ""}}

BTW, the version used is v1.4.0 Thanks for any reply.

rambohe-ch commented 1 month ago

@obitoquilt would you like to upload the details logs of yurthub component on this node? I think maybe we can find some clues in the logs.

obitoquilt commented 1 month ago

@obitoquilt would you like to upload the details logs of yurthub component on this node? I think maybe we can find some clues in the logs.

Yes. Here are the error logs of Yurthub.

util.go:303] cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, in flight requests: 18
cache_manager.go:185] could not get gvk for ListObject for req: cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, GroupVersionResource is not recognized
local.go:224] object not found for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0
local.go:88] could not proxy local for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, customresourcedefinitions.apiextensions.k8s.io "" not found

I read the related source code briefly in pkg/yurthub/kubernetes/meta/restmapper.go:58. The defaultGroupVersions doesn't seem to include apiextension.k8s.io/v1, I'm not sure if this is the reason causing it.

func NewDefaultRESTMapperFromScheme() *meta.DefaultRESTMapper {
    s := scheme.Scheme
    defaultGroupVersions := s.PrioritizedVersionsAllGroups()
    mapper := meta.NewDefaultRESTMapper(defaultGroupVersions)
    // enumerate all supported versions, get the kinds, and register with the mapper how to address
    // our resources.
    for _, gv := range defaultGroupVersions {
        for kind := range s.KnownTypes(gv) {
            // Only need to process non-list resources
            if !strings.HasSuffix(kind, "List") {
                // Since RESTMapper is only used for mapping GVR to GVK information,
                // the scope field is not involved in actual use,
                // so all scope are currently set to meta.RESTScopeNamespace
                scope := meta.RESTScopeNamespace
                mapper.Add(gv.WithKind(kind), scope)
            }
        }
    }
    return mapper
}

Thanks for your kind help!

rambohe-ch commented 1 month ago

@obitoquilt would you like to upload the details logs of yurthub component on this node? I think maybe we can find some clues in the logs.

Yes. Here are the error logs of Yurthub.

util.go:303] cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, in flight requests: 18
cache_manager.go:185] could not get gvk for ListObject for req: cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, GroupVersionResource is not recognized
local.go:224] object not found for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0
local.go:88] could not proxy local for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, customresourcedefinitions.apiextensions.k8s.io "" not found

I read the related source code briefly in pkg/yurthub/kubernetes/meta/restmapper.go:58. The defaultGroupVersions doesn't seem to include apiextension.k8s.io/v1, I'm not sure if this is the reason causing it.

func NewDefaultRESTMapperFromScheme() *meta.DefaultRESTMapper {
  s := scheme.Scheme
  defaultGroupVersions := s.PrioritizedVersionsAllGroups()
  mapper := meta.NewDefaultRESTMapper(defaultGroupVersions)
  // enumerate all supported versions, get the kinds, and register with the mapper how to address
  // our resources.
  for _, gv := range defaultGroupVersions {
      for kind := range s.KnownTypes(gv) {
          // Only need to process non-list resources
          if !strings.HasSuffix(kind, "List") {
              // Since RESTMapper is only used for mapping GVR to GVK information,
              // the scope field is not involved in actual use,
              // so all scope are currently set to meta.RESTScopeNamespace
              scope := meta.RESTScopeNamespace
              mapper.Add(gv.WithKind(kind), scope)
          }
      }
  }
  return mapper
}

Thanks for your kind help!

@obitoquilt When yurthub cache response for cilium-agent list/get crds requests, yurthub will update unrecognized GVK info as following:

https://github.com/openyurtio/openyurt/blob/dadab08c243639c5c0316bd913aae1fcdd46a906/pkg/yurthub/cachemanager/cache_manager.go#L475

The gvk info is stored on the local disk and the file path is /etc/kubernetes/cache/_internal/restmapper/cache-crd-restmapper.conf.

so when cilium-agent list/get crds when cloud-edge network offline, yurthub will recognize the requests, then retrieve the cache data and return back to the cilium-agent.

would you upload the contents of configmap kube-system/yurt-hub-cfg? so i can check that the cache configuration is correct or not.

obitoquilt commented 1 month ago

@rambohe-ch Here are the configmap kube-system/yurt-hub-cfg:

data:
  cache_agents: '*'
  discardcloudservice: ""
  masterservice: ""
  servicetopology: ""
obitoquilt commented 1 month ago

@obitoquilt would you like to upload the details logs of yurthub component on this node? I think maybe we can find some clues in the logs.

Yes. Here are the error logs of Yurthub.

util.go:303] cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, in flight requests: 18
cache_manager.go:185] could not get gvk for ListObject for req: cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, GroupVersionResource is not recognized
local.go:224] object not found for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0
local.go:88] could not proxy local for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, customresourcedefinitions.apiextensions.k8s.io "" not found

I read the related source code briefly in pkg/yurthub/kubernetes/meta/restmapper.go:58. The defaultGroupVersions doesn't seem to include apiextension.k8s.io/v1, I'm not sure if this is the reason causing it.

func NewDefaultRESTMapperFromScheme() *meta.DefaultRESTMapper {
    s := scheme.Scheme
    defaultGroupVersions := s.PrioritizedVersionsAllGroups()
    mapper := meta.NewDefaultRESTMapper(defaultGroupVersions)
    // enumerate all supported versions, get the kinds, and register with the mapper how to address
    // our resources.
    for _, gv := range defaultGroupVersions {
        for kind := range s.KnownTypes(gv) {
            // Only need to process non-list resources
            if !strings.HasSuffix(kind, "List") {
                // Since RESTMapper is only used for mapping GVR to GVK information,
                // the scope field is not involved in actual use,
                // so all scope are currently set to meta.RESTScopeNamespace
                scope := meta.RESTScopeNamespace
                mapper.Add(gv.WithKind(kind), scope)
            }
        }
    }
    return mapper
}

Thanks for your kind help!

@obitoquilt When yurthub cache response for cilium-agent list/get crds requests, yurthub will update unrecognized GVK info as following:

https://github.com/openyurtio/openyurt/blob/dadab08c243639c5c0316bd913aae1fcdd46a906/pkg/yurthub/cachemanager/cache_manager.go#L475

The gvk info is stored on the local disk and the file path is /etc/kubernetes/cache/_internal/restmapper/cache-crd-restmapper.conf.

so when cilium-agent list/get crds when cloud-edge network offline, yurthub will recognize the requests, then retrieve the cache data and return back to the cilium-agent.

would you upload the contents of configmap kube-system/yurt-hub-cfg? so i can check that the cache configuration is correct or not.

Thanks for your detailed reply. I check the file /etc/kubernetes/cache/_internal/restmapper/cache-crd-restmapper.conf and find out a mistake.

{"apiextensions.k8s.io/v1/partialobjectmetadata": "PartialObjectMetadata", "apiextensions.k8s.io/v1/partialobjectmetadatas": "PartialObjectMetadata"}

The cilium-agent requests crd /apis/apiextensions.k8s.io/v1/customresourcedefinitions with PartialObjectMedata content-type，causing cache crd error.

The normal content of cache-crd-restmapper.conf should look like this, right?

{
  "apiextensions.k8s.io/v1/customresourcedefinition": "CustomResourceDefinition",
  "apiextensions.k8s.io/v1/customresourcedefinitions": "CustomResourceDefinition"

How to solve this error where request with PartialObjectMedata content-type? Thanks again for your very very kind help!

rambohe-ch commented 1 month ago

@obitoquilt would you like to upload the details logs of yurthub component on this node? I think maybe we can find some clues in the logs.

Yes. Here are the error logs of Yurthub.

util.go:303] cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, in flight requests: 18
cache_manager.go:185] could not get gvk for ListObject for req: cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, GroupVersionResource is not recognized
local.go:224] object not found for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0
local.go:88] could not proxy local for cilium-agent list customresourcedefinitions: /api/apiextensions.k8s.io/v1/customresourcedefinitions?limit=500&resourceVersion=0, customresourcedefinitions.apiextensions.k8s.io "" not found

I read the related source code briefly in pkg/yurthub/kubernetes/meta/restmapper.go:58. The defaultGroupVersions doesn't seem to include apiextension.k8s.io/v1, I'm not sure if this is the reason causing it.

func NewDefaultRESTMapperFromScheme() *meta.DefaultRESTMapper {
  s := scheme.Scheme
  defaultGroupVersions := s.PrioritizedVersionsAllGroups()
  mapper := meta.NewDefaultRESTMapper(defaultGroupVersions)
  // enumerate all supported versions, get the kinds, and register with the mapper how to address
  // our resources.
  for _, gv := range defaultGroupVersions {
      for kind := range s.KnownTypes(gv) {
          // Only need to process non-list resources
          if !strings.HasSuffix(kind, "List") {
              // Since RESTMapper is only used for mapping GVR to GVK information,
              // the scope field is not involved in actual use,
              // so all scope are currently set to meta.RESTScopeNamespace
              scope := meta.RESTScopeNamespace
              mapper.Add(gv.WithKind(kind), scope)
          }
      }
  }
  return mapper
}

Thanks for your kind help!

@obitoquilt When yurthub cache response for cilium-agent list/get crds requests, yurthub will update unrecognized GVK info as following: https://github.com/openyurtio/openyurt/blob/dadab08c243639c5c0316bd913aae1fcdd46a906/pkg/yurthub/cachemanager/cache_manager.go#L475

The gvk info is stored on the local disk and the file path is /etc/kubernetes/cache/_internal/restmapper/cache-crd-restmapper.conf. so when cilium-agent list/get crds when cloud-edge network offline, yurthub will recognize the requests, then retrieve the cache data and return back to the cilium-agent. would you upload the contents of configmap kube-system/yurt-hub-cfg? so i can check that the cache configuration is correct or not.

Thanks for your detailed reply. I check the file /etc/kubernetes/cache/_internal/restmapper/cache-crd-restmapper.conf and find out a mistake.

{"apiextensions.k8s.io/v1/partialobjectmetadata": "PartialObjectMetadata", "apiextensions.k8s.io/v1/partialobjectmetadatas": "PartialObjectMetadata"}

The cilium-agent requests crd /apis/apiextensions.k8s.io/v1/customresourcedefinitions with PartialObjectMedata content-type，causing cache crd error.

The normal content of cache-crd-restmapper.conf should look like this, right?

{
  "apiextensions.k8s.io/v1/customresourcedefinition": "CustomResourceDefinition",
  "apiextensions.k8s.io/v1/customresourcedefinitions": "CustomResourceDefinition"

How to solve this error where request with PartialObjectMedata content-type? Thanks again for your very very kind help!

@obitoquilt I have checked metadata request in my environment, and Yurthub component can not deal with metadata request correctly. so this is a bug of Yurthub component.

In the following metadata request, Yurthub recognized this request as List CRDs request currently.

curl -H "Accept: application/json;as=PartialObjectMetadataList;g=meta.k8s.io;v=v1" "$API_SERVER/apis/apiextensions.k8s.io/v1/customresourcedefinitions"

but this request is actually a List PartialObjectMetadata request, and the response is as following:

{
  "kind": "PartialObjectMetadataList",
  "apiVersion": "meta.k8s.io/v1",
  "metadata": {
    "resourceVersion": "xxxx"
  },
  "items": [
    {
      "kind": "PartialObjectMetadata"
      "apiVersion": "meta.k8s.io/v1",
      "metadata": {
        "name": "example-crd.mygroup.k8s.io",
        "namespace": "", // CRDs are cluster-wide, so no namespace
        "uid": "some-uid",
        "resourceVersion": "some-version",
        "generation": 1,
        "creationTimestamp": "2020-01-01T00:00:00Z",
        "labels": {
          "key": "value"
        },
        "annotations": {
          "key": "value"
        }
      }
    }
    // Other CRDs would follow in similar objects
  ]
}

I will make a pull request to handle this bug, please stay tuned.

rambohe-ch commented 1 month ago

@obitoquilt I want to describe some points about the solution of this bug, any comments are welcome.

  1. A new http.Handler will be added to convert list crds request to list partialObjectMetadata request for Yurthub cache.(request proxy will not be effected)
  2. response will be cached on the path: /etc/kubernetes/cache/{component}/partialobjectmetadata.v1.meta.k8s.io/{original-resource}/{ns}/{name}
    • original-resource: customresourcedefinitions
  3. local cache will be retrieved by the clients when network offline.
obitoquilt commented 1 month ago

@obitoquilt I want to describe some points about the solution of this bug, any comments are welcome.

  1. A new http.Handler will be added to convert list crds request to list partialObjectMetadata request for Yurthub cache.(request proxy will not be effected)

  2. response will be cached on the path: /etc/kubernetes/cache/{component}/partialobjectmetadata.v1.meta.k8s.io/{original-resource}/{ns}/{name}

    • original-resource: customresourcedefinitions
  3. local cache will be retrieved by the clients when network offline.

Looks good to me!

rambohe-ch commented 1 month ago

/assign @rambohe-ch