Closed imliuda closed 3 weeks ago
This issue is currently awaiting triage.
If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted
label and provide further guidance.
The triage/accepted
label can be added by org members by writing /triage accepted
in a comment.
/sig api-machinery
IMO, client side should not be aware of cache because the cache should always make sure the same semantics with etcd.
For resourceVersion=0
and limit != 0
case, the behavior is different indeed, but this is the lesser of two evils. Bypass cache will leads lots of promblems, you can find more disscuss here.
As for the long-term plan, WatchList
has achieved good results now. In the design, client side fetch objects by using watch
instead of list
, and all objects are returned from watchcache. It's a Alpha feature, you can enable it by feature gate for apiserver and env variable for client-go
. you can find more detail here: KEP-3157.
Actually, there are two problems:
TweakListOptionsFunc
, set limit=500 and resourceVersion to "" if it is "0" (to avoid problem 1), it may cause ListAndWatch's abnormal behavior.We are facing a problem that there are lots of jobs running in the cluster, but the controller can not list all of them within 60s.
I see, you tweaked the underlay ListWacher
inside of Reflector
. When Reflector decides to list full contents, your TweakListOptionsFunc
unconditional set a limit to the request, and make Reflector abnormal.
Well, I have to say what you have done has some kind of tricky. IMO, some list options, eg the limit
and resourceVersion
, are internal implementation detail of apiserver and Reflector and others should not be aware or change it. additional, client-side should not be aware of if using cache. so something like rv=""
will go through into etcd is not guaranteed, as pervious mentioned. The option will only guarantee the semantics, not implementation detail.
To solve your timeout promblem, maybe you can try to increase the timeout when do list
, or just try watchlist
feature.
@xuzhenglun Thanks, I will try to solve my problem in some other ways.
I'm closing this issue now, feel free to reopen it or open a new issue if you have something other to discuss.
/closing
/close
@xuzhenglun: Closing this issue.
What happened?
There are some situations user wants to always list resources from etcd. eg, with no limit option, the server may return full list from apiserver cache, if there are lots of resources, there may be a 60s timeout of apiserver, and user can provide a limit=500 option to force request go through etcd with pagination.
But the following codes have a problem, when an expired error occurs, it will try to do a full list, but user code update the limit options in
TweakListOptionsFunc
, so this will get only 500 items. In another word, currently, client-go do not support user provided limit option.What did you expect to happen?
Support user supplied limit option, so there a way user can force request go through etcd, to avoid apiserver 60s timeout problem, this is very import for long running controllers, there controllers usually do very small amount list operations. to some extent, this will also help to reduce apiserver memory usage.
How can we reproduce it (as minimally and precisely as possible)?
TweakListOptionsFunc
and set option.Limit to 500, and set option.ResourceVersion to "" if it is "0" (because apiserver dosn't honor limit when resourceVersion is "0")etcdctl compaction $(etcdctl endpoint status --write-out="json" | jq '.[0].Status.header.revision' | tr -d '\n')
Anything else we need to know?
No response
Kubernetes version
I think it is for all versions
Cloud provider
I think all clouds
OS version
Install tools
Container runtime (CRI) and version (if applicable)
Related plugins (CNI, CSI, ...) and versions (if applicable)