Open dev-maniac opened 9 months ago
I encountered the exact same issue. I could narrow it down to the realpath cache of php. Because the token file is a symlink to a directory of the latest token and kubernetes is creating a new directory if the token get renewed. So PHP is referencing to the old event if the old token does not longer exists. This leads us to a kind of misleading error message.
So even sleeping for some seconds wouldn't help in this case.
luckily php has a function to clear the entire realpath cache or just a single entry. (clearstatcache(true)
or clearstatcache(true, '/path/to/symlink')
)
So what i am doing now is to clear this one path of the token symlink every time i am connecting to the cluster. And now it works without the annoying error:
clearstatcache(true, '/var/run/secrets/kubernetes.io/serviceaccount/token');
KubernetesCluster::inClusterConfiguration(config('k8s.cluster.apiUri'));
This is fixing the error, but it would be nice if this would be done in the library itself.
After a long term observation I still have encountered this issue a few times a week, even with this line of code:
clearstatcache(true, '/var/run/secrets/kubernetes.io/serviceaccount/token');
But after replacing the explicit cache clear call, with:
clearstatcache(true);
every time before we initialize the this library, it works since 3-4 weeks without a single Error. It is not the cleanest approach, but if this isn't called every request of your webserver, it could be a good workaround.
Since k8s 1.22 automounted service account tokens have a limited lifespan and will be updated by k8s when they expire.
It seems that there can be a short timespan where access token is not accessible. Got following exception at Dec 8, 2023 13:36 UTC:
file_get_contents(/var/run/secrets/kubernetes.io/serviceaccount/token): Failed to open stream: No such file or directory
Directory in pod (times also in UTC):
So it seems like it's really related...
I did not find anything related to this issue on the net, yet. Not entirely sure, if I should report this somewhere else.
Proposed, simple fix would be to try reading file two or three times with a short sleep inbetween.