Upon an attempted upgrade for the nri-bundle chart from 5.0.67 to 5.0.82, we've started seeing one pod in the newrelic-logging deamonset running fine but others throwing cannot execute insert file ... and other db related errors.
For a bit of context, running on k8s 1.29 in EKS with :
On further inspection, it was noted that there was only one db file being created on the volume, named '$(NODE_NAME)-fb.db', which is the obvious reason why one pod would work fine, as other's wouldn't be able to get the lock on the same file.
We believe the issue was introduced with newrelic-logging 1.21.0 , where this change was made.
The reason why this is not working any more is because NODE_NAME is now defined after FB_DB. Here's the snippet from k8s docs that explains why this is an issue:
Note that order matters in the env list. An environment variable is not considered "defined" if it is specified further down the list.
At the moment, this is stopping us from applying updates Newrelic has notified us about resolving some vulnerabilities.
workato-integration[bot]
commented
1 week ago
Upon an attempted upgrade for the
nri-bundlechart from5.0.67to5.0.82, we've started seeing one pod in the newrelic-logging deamonset running fine but others throwingcannot execute insert file ...and other db related errors.For a bit of context, running on k8s 1.29 in EKS with :
On further inspection, it was noted that there was only one db file being created on the volume, named
'$(NODE_NAME)-fb.db', which is the obvious reason why one pod would work fine, as other's wouldn't be able to get the lock on the same file.We believe the issue was introduced with
newrelic-logging 1.21.0, where this change was made. The reason why this is not working any more is becauseNODE_NAMEis now defined afterFB_DB. Here's the snippet from k8s docs that explains why this is an issue:At the moment, this is stopping us from applying updates Newrelic has notified us about resolving some vulnerabilities.