Open zbialik opened 1 week ago
Update: i've tried using 3.0.0
and 3.1.1
and both have produced similar results.
I'm forced to use a downgraded loki version (2.9) for the loki-ruler
until this is resolved because we require all our alert rules to be defined in-core as ConfigMaps
(GitOps), rather than enabling definitions through Grafana UI.
Describe the bug
I am trying to upgrade Loki from
v2.9
tov3.2
and have already done this for majority of my k8s clusters. I have migrated from theloki-distributed
helm chart to use the latestloki
helm chart, which now supports the microservice deployment mode.Migrating to the new helm chart + upgrading to
v3.2
caused loki-ruler to change to aStatefulSet
manifest that's got pretty much identical specs to theDeployment
from before (v2.9
).Upon deployment, the upgraded
loki-ruler
, the Pod immediately Errors/CrashLoops with below log:I want to emphasize that all other loki components were a breeze to upgrade. It's just the
loki-ruler
that I am observing issues with.Previously (
v2.9
),loki-ruler
ran as aDeployment
with the below spec:The loki
config.yaml
I am using has the following in theruler:
sectionI'm not exactly sure what has changed since Loki
v2.9
, but I've not been able to upgradeloki-ruler
tov3.2
due to this issue.Important Notes
ConfigMap
volume/volumeMount, things will work fine, but that obviously doesn't meet our needs for maintaining our logging alert rules asConfigMap
objects.loki
helm chart withdeploymentMode=Distributed
and definingruler.directories
for alerts, the deployment will silently fail because the lokiconfig.yaml
will default to use S3 storage backend when at that point, it should configure the ruler to use the local storage.Expected behavior
Loki Ruler should run fine. It feel like the helm chart either needs some
initContainer
to copy the alert files to where ruler wants to see them or better handle the synchronization. I see a few issues that are still open and relevant to this today:Environment:
v1.30.4
Screenshots, Promtail config, or terminal output N/A