Open iljalukin opened 4 months ago
I can confirm that I am seeing the same issue with the RPM repository when installing on AlmaLinux 8.10.
Can confirm the same on Ubuntu 20.04.6 LTS (Focal Fossa):
Jul 3 10:52:36 loki promtail[377]: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)
A rollback/downgrade to promtail 3.0 is fixing it temporarily.
Please roll back any rpm+apt repositories distributing this binary. How can something like this just be distributed to a wide audience? Is there no testing for this stuff? This just breaks log aggregation for whole companies.
I have the same problem
Same with Rocky Linux 8.8
the same issue on AlmaLinux release 8.9
Same issue on Amazon Linux 2
same issue on Rocky8
# promtail --version
promtail: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by promtail)
promtail: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by promtail)
almalinux 8.10 also :(
@JStickler any updates when this will be fixed?
Same issue here on Ubuntu 20
Edit: Working on Ubuntu 22 because GLIBC is version 2.35, but on Ubuntu 20, GLIBC is version 2.31
Good evening,
Issue is present with promtail-3.1.1-1.x86_64, so i went for an older version this package for the moment. In my case, for AlmaLinux release 8.10 (Cerulean Leopard) i have installed an older version: dnf install promtail-2.9.9-1.x86_64
CentOS 8 stream problem version 3.1.1 - /lib64/libc.so.6: version `GLIBC_2.32' not found
3.0.1 is also affected
This affects Amazon Linux 2 as well (CentOS 7 derivative + updated kernel etc.)
Amazon Linux 2023 works ok.
EDIT: Downgrading to 2.9.9 works on older OSes.
Tried downgrading to 3.0.1 and 2.9.10 and finally got it working with 2.9.9 on Ubuntu 20
$ grep -i pretty /etc/os-release
PRETTY_NAME="Ubuntu 20.04.6 LTS"
$ dpkg -s promtail | grep Version
Version: 2.9.9
I assume the fix would be to use a build container/os of a lower glibc version!
Or build trully static go binary.
Same here, install from binaries, downgrade from 2.9.10 to 2.9.9 as a workaround (Debian 11, x86_64).
Same Issue occurs to me.
My OS is Debian 11.7
My Kernel version is 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux
I check my GLIBC Version as belows.
ldd --version
My GLIBC Version is ldd (Debian GLIBC 2.31-13+deb11u6) 2.31
If you use a version of Promtail that is compatible with the GLIBC version of your OS, this may fix the problem.
@iljalukin
In my case, Promtail v2.9.9
, It works well.
@iljalukin In my case,
Promtail v2.9.9
, It works well.
This does not solve my problem, as I want to install the latest available version via the APT repository as described above.
My workaround is to install version 3.0.0 with apt install promtail=3.0.0
and pin it with apt-mark hold promtail
.
Debian 11 is the current LTS version, which will be maintained until 2026 and it would be nice if promtail could be installed there in the current version.
I hope that the developers will pay attention to this issue and fix it.
It's honestly a little concerning that this has not been fixed after two months. This breaks log aggregation on most enterprise Linux distributions used in production. I have decided to disabled the Grafana RPM repository completely and moved working versions of the RPMs that I need into a private repository to avoid these things affecting production. But if this is not fixed in a reasonable time, then Grafana Loki is not the right solution.
Indeed concerning, hopefully due to holiday, meanwhile running old version. If not fix in long term, will need to look for alternative
I think there is no different way except updating OS which support target GLIBC library.
In production, Because It is hard to update OS version, It is better to make binary file less dependent on libraries like GLIBC closely related to the operating system.
https://github.com/grafana/loki/issues/13393#issuecomment-2295819830
https://github.com/grafana/loki/issues/13393#issuecomment-2302676145
I am not a GitHub workflow expert, so I am not able to submit the relevant PR to downgrade the "build environment", but I believe that the workflow is building on the latest Ubuntu (based on a rudimentary search inside the .github/ directory) which would have a glibc >= 2.32
My use case is to support Amazon Linux 2, which uses glibc 2.26
$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)
$ ldd --version
ldd (GNU libc) 2.26
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.
$ rpm -qa | grep glibc-2
glibc-2.26-64.amzn2.0.2.x86_64
@cyriltovena could you help with this?
It would be helpful to have a web page that shows the available Promtail Binary versions for each OS version.
Same with Rocky 8.9. Surprised this has not been dealt with given its severity.
I could be completely wrong, but could changing the "build OS" here, help?
Just found this issue, I have been meeting the same issue with Ubuntu 20.04.
root@tomoyafujita:~# systemctl status promtail
● promtail.service - Promtail service
Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Thu 2024-09-05 15:14:11 PDT; 174ms ago
Process: 225268 ExecStart=/usr/bin/promtail -config.file /etc/promtail/config.yml (code=exited, status=1/FAILURE)
Main PID: 225268 (code=exited, status=1/FAILURE)
root@tomoyafujita:~# /usr/bin/promtail -config.file /etc/promtail/config.yml
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/promtail)
I agree with @adamw-linadm above; why not do a static go build that includes all dependencies?
We run distroless, which requires us to do multi-stage builds and take the binaries out of many public images. We don't encounter problems like this when projects do a static go build.
any updates when this will be fixed?
This looks to me like a pretty significant issue, with a relatively easy fix. This issue is preventing us from updating above 2.9.9.
This looks to me like a pretty significant issue, with a relatively easy fix. This issue is preventing us from updating above 2.9.9.
Same for us.
same issue persists with the latest 3.2.0 release as well.
Very concerning, 3.2.0 rollout and still not working on linux.
So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application
So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application
I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.
So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application
I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.
Tested this am, alloy. Compatible with existing promtail conf.
Alloy run --storage.path=/....(data folder) --config.format=/alloypath/conf/promtail_config.yaml
Feeding nicely loki, clearly the way to go, running in uat in one server, need to check further. Thx for your input.
You can also convert promtail conf but for now easier to use the option to run promtail format easier for migration.
Debian 11 end of life as August 31st, 2026, but it has glibc 2.31. Why I have to use only 2.9.9 and cannot latest version promtail on the Debian 11?
I could be completely wrong, but could changing the "build OS" here, help?
loki/.github/workflows/release.yml
Line 21 in 4e17ad8 runs-on: "ubuntu-latest"
The runner image doesn't matter here, they build the binaries in the loki-build-image. I think this was introduced with 71a8f2c. Here, they're pulling in libsystemd
from bookworm
which is build against the newer version of glibc
in Debian:
https://github.com/grafana/loki/blob/ae955ed30d841675dbb9e30327b84728050e724a/loki-build-image/Dockerfile#L85
This image is then used to build the binaries in .github/workflows/minor-release-pr.yml respectively patch-release-pr.yml. They both upload their artifacts to a BUILD_ARTIFACTS_BUCKET. release.yml is just downloading it from there to publish on GitHub.
I'd suggest to revert the build-image to FROM golang:${GO_VERSION}-bullseye
for now. The bullseye
tag is supported until the GA of trixie
. See also https://github.com/docker-library/golang/issues/466.
Same here on CentOS Linux 7 (Core)
using promtail-3.2.0.x86_64.rpm
Another workaround is to build the binary on the OS you want to run it on (lowest GLIBC versioned OS).
Needs an installed version of golang
and gcc
git clone from main
or the tag v3.2.0
# make promtail
# ./clients/cmd/promtail/promtail --version
# promtail, version 3.2.0 (branch: HEAD, revision: 659f5421d)
build user: user@machine
build date: 2024-09-27T11:31:41Z
go version: go1.23.1
platform: linux/arm64
tags: unknown
Hello, same bug here, So i just build promtail from source.
In WSL, just install goland
sudo apt install golang gcc
Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0
cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail
redhat 8.10 also -_-
git clone from
main
or the tagv3.2.0
# make promtail # ./clients/cmd/promtail/promtail --version
Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0
cd /tmp git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki cd loki CGO_ENABLED=0 go build ./clients/cmd/promtail ls -al promtail
You need to set PROMTAIL_JOURNAL_ENABLED="true"
or build with -tags promtail_journal_enabled
otherwise promtail
is not linked to libsystemd
/ glibc
at all.
Hi all, we are extremely sorry this issue has gone under the radar for so long. We should have identified this issue faster. The Loki team is now aware of the issue and is working on a fix within the build process.
Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.
Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.
Promtail is in essence deprecated in the sense that all of our long term efforts are behind Grafana Alloy. For Promtail we're continuing to accept bug fixes, security patches, and minor updates/fixes such as this one. However, Promtail will not be getting new features.
Describe the bug When attempting to run Promtail, the following error occurs: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail) /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)
To Reproduce Install and run promtail 3.1.0 from apt.grafana.com using instructions from https://apt.grafana.com
Expected behavior Promtail should run without GLIBC version errors.
Environment:
Screenshots, Promtail config, or terminal output /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail) /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)