search

grafana / loki

Like Prometheus, but for logs.
https://grafana.com/loki
GNU Affero General Public License v3.0
24.02k stars 3.46k forks source link

GLIBC version mismatch in Promtail binary #13393

Open iljalukin opened 4 months ago

iljalukin commented 4 months ago

Describe the bug When attempting to run Promtail, the following error occurs: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail) /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

To Reproduce Install and run promtail 3.1.0 from apt.grafana.com using instructions from https://apt.grafana.com

Expected behavior Promtail should run without GLIBC version errors.

Environment:

Screenshots, Promtail config, or terminal output /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.32' not found (required by /usr/bin/promtail) /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version GLIBC_2.34' not found (required by /usr/bin/promtail)

mrlhansen commented 4 months ago

I can confirm that I am seeing the same issue with the RPM repository when installing on AlmaLinux 8.10.

jostrasser commented 4 months ago

Can confirm the same on Ubuntu 20.04.6 LTS (Focal Fossa):

Jul 3 10:52:36 loki promtail[377]: /usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)

A rollback/downgrade to promtail 3.0 is fixing it temporarily.

Smith4545 commented 4 months ago

Please roll back any rpm+apt repositories distributing this binary. How can something like this just be distributed to a wide audience? Is there no testing for this stuff? This just breaks log aggregation for whole companies.

jytx commented 4 months ago

I have the same problem

anhdowastaken commented 4 months ago

Same with Rocky Linux 8.8

Vitaliy155 commented 4 months ago

the same issue on AlmaLinux release 8.9

jacobamey commented 4 months ago

Same issue on Amazon Linux 2

Aethylred commented 4 months ago

same issue on Rocky8

# promtail --version
promtail: /lib64/libc.so.6: version `GLIBC_2.32' not found (required by promtail)
promtail: /lib64/libc.so.6: version `GLIBC_2.34' not found (required by promtail)
adamw-linadm commented 4 months ago

almalinux 8.10 also :(

jostrasser commented 3 months ago

@JStickler any updates when this will be fixed?

mmagdy88 commented 3 months ago

Same issue here on Ubuntu 20

Edit: Working on Ubuntu 22 because GLIBC is version 2.35, but on Ubuntu 20, GLIBC is version 2.31

bursan85 commented 3 months ago

Good evening,

Issue is present with promtail-3.1.1-1.x86_64, so i went for an older version this package for the moment. In my case, for AlmaLinux release 8.10 (Cerulean Leopard) i have installed an older version: dnf install promtail-2.9.9-1.x86_64

Alex4Codes commented 3 months ago

CentOS 8 stream problem version 3.1.1 - /lib64/libc.so.6: version `GLIBC_2.32' not found

bpfoster commented 3 months ago

3.0.1 is also affected

shantanugadgil commented 3 months ago

This affects Amazon Linux 2 as well (CentOS 7 derivative + updated kernel etc.)

Amazon Linux 2023 works ok.

EDIT: Downgrading to 2.9.9 works on older OSes.

boite commented 3 months ago

Tried downgrading to 3.0.1 and 2.9.10 and finally got it working with 2.9.9 on Ubuntu 20

$ grep -i pretty /etc/os-release
PRETTY_NAME="Ubuntu 20.04.6 LTS"
$ dpkg -s promtail | grep Version
Version: 2.9.9
shantanugadgil commented 3 months ago

I assume the fix would be to use a build container/os of a lower glibc version!

adamw-linadm commented 3 months ago

Or build trully static go binary.

fauust commented 3 months ago

Same here, install from binaries, downgrade from 2.9.10 to 2.9.9 as a workaround (Debian 11, x86_64).

bkkoli commented 3 months ago

Same Issue occurs to me.

My OS is Debian 11.7 My Kernel version is 5.10.0-23-amd64 #1 SMP Debian 5.10.179-1 (2023-05-12) x86_64 GNU/Linux

bkkoli commented 3 months ago

I check my GLIBC Version as belows.

ldd --version

My GLIBC Version is ldd (Debian GLIBC 2.31-13+deb11u6) 2.31

If you use a version of Promtail that is compatible with the GLIBC version of your OS, this may fix the problem.

bkkoli commented 3 months ago

@iljalukin In my case, Promtail v2.9.9, It works well.

iljalukin commented 3 months ago

@iljalukin In my case, Promtail v2.9.9, It works well.

This does not solve my problem, as I want to install the latest available version via the APT repository as described above. My workaround is to install version 3.0.0 with apt install promtail=3.0.0 and pin it with apt-mark hold promtail. Debian 11 is the current LTS version, which will be maintained until 2026 and it would be nice if promtail could be installed there in the current version.

I hope that the developers will pay attention to this issue and fix it.

mrlhansen commented 3 months ago

It's honestly a little concerning that this has not been fixed after two months. This breaks log aggregation on most enterprise Linux distributions used in production. I have decided to disabled the Grafana RPM repository completely and moved working versions of the RPMs that I need into a private repository to avoid these things affecting production. But if this is not fixed in a reasonable time, then Grafana Loki is not the right solution.

BlackMoon2020 commented 3 months ago

Indeed concerning, hopefully due to holiday, meanwhile running old version. If not fix in long term, will need to look for alternative

bkkoli commented 3 months ago

I think there is no different way except updating OS which support target GLIBC library.

In production, Because It is hard to update OS version, It is better to make binary file less dependent on libraries like GLIBC closely related to the operating system.

shantanugadgil commented 3 months ago

https://github.com/grafana/loki/issues/13393#issuecomment-2295819830

https://github.com/grafana/loki/issues/13393#issuecomment-2302676145

I am not a GitHub workflow expert, so I am not able to submit the relevant PR to downgrade the "build environment", but I believe that the workflow is building on the latest Ubuntu (based on a rudimentary search inside the .github/ directory) which would have a glibc >= 2.32

My use case is to support Amazon Linux 2, which uses glibc 2.26

$ cat /etc/*release
NAME="Amazon Linux"
VERSION="2"
ID="amzn"
ID_LIKE="centos rhel fedora"
VERSION_ID="2"
PRETTY_NAME="Amazon Linux 2"
ANSI_COLOR="0;33"
CPE_NAME="cpe:2.3:o:amazon:amazon_linux:2"
HOME_URL="https://amazonlinux.com/"
SUPPORT_END="2025-06-30"
Amazon Linux release 2 (Karoo)

$ ldd --version
ldd (GNU libc) 2.26
Copyright (C) 2017 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
Written by Roland McGrath and Ulrich Drepper.

$ rpm -qa | grep glibc-2
glibc-2.26-64.amzn2.0.2.x86_64
shantanugadgil commented 3 months ago

@cyriltovena could you help with this?

bkkoli commented 2 months ago

It would be helpful to have a web page that shows the available Promtail Binary versions for each OS version.

digdilem commented 2 months ago

Same with Rocky 8.9. Surprised this has not been dealt with given its severity.

shantanugadgil commented 2 months ago

I could be completely wrong, but could changing the "build OS" here, help?

https://github.com/grafana/loki/blob/4e17ad87563b37a7b514ff4874610e0750c60084/.github/workflows/release.yml#L21

fujitatomoya commented 2 months ago

Just found this issue, I have been meeting the same issue with Ubuntu 20.04.

root@tomoyafujita:~# systemctl status promtail
● promtail.service - Promtail service
     Loaded: loaded (/etc/systemd/system/promtail.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Thu 2024-09-05 15:14:11 PDT; 174ms ago
    Process: 225268 ExecStart=/usr/bin/promtail -config.file /etc/promtail/config.yml (code=exited, status=1/FAILURE)
   Main PID: 225268 (code=exited, status=1/FAILURE)
root@tomoyafujita:~# /usr/bin/promtail -config.file /etc/promtail/config.yml
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /usr/bin/promtail)
/usr/bin/promtail: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /usr/bin/promtail)
dhoward-medallia commented 2 months ago

I agree with @adamw-linadm above; why not do a static go build that includes all dependencies?

We run distroless, which requires us to do multi-stage builds and take the binaries out of many public images. We don't encounter problems like this when projects do a static go build.

Vitaliy155 commented 2 months ago

any updates when this will be fixed?

hafkensite commented 2 months ago

This looks to me like a pretty significant issue, with a relatively easy fix. This issue is preventing us from updating above 2.9.9.

cervenka-riege commented 2 months ago

This looks to me like a pretty significant issue, with a relatively easy fix. This issue is preventing us from updating above 2.9.9.

Same for us.

shantanugadgil commented 2 months ago

same issue persists with the latest 3.2.0 release as well.

BlackMoon2020 commented 2 months ago

Very concerning, 3.2.0 rollout and still not working on linux.

So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application

danpoltawski commented 2 months ago

So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application

I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.

BlackMoon2020 commented 2 months ago

So its now a fact loki/promtail will not support linux then it should be mention in main page. Will start looking for alternative, cannot stay in prod with unsupported application

I think the alternative that Grafanalabs want us to take is to move to Alloy - but it would be helpful if this was communicated explicitly rather than just leaving promtail to rot. 😢 it’s quite concerning lack of engagement given the impact, like you say.

Tested this am, alloy. Compatible with existing promtail conf.

Alloy run --storage.path=/....(data folder) --config.format=/alloypath/conf/promtail_config.yaml

Feeding nicely loki, clearly the way to go, running in uat in one server, need to check further. Thx for your input.

You can also convert promtail conf but for now easier to use the option to run promtail format easier for migration.

IgorOhrimenko commented 2 months ago

Debian 11 end of life as August 31st, 2026, but it has glibc 2.31. Why I have to use only 2.9.9 and cannot latest version promtail on the Debian 11?

J0WI commented 2 months ago

I could be completely wrong, but could changing the "build OS" here, help?

loki/.github/workflows/release.yml

Line 21 in 4e17ad8 runs-on: "ubuntu-latest"

The runner image doesn't matter here, they build the binaries in the loki-build-image. I think this was introduced with 71a8f2c. Here, they're pulling in libsystemd from bookworm which is build against the newer version of glibc in Debian: https://github.com/grafana/loki/blob/ae955ed30d841675dbb9e30327b84728050e724a/loki-build-image/Dockerfile#L85

This image is then used to build the binaries in .github/workflows/minor-release-pr.yml respectively patch-release-pr.yml. They both upload their artifacts to a BUILD_ARTIFACTS_BUCKET. release.yml is just downloading it from there to publish on GitHub.

I'd suggest to revert the build-image to FROM golang:${GO_VERSION}-bullseye for now. The bullseye tag is supported until the GA of trixie. See also https://github.com/docker-library/golang/issues/466.

nevesm commented 2 months ago

Same here on CentOS Linux 7 (Core) using promtail-3.2.0.x86_64.rpm

shantanugadgil commented 2 months ago

Another workaround is to build the binary on the OS you want to run it on (lowest GLIBC versioned OS).

Needs an installed version of golang and gcc

git clone from main or the tag v3.2.0


# make promtail
# ./clients/cmd/promtail/promtail --version
# promtail, version 3.2.0 (branch: HEAD, revision: 659f5421d)
  build user:       user@machine
  build date:       2024-09-27T11:31:41Z
  go version:       go1.23.1
  platform:         linux/arm64
  tags:             unknown
SckyzO commented 1 month ago

Hello, same bug here, So i just build promtail from source.

In WSL, just install goland

sudo apt install golang gcc

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail
TomiCheng commented 1 month ago

redhat 8.10 also -_-

x4e-jonas commented 1 month ago

git clone from main or the tag v3.2.0

# make promtail
# ./clients/cmd/promtail/promtail --version

Download Loki 3.2.0 from github, and build promtail with CGO_ENABLED=0

cd /tmp
git clone --depth 1 --branch v3.2.0 https://github.com/grafana/loki
cd loki
CGO_ENABLED=0 go build ./clients/cmd/promtail
ls -al promtail

You need to set PROMTAIL_JOURNAL_ENABLED="true" or build with -tags promtail_journal_enabled otherwise promtail is not linked to libsystemd/ glibc at all.

Jayclifford345 commented 1 month ago

Hi all, we are extremely sorry this issue has gone under the radar for so long. We should have identified this issue faster. The Loki team is now aware of the issue and is working on a fix within the build process.

digdilem commented 1 month ago

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

cstyan commented 1 month ago

Good news. I was beginning to think promtail was being deprecated in favour of alloy. Glad that's not the case.

Promtail is in essence deprecated in the sense that all of our long term efforts are behind Grafana Alloy. For Promtail we're continuing to accept bug fixes, security patches, and minor updates/fixes such as this one. However, Promtail will not be getting new features.