Alpine curl issues in 6.4.16 / 7.0.0 due to Alpine 3.20 upgrade

syphernl commented 2 weeks ago

SUMMARY

Many web-related things (webchecks, Javascript HTTP Client) are throwing this error Failed writing received data to disk/application after upgrading to Zabbix 6.4.16 and 7.0.0. Both have in common that they are using alpine:3.20 instead of alpine:3.19.

A workaround is to explicitly set the Accept-Encoding: identity header in the webchecks / Javascript HTTP client or to use the Ubuntu container as it is not affected by this issue.

It appears that shipped libcurl is causing issues with this.

Ubuntu: libcurl4t64:amd64 8.5.0-2ubuntu10.1
Alpine: libcurl-8.7.1-r0

There are quite a few commits between the 8.5.0 and 8.7.1 release, something must have changed that causes this issue.

OS / ENVIRONMENT / Used docker-compose files

zabbix/zabbix-server-pgsql:alpine-6.4.16

CONFIGURATION

STEPS TO REPRODUCE

Run server container zabbix/zabbix-server-pgsql:alpine-6.4.15 on Alpine
Create a webcheck
See it works
Upgrade to zabbix/zabbix-server-pgsql:alpine-6.4.16
See it is showing an error: Failed writing received data to disk/application
Swap to zabbix/zabbix-server-pgsql:ubuntu-6.4.16
Error is gone

EXPECTED RESULTS

See above

ACTUAL RESULTS

Errors given: `Failed writing received data to disk/application``

RELATED ISSUES

These issues are (potentially) related to this:

dotneft commented 2 weeks ago

Sorry, but what do you want from our side? We can not fix OS issues...

syphernl commented 2 weeks ago

I would recommend reporting this issue upstream and downgrading the base image to alpine:3.19 until this issue is resolved as this breaks webchecks, javascript client and notifications.

paskal commented 2 weeks ago

I vote for pinning the Alpine image before the issue is fixed because it's really annoying. Users would need to change many webchecks to mitigate it so that their monitoring works again, changing the base image is way easier mitigation.

dotneft commented 2 weeks ago

The problem must be fixed in Curl or in Zabbix, not guarantee that the affected version will not be available in other distributions or Alpine 3.19 as well.

paskal commented 1 week ago

The curl issue is linked, and they won't change the library, so Zabbix must work around it on their side. The issue won't be present in Alpine 3.19 as it has curl 8.5.0, while 3.20 got 8.7.1 and edge 8.8.0: minor version in Alpine releases will stay fixed except for edge, which could be updated at any time.

Sticking to Alpine 3.19 is the cheapest and most robust workaround until the Zabbix team fixes the issue.

dotneft commented 1 week ago

Unfortunately I can not reproduce the issue. May I ask somebody to share info how to reproduce it? I tried to use https://support.zabbix.com/browse/ZBX-23702, but without success.

dotneft commented 1 week ago

Reproduced the issue, on Alpine 3.19 as well:

# docker run -ti alpine:3.19 sh
# apk update && apk add curl
# curl -V
curl 8.5.0 (aarch64-alpine-linux-musl) libcurl/8.5.0 OpenSSL/3.1.5 zlib/1.3.1 brotli/1.1.0 c-ares/1.27.0 libidn2/2.3.4 nghttp2/1.58.0
Release-Date: 2023-12-06
Protocols: dict file ftp ftps gopher gophers http https imap imaps mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets
# curl https://www.fust.ch/ --compressed
...
curl: (23) Failed writing received data to disk/application

So the problem exists on 3.19 as well...

dotneft commented 4 days ago

If you try to use the following way, does it help: https://wiki.defect.ch/apps/zabbix/template/docker#failed_writing_received_data_to_diskapplication ?

paskal commented 2 days ago

I'm using zabbix/zabbix-server-mysql:latest, which is currently affected by the bug. However, adding RUN wget "https://dl-cdn.alpinelinux.org/alpine/edge/main/$(apk --print-arch)/libcurl-8.8.0-r0.apk" && apk add libcurl-8.8.0-r0.apk && rm libcurl-8.8.0-r0.apk to my local build fixed the issue.

Fix was introduced in https://github.com/curl/curl/commit/b30d694a027eb771c02a3db0dee0ca03ccab7377 which was released as 8.8.0 and was cherry-picked into Debian but not Alpine yet it seems.

dotneft commented 2 days ago

Hmm... I tested 8.8.0, but this one still failed:

curl https://www.fust.ch/ --compressed

no?

paskal commented 2 days ago

Please verify that you are using curl 8.8.0 as with the instruction by the link and your comment above you would end up with libcurl 8.8.0 but curl with version from the repo (8.5.0 or 8.7.0).

paskal commented 2 days ago

I can reproduce the issue on website you used for example but not in Zabbix on my website:

/var/lib/zabbix # curl --version
curl 8.8.0 (aarch64-alpine-linux-musl) libcurl/8.8.0 OpenSSL/3.3.1 zlib/1.3.1 brotli/1.1.0 zstd/1.5.6 c-ares/1.28.1 libidn2/2.3.7 libpsl/0.21.5 nghttp2/1.62.0
Release-Date: 2024-05-22
Protocols: dict file ftp ftps gopher gophers http https imap imaps ipfs ipns mqtt pop3 pop3s rtsp smb smbs smtp smtps telnet tftp ws wss
Features: alt-svc AsynchDNS brotli HSTS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM PSL SSL threadsafe TLS-SRP UnixSockets zstd
/var/lib/zabbix # curl https://www.fust.ch/ --compressed >/dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  112k    0  112k    0     0  1090k      0 --:--:-- --:--:-- --:--:-- 1098k
curl: (23) Failed writing received data to disk/application

Either curl behaves differently than libcurl, or only a portion of the problem is solved. In my case, the issue seems to be related to brotli compression

dotneft commented 2 days ago

So if you use web item / web monitoring for https://www.fust.ch/ everything is fine on Zabbix side?

zabbix / zabbix-docker