grobian / carbon-c-relay

Enhanced C implementation of Carbon relay, aggregator and rewriter
Apache License 2.0
379 stars 115 forks source link

Support lz4 and gzip #458

Open aklyuk opened 8 months ago

aklyuk commented 8 months ago

Hello. I build docker image from last version carbon-c-relay 3.7.4
I get error:

[2023-11-21 16:21:58] starting carbon-c-relay v3.7.4 (2022-02-13), pid=1
[2023-11-21 16:21:58] /etc/carbon-c-relay/carbon-c-relay.conf:2:26: feature gzip not compiled in
[2023-11-21 16:21:58]   type linemode transport gzip
[2023-11-21 16:21:58]                           ^^^^
[2023-11-21 16:21:58] failed to read configuration '/etc/carbon-c-relay/carbon-c-relay.conf'

On this line i add --with-gzip: ./configure --disable-maintainer-mode --with-gzip && make but get error

checking for gzopen in -lz_missing_header... no
configure: error: in `/opt/carbon-c-relay-build':
configure: error: --with-gzip was given, but test for gzip failed

How to compile gzip or lz4 support?

grobian commented 8 months ago

you need zlib-devel or something similar, such that the configure script can find it

aklyuk commented 8 months ago

Thank. Finally RUN incruction looks like:

RUN \
  apk add --no-cache build-base zlib-dev lz4-dev && \
  touch conffile.tab.c conffile.tab.h conffile.yy.c \
  configure.ac Makefile.am aclocal.m4 configure \
  Makefile.in config.h.in && \
  ./configure --disable-maintainer-mode --with-gzip --with-lz4 && make

But i get error: /etc/carbon-c-relay/carbon-c-relay.conf:13:0: cannot use UDP transport for compressed/encrypted stream

It is true, carbon-c-relay don't work with inbound compressed udp?

my config is:

listen
  type linemode transport gzip
    127.0.0.1:2003 proto udp
  ;
grobian commented 8 months ago

compression is done in streams, if you lose one packet, you'd not be able to do anything, same for encryption, hence the requirement for TCP

aklyuk commented 8 months ago

compressed tcp is working. But, i get this message: buffer overflow during read of gzip stream

carbon-c-relay is running in docker. I have increased sysctl's parameters - net.ipv4.tcp_rmem, net.core.rmem_max to 1Gb, but no changes. I have build docker-image with it ENTRYPOINT instruction - ENTRYPOINT ["/usr/bin/carbon-c-relay", "-f", "/etc/carbon-c-relay/carbon-c-relay.conf", "-U", "268435456"] - no changes.

What can i do?

grobian commented 8 months ago

hmmm, who is generating/sending the gzipped stream?

aklyuk commented 8 months ago

sending is carbon-c-relay v3.3 receiving is carbon-c-relay v.3.7.4