api to update stick table using a rest/grpc api

amitlimaye commented 2 years ago

Your Feature Request

The haproxy peers protocol allows exporting and updating stick tables in a cluster. I want to develop a REST/GRPC api for the same behavior. This will allow standard backend services which will consume this data to do it without having to implement another protocol.

What are you trying to do?

i am trying to develop a service which has a global view of all the stick tables configured in all my haproxy instances and take actions based on this centralized view which will involve updating some other stick tables to block/rate limit traffic. The rest of the services understand rest/grpc and wanted to avoid developing a peers protocol implementation if possible.

Output of `haproxy -vv`

haproxy -vv
HAProxy version 2.4.4-acb1d0b 2021/09/07 - https://haproxy.org/
Status: long-term supported branch - will stop receiving fixes around Q2 2026.
Known bugs: http://www.haproxy.org/bugs/bugs-2.4.4.html
Running on: Darwin 20.4.0 Darwin Kernel Version 20.4.0: Thu Apr 22 21:46:47 PDT 2021; root:xnu-7195.101.2~1/RELEASE_X86_64 x86_64
Build options :
  TARGET  = generic
  CPU     = generic
  CC      = cc
  CFLAGS  = -O2 -g -Wall -Wextra -Wdeclaration-after-statement -fwrapv -Wno-address-of-packed-member -Wno-unused-label -Wno-sign-compare -Wno-unused-parameter -Wno-clobbered -Wno-missing-field-initializers -Wno-cast-function-type -Wno-string-plus-int -Wtype-limits -Wshift-negative-value -Wshift-overflow=2 -Wduplicated-cond -Wnull-dereference
  OPTIONS = USE_KQUEUE=1 USE_PCRE=1 USE_POLL=1 USE_THREAD=1 USE_OPENSSL=1 USE_ZLIB=1
  DEBUG   = 

Feature list : -EPOLL +KQUEUE -NETFILTER +PCRE -PCRE_JIT -PCRE2 -PCRE2_JIT +POLL -PRIVATE_CACHE +THREAD -PTHREAD_PSHARED -BACKTRACE -STATIC_PCRE -STATIC_PCRE2 +TPROXY -LINUX_TPROXY -LINUX_SPLICE -LIBCRYPT -CRYPT_H -GETADDRINFO +OPENSSL -LUA -FUTEX -ACCEPT4 -CLOSEFROM +ZLIB -SLZ -CPU_AFFINITY -TFO -NS -DL -RT -DEVICEATLAS -51DEGREES -WURFL -SYSTEMD -OBSOLETE_LINKER -PRCTL -THREAD_DUMP -EVPORTS -OT -QUIC -PROMEX -MEMORY_PROFILING

Default settings :
  bufsize = 16384, maxrewrite = 1024, maxpollevents = 200

Built with multi-threading support (MAX_THREADS=64, default=1).
Built with OpenSSL version : OpenSSL 1.1.1l  24 Aug 2021
Running on OpenSSL version : OpenSSL 1.1.1l  24 Aug 2021
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports : TLSv1.0 TLSv1.1 TLSv1.2 TLSv1.3
Built with zlib version : 1.2.11
Running on zlib version : 1.2.11
Compression algorithms supported : identity("identity"), deflate("deflate"), raw-deflate("deflate"), gzip("gzip")
Built with transparent proxy support using:
Built with PCRE version : 8.45 2021-06-15
Running on PCRE version : 8.45 2021-06-15
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Encrypted password support via crypt(3): no
Built with clang compiler version 12.0.5 (clang-1205.0.22.9)

Available polling systems :
     kqueue : pref=300,  test result OK
       poll : pref=200,  test result OK
     select : pref=150,  test result OK
Total: 3 (3 usable), will use kqueue.

Available multiplexer protocols :
(protocols marked as <default> cannot be specified using 'proto' keyword)
              h2 : mode=HTTP       side=FE|BE     mux=H2       flags=HTX|CLEAN_ABRT|HOL_RISK|NO_UPG
            fcgi : mode=HTTP       side=BE        mux=FCGI     flags=HTX|HOL_RISK|NO_UPG
              h1 : mode=HTTP       side=FE|BE     mux=H1       flags=HTX|NO_UPG
       <default> : mode=HTTP       side=FE|BE     mux=H1       flags=HTX
            none : mode=TCP        side=FE|BE     mux=PASS     flags=NO_UPG
       <default> : mode=TCP        side=FE|BE     mux=PASS     flags=

Available services : none

Available filters :
        [SPOE] spoe
        [CACHE] cache
        [FCGI] fcgi-app
        [COMP] compression
        [TRACE] trace

wtarreau commented 2 years ago

There have been some thoughts about reworking the protocol to make it even more efficient network-wise and CPU-wise. However I strongly doubt that this will go in the direction of a very generic protocol like gRPC, because CPU and bandwidth currently are the main challenges, not the ease of implementation by external components, and any time you replace a tailor-made protocol by a generic one, your processing costs explode. That doesn't mean that it couldn't happen, it can be considered as part of the options, but I don't have big hopes to ever announce you a good news on this front, but rather to all those who want to always process more messages more efficiently.

amitlimaye commented 2 years ago

I look at the peer protocol and it looks like it is designed for a cluster of haproxy servers to communicate between themselves without the need for an external abritrator. The design i am thinking is of an aggregator which pulls periodically rather than through continuous updates. I was hoping to write a filter/service to pull this data and export it using grpc/rest ( something similar to prometheus exporter) and also provide a mechanism to update tables. The filter it looks like cannot be built outside the haproxy tree ( it cannot be a shared object that is loaded) which is why i was thinking about this feature. If i add this as a filter is there a chance of this filter making it to upstream.

I understand the efficiency concerns when there are continuous updates and cpu/networks concern become a very valid reason. i am not proposing that we modify the peer protocol but rather provide another mechanism to update/retrieve these tables for use cases which don't need these continuous updates

haproxy / haproxy