envoyproxy / envoy

Cloud-native high-performance edge/middle/service proxy
https://www.envoyproxy.io
Apache License 2.0
24.67k stars 4.75k forks source link

How to balance the number of quic connections on each thread #35396

Open hyasysu opened 1 month ago

hyasysu commented 1 month ago

Title: How to balance the number of quic connections on each thread

Description: I have a QUIC Listener as follows:

  - name: listener_5  # listen tcp connection from high_domian get response
    address:
      socket_address:
        protocol: UDP
        address: ip
        port_value: port
    udp_listener_config:
      quic_options:
        quic_protocol_options:
          initial_stream_window_size: 65536
          max_concurrent_streams: 8192
      downstream_socket_config:
        prefer_gro: true  # GRO(Generic Receive Offload)
      udp_packet_packet_writer_config:
        name: envoy.udp_packet_writer.udp_gso_batch_writer
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.udp_packet_writer.v3.UdpGsoBatchWriterFactory
    per_connection_buffer_limit_bytes: 10000000

I found that the number of quic connections per thread was not evenly distributed. Is there a mechanism to distribute it as evenly as possible on each worker thread?

yanavlasov commented 1 month ago

@danzh2010 @RyanTheOptimist for any recommendations

RyanTheOptimist commented 1 month ago

Are you using BPF as recommended here: https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/http/http3#bpf-usage

hyasysu commented 1 month ago

Yes. I've already used the BPF.

RyanTheOptimist commented 1 month ago

@danzh2010 Do you have any thoughts here?

danzh2010 commented 1 month ago

BPF should have distributed connections evenly. Did you use a customized connection id generator? The default one distribute the connections based on the first 4 bytes of their connection ids: https://www.envoyproxy.io/docs/envoy/latest/api-v3/extensions/quic/connection_id_generator/v3/envoy_deterministic_connection_id_generator.proto#extensions-quic-connection-id-generator-v3-deterministicconnectionidgeneratorconfig

github-actions[bot] commented 3 days ago

This issue has been automatically marked as stale because it has not had activity in the last 30 days. It will be closed in the next 7 days unless it is tagged "help wanted" or "no stalebot" or other activity occurs. Thank you for your contributions.