[Issues] RedWarden will get laggy when beacon increases

[XiaoliChan]

(Same as the title...) ENV:

• CS 4.3
• RedWarden latest commit

[mgeeky]

Hi!

Thanks for this issue report. Can you please let me know how many beacons are we talking here?

Regards, Mariusz.

[XiaoliChan]

About 52 beacons.

[mgeeky]

Can you please try tweaking your configuration to see if that helps?

Initial ideas could include:

1. Increase logging thresholds to slow down logging facility, for instance:

throttle_down_peer_logging:
  log_request_delay: 180
  requests_threshold: 1

2. Decrease threshold for considering a Beacon trusted:

add_peers_to_whitelist_if_they_sent_valid_requests:
  number_of_valid_http_get_requests: 5
  number_of_valid_http_post_requests: 1

3. To let Tornado faster terminate connections, I'd suggest considering terminating invalid request instead of serving them further by offering redirection pages. So, adjusting drop_action could help here:

drop_action: reset

4. Finally, if above doesn't make a difference - I'd try playing with policies to see which one introduces the most performance impact. One (or few) of them disabled could introduce significant request serving speed. Unfortunately to understand disabling which policy could help is an entirely try-and-error effort:

policy:
  # [IP: ALLOW, reason:0] Request conforms ProxyPass entry (url="..." host="..."). Passing request to specified host
  allow_proxy_pass: True
  # [IP: ALLOW, reason:2] Peer's IP was added dynamically to a whitelist based on a number of allowed requests
  allow_dynamic_peer_whitelisting: True
  # [IP: DROP, reason:1] inbound User-Agent differs from the one defined in C2 profile.
  drop_invalid_useragent: True
  # [IP: DROP, reason:2] HTTP header name contained banned word
  drop_http_banned_header_names: True
  # [IP: DROP, reason:3] HTTP header value contained banned word:
  drop_http_banned_header_value: True
  # [IP: DROP, reason:4b] peer's reverse-IP lookup contained banned word
  drop_dangerous_ip_reverse_lookup: True
  # [IP: DROP, reason:4e] Peer's IP geolocation metadata contained banned keyword! Peer banned in generic fashion.
  drop_ipgeo_metadata_containing_banned_keywords: True
  # [IP: DROP, reason:5] HTTP request did not contain expected header
  drop_malleable_without_expected_header: True
  # [IP: DROP, reason:6] HTTP request did not contain expected header value:
  drop_malleable_without_expected_header_value: True
  # [IP: DROP, reason:7] HTTP request did not contain expected (metadata|id|output) section header:
  drop_malleable_without_expected_request_section: True
  # [IP: DROP, reason:8] HTTP request was expected to contain (metadata|id|output) section with parameter in URI:
  drop_malleable_without_request_section_in_uri: True
  # [IP: DROP, reason:9] Did not found append pattern (this logic is known to cause troubles, use with caution):
  drop_malleable_without_prepend_pattern: False
  # [IP: DROP, reason:10] Did not found append pattern (this logic is known to cause troubles, use with caution):
  drop_malleable_without_apppend_pattern: False
  # [IP: DROP, reason:11] Requested URI does not aligns any of Malleable defined variants:
  drop_malleable_unknown_uris: True
  # [IP: DROP, reason:12] HTTP request was expected to contain <> section with URI-append containing prepend/append fragments
  drop_malleable_with_invalid_uri_append: True

Currently I don't have time available to conduct performance testing myself, but I would really appreciate if you could try to see if any of the above help at all. Your input would be really valuable to all RedWarden users!

Let me know if you conduct some tests and how they went.

Best regards, Mariusz.

[XiaoliChan]

Can you please try tweaking your configuration to see if that helps?

Sure, here is my configuration : ) xiaolichan's redwarden.yaml.txt

In my way, I use this IP blacklist instead of the original one. RedGuard's IP blacklist

[XiaoliChan]

Another issues is: how to configure RedWarden to make CS show the actual external IP address?

[XiaoliChan]

I think I explained the issues not clearly, I mean beacon's heartbeat will get laggy when the beacon increases, so I think probably something happening in RedWarden.

In this screenshot, you can see even though this beacon already receive "sleep 1" to decrease heartbeat, it still got a huge heartbeat packet.

[XiaoliChan]

1. Increase logging thresholds to slow down logging facility, for instance.
2. Decrease threshold for considering a Beacon trusted.
3. To let Tornado faster terminate connections, I'd suggest considering terminating invalid request instead of serving them further by offering redirection pages.

I got nothing changed by following these steps.

4. Finally, if above doesn't make a difference - I'd try playing with policies to see which one introduces the most performance impact. One (or few) of them disabled could introduce significant request serving speed. Unfortunately to understand disabling which policy could help is an entirely try-and-error effort.

I don't know which policy cause performance issues even changing it one by one.

[mgeeky]

Oh dammit, I'm clueless here. Indeed RedWarden gets choked by the number of Beacons you have there.

For the last resort I would try load balancing traffic into several RedWardens.

So you could use AWS/Azure/HAProxy or put an nginx in front that would balance requests to the round-robin selected RedWardens, thus spreading the load.

I know that's not perfect but I've got no better answer here, sorry. Since I've never needed to use that many Beacons I didnt favor perfomance while desiging the tool.

Regards, Mariusz.

[XiaoliChan]

@mgeeky I see, thank you for your answer. :)

RedWarden needs a performance improvement.