search

crowdsecurity / cs-haproxy-bouncer

Crowdsec bouncer for HAProxy
MIT License
18 stars 7 forks source link

Intermittend failures #28

Closed roedie closed 10 months ago

roedie commented 11 months ago

When running multiple HAProxy bouncers I see this happen randomly:

[ALERT]    (3040388) : Lua task: [state-id 0] runtime error: /usr/lib/crowdsec/lua/haproxy/json.lua:185: expected closing quote for string at line 1 col 1134323 from [C]: in global 'error', /usr/lib/crowdsec/lua/haproxy/json.lua:185: in upvalue 'decode_error', /usr/lib/crowdsec/lua/haproxy/json.lua:255: in function line 218 ..., /usr/lib/crowdsec/lua/haproxy/json.lua:330: in function line 307 ..., /usr/lib/crowdsec/lua/haproxy/json.lua:293: in function line 280 ..., /usr/lib/crowdsec/lua/haproxy/json.lua:330: in function line 307 ..., /usr/lib/crowdsec/lua/haproxy/json.lua:379: in field 'decode', /usr/lib/crowdsec/lua/haproxy/crowdsec.lua:219: in upvalue 'refresh_decisions', /usr/lib/crowdsec/lua/haproxy/crowdsec.lua:271: in function line 260.

Not sure if it's a LAPI issue when too much bouncers connect or if it is a bouncer issue. At least a retry would be nice instead of a hard failure.

LaurenceJJones commented 11 months ago

How many bouncers you got connected and they all using unique api keys?

roedie commented 11 months ago

I've got 28 bouncers connected with unique keys.

The problem mostly seems to occur when the HAProxy instances are reloaded at the same time. When I modify the cronjob to make sure they do not reload at the same time the problem almost goes away.

LaurenceJJones commented 11 months ago

The problem mostly seems to occur when the HAProxy instances are reloaded at the same time. When I modify the cronjob to make sure they do not reload at the same time the problem almost goes away.

Yeah that wouldn't surprise me as when they load they will ask LAPI for the full blocklist on startup. We have a feature flag on 1.5 that is called chunked_stream this may help. To enable the feature you can follow the instruction printed when running

cscli config feature-flags
roedie commented 10 months ago

@LaurenceJJones Sorry I didn't respond, I totally missed your reply. First testing seems to show that it helps indeed. Thanks!

LaurenceJJones commented 10 months ago

@LaurenceJJones Sorry I didn't respond, I totally missed your reply. First testing seems to show that it helps indeed. Thanks!

I'm glad to hear it! If we get enough positive feedback about it, it may become the default.

roedie commented 10 months ago

@LaurenceJJones I just wanted to let you know that my bouncers have been running fine for 2 weeks now with chunked_stream enabled. Thanks again.

LaurenceJJones commented 10 months ago

@LaurenceJJones I just wanted to let you know that my bouncers have been running fine for 2 weeks now with chunked_stream enabled. Thanks again.

Thank you for the update! This feedback is very helpful as it will push us to make it the default