Closed jzemla closed 2 years ago
So, I am openly calling myself out on knowing just enough to be dangerous and not having a full understanding of what I'm doing. Hello, world!
In short, this is resolved by changing:
"encoder": {
"format": "formatted",
"template": "{common_log} \"{request>headers>Referer>[0]}\" \"{request>headers>User-Agent>[0]}\""
},
to:
"encoder": {
"format": "json"
},
...and thus benefitting from caddy's structured log files which is the purpose of this bouncer to begin with.
@jzemla: great that you found out yourself!
I should probably update the example for the logs in config.json
. Back when I included it for the first time I had to output Caddy logs in the Apache format for CrowdSec to parse it. It seems support for the Caddy structured format was added to CrowdSec not too long ago, so it's nice that it now works out of the box 😄.
I've always considered the example for the logs as a kind of extra. It's not required to ingest the Caddy logs into CrowdSec to make the bouncer work, but it's a good thing to do, nonetheless.
Have opened #10 to track this. Your example will help me test this. Thanks!
Sorry to open an old issue. Could anyone get the caddy-logs parser to work in 2.6.2? I also tried using a grok debugger to find what changed, but couldn't get it to work with both console and json log formats. I was able to get it to work by downloading caddy with the transform plugin and outputting in the common_log format and use the apache2 collection.
log {
format transform "{common_log}"
}
Also, I had to change the apache2-logs.yaml file to look for the logs coming from caddy instead of apache (I use homeassistant, so needed to use the plugin name)
filter: "evt.Parsed.program startsWith 'addon_c80c7555_caddy-2'"
Direct caddy logs would be nicer, but this method works. The bouncer works fine!
So, what is the correct config file currently? And how can I test that? Edit: I also succeeded with transformation to apache2 like jdeath did. Would be ncie if native caddy format could be suported again . . .
Hello,
Environment:
I'm having trouble getting this to parse my caddy access.log. I am using the suggested config from the example, but crowdsec is unable to parse the file. I apologize in advance for being a github/devops newbie -- if there is something I missed or can provide more insight into, please let me know!
Caddy - config.json:
Failed grok parse via caddy-logs:
I found that I can force crowdsec to use the apache2-logs parser by modifying
/etc/crowdsec/parsers/s01-parse/apache2-logs.yaml
to:...which then gets me this...
Successful grok parse via apache2-logs:
Did I configure something incorrectly?