HTTP access log produces invalid json

[pmhausen]

I am moving my reverse proxy from a standalone installation to your plugin specifically to have CrowdSec parse the access logs and block malicious requests.

For some reason the log files contain invalid json. Each line starts like this:

<14>1 2024-02-16T11:36:26+01:00 opnsense.ettlingen.hausen.com caddy - - [meta sequenceId="353"] [...]

Any idea what is going wrong? On my old standalone system logs look like this:

{"level":"info","ts":1708079953.2351027,"logger":"http.log.access.log10" [...]

Kind regards, Patrick

[Monviech]

Yeah the problem is the syslog-ng integration, it requires a very specific format so that the default parser and log view in the opnsense can display it.

/usr/local/etc/syslog-ng.conf.d/caddy.conf

The caddy logs in standard json format are sent to the unix datagram socket /var/caddy/var/run/log and from there are caught by syslog-ng. In the above configuration are specific parsing rules that rewrite the logs. Then it's written into log files.

I have an older branch that doesn't have the syslog-ng integration yet, the Caddyfile template would have to be changed like this in the global options:

https://github.com/Monviech/os-caddy-plugin/blob/1.3.4/usr/plugins/devel/caddy/src/opnsense/service/templates/Pischem/Caddy/Caddyfile

log {
        output file /var/log/caddy/caddy.log {
            roll_size 10MiB
            roll_keep 10
            roll_keep_for 720h
        }
    }

Though doing that comes with the regression that the log view in the OPNsense wouldn't work properly anymore. You would need to use this link then /ui/diagnostics/log/caddy/caddy. You can see that the unformatted json is displayed in the GUI.

[pmhausen]

Ah ... understood. May I suggest a new feature? ;)

1. Keep the OPNsense specific logging for everything that is enabled by default and relates to the Caddy process itself.
2. Add an option to the domains - when enabling HTTP request logging, have a checkbox to log these requests only to a per domain traditional logfile and not to the central OPNsense logfile at all.

Might even send a pull request later this evening, I'll look into it. Unless you like the idea and you are faster,

Kind regards, Patrick

P.S. Initial motivation was that I had CrowdSec in a distributed setup with a log collector on my standalone Caddy and engine and bounce on OPNsense, but both the OPNsense plugin as well as the FreeBSD package notoriously overwrite the local API settings with "localhost" instead of a remote engine, so I was always busy fixing the setup. Looks like it's best run on a single system.

[Monviech]

This requires some more logic in the Caddyfile template. Right now, the http access logs are added via "include" with the UUID of said domain. That in return makes these logs appear in the syslog-ng logs.

https://github.com/Monviech/os-caddy-plugin/blob/dd2a6741c375b44e65bd3c6558041c7a1440eaa2/usr/plugins/devel/caddy/src/opnsense/service/templates/Pischem/Caddy/Caddyfile#L10-L11

https://github.com/Monviech/os-caddy-plugin/blob/dd2a6741c375b44e65bd3c6558041c7a1440eaa2/usr/plugins/devel/caddy/src/opnsense/service/templates/Pischem/Caddy/Caddyfile#L210-L213

Maybe a global option could be added, that would be like "Log HTTP Access Logs to seperate Files", or something like that. It would then write actual "log option blocks" with the UUID.log of the domain as log files into each Domain that has the "Http Access Logs" checkbox enabled, instead of creating the include statement in the main log.

Example:

example.com {
    log {
    output file /var/log/access_UUID.log {
        roll_size 10mb
        roll_keep 5
        roll_keep_for 720h
    }
}

This can be a little tricky, the template is already the hardest part (for me) to maintain. But I think its possible without making it too convoluted. Also, it's important to work with UUIDs in the log names, since only domain names are a little dangerous since Caddy hates duplicate names.

[pmhausen]

Good idea. The template is not that hard, actually. You can loop in Jinja.

Using the access uuid for the filename is also a great idea, that avoids problems with wildcard domains etc. And the logs are parsed automatically, anyway, and if you must look, there's grep.

I'll look into it. Isn't Open Source wonderful? :)

[Monviech]

Sure you can try, if you're stuck I can give it a shot too. ^^

[pmhausen]

Done - thanks.