Looking for help to fine tune goaccess log format for a custom log pattern

[SureshCharan]

Hi,

I have been exploring goaccess for last couple of days to leverage it to analyze the log files created by one of the open source projects from ONAP community. The project uses a custom log pattern and below is the custom log pattern used:

<configuration>
    <property name="AJSC_HOME" value="${AJSC_HOME:-.}" />
    <appender name="ACCESS"
        class="ch.qos.logback.core.rolling.RollingFileAppender">
        <file>${AJSC_HOME}/logs/ajsc-jetty/localhost_access.log</file>
        <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
            <fileNamePattern>${AJSC_HOME}/logs/ajsc-jetty/localhost_access.log.%d{yyyy-MM-dd}
            </fileNamePattern>
        </rollingPolicy>
        <encoder class="org.onap.aai.logging.CustomLogPatternLayoutEncoder">
            <Pattern>%a %u %z [%t] "%m %U%q" %s %b %y %i{X-TransactionId} %i{X-FromAppId} %i{X-Forwarded-For} %i{X-AAI-SSL-Client-CN} %i{X-AAI-SSL-Client-OU} %i{X-AAI-SSL-Client-O} %i{X-AAI-SSL-Client-L} %i{X-AAI-SSL-Client-ST} %i{X-AAI-SSL-Client-C} %i{X-AAI-SSL-Client-NotBefore} %i{X-AAI-SSL-Client-NotAfter} %i{X-AAI-SSL-Client-DN} %D</Pattern>
        </encoder>
    </appender>
    <appender-ref ref="ACCESS" />
</configuration>

<!-- 
%a - Remote IP address
%A - Local IP address
%b - Bytes sent, excluding HTTP headers, or '-' if no bytes were sent
%B - Bytes sent, excluding HTTP headers
%h - Remote host name
%H - Request protocol
%l - Remote logical username from identd (always returns '-')
%m - Request method
%p - Local port
%q - Query string (prepended with a '?' if it exists, otherwise an empty string
%r - First line of the request
%s - HTTP status code of the response
%S - User session ID
%t - Date and time, in Common Log Format format
%u - Remote user that was authenticated
%U - Requested URL path
%v - Local server name
%I - current request thread name (can compare later with stacktraces)

%z - Custom pattern that parses the cert for the subject
%y - Custom pattern determines rest or dme2
 -->

You can have a look at the pattern and its configuration at here.

And below is the sample log:

<Remote IP Address> - - [14/Dec/2021:16:01:30 +0000] "GET /aai/v16/network/pnfs/pnf/XXXXX?depth=3" 200 1048 REST APP_1 AAI <SOME IP>, <SOME IP>          28

<Remote IP Address> - - [14/Dec/2021:16:01:30 +0000] "GET /aai/v16/network/pnfs/pnf/YYYYYY?depth=3" 200 1103 REST APP_2 AAI <SOME IP> <SOME IP>          18

<Remote IP Address> - - [14/Dec/2021:16:01:30 +0000] "GET /aai/v16/network/physical-links/physical-link/ZZZZZZZ" 200 1301 REST APP_2 AAI <SOME IP> <SOME IP>          20

Note that the last column is the response time in milliseconds. Following the documentation and the responses provided in other issues, I have come up with the following log format, date format, and time formats.

DATE_FORMAT='%d/%b/%Y'

TIME_FORMAT='%H:%M:%S'

LOG_FORMAT='%h %^ %^ [%d:%t %^] "%m %U" %s %b %^ %u %^ %^ %^ %^ %^ %^ %^ %^ %^ %^ %^ %T'

I would like to get the average, cumulative, and maximum time served in GoAccess. Also, I would like to configure goaccess to consider APP_1 and APP_2 as different visitors. I am looking for some help here to validate if the above-mentioned formats are proper or do I have to make any modifications to make it perfect to work with goaccess?

Thanks in advance.

[allinurl]

Just to be sure, are these tab separated? If they are, are you able to attach a text file with a few lines straight from the log. e.g.,

tail -10 access.log > test.log

and upload the test log, it's just easier to work with it. Thanks

[SureshCharan]

No. They are not tab-separated. Attached is a sample log file. test.log

[allinurl]

Honestly I couldn't match the specifiers you posted and the actual sample test.log. Though, I'd try this:

goaccess test.log --log-format='%h %^[%d:%t %^] "%m %U" %s %b %^ %v %^' --date-format=%d/%b/%Y --time-format=%T --http-protocol=no