07/13/2024

Due to the other priorities and projects I will be updating this code at least once a month

GoAccess for Nginx Proxy Manager Logs

Still in development... You might need to wait a bit if you have a large amount of logs for it to parse.

Parses the following log types:

• NPM
• NPM Redirection
• NPM Error
• Traefik
• Load your own custom config as well to parse other logs

Dependencies:

• GoAccess version: 1.9.3
• GeoLite2-City.mmdb (2024-09-23)
• GeoLite2-Country.mmdb (2024-09-23)
• GeoLite2-ASN.mmdb (2024-09-23)

Docker

• Image: https://hub.docker.com/r/xavierh/goaccess-for-nginxproxymanager
• OS/ARCH
  • linux/amd64
  • linux/arm/v7
  • linux/arm64/v8
• ARM also available from Just5KY - justsky/goaccess-for-nginxproxymanager
• Tags: https://hub.docker.com/r/xavierh/goaccess-for-nginxproxymanager/tags
  • stable - xavierh/goaccess-for-nginxproxymanager:latest
  • latest stable development - xavierh/goaccess-for-nginxproxymanager:develop

Github Repo

• https://github.com/xavier-hernandez/goaccess-for-nginxproxymanager

version: '3.3'
services:
    goaccess:
        image: 'xavierh/goaccess-for-nginxproxymanager:latest'
        container_name: goaccess
        restart: always
        ports:
            - '7880:7880'
        environment:
            - TZ=America/New_York         
            - SKIP_ARCHIVED_LOGS=False #optional
            - DEBUG=False #optional
            - BASIC_AUTH=False #optional
            - BASIC_AUTH_USERNAME=user #optional
            - BASIC_AUTH_PASSWORD=pass #optional   
            - EXCLUDE_IPS=127.0.0.1 #optional - comma delimited 
            - LOG_TYPE=NPM #optional - more information below
            - ENABLE_BROWSERS_LIST=True #optional - more information below
            - CUSTOM_BROWSERS=Kuma:Uptime,TestBrowser:Crawler #optional - comma delimited, more information below
            - HTML_REFRESH=5 #optional - Refresh the HTML report every X seconds. https://goaccess.io/man
            - KEEP_LAST=30 #optional - Keep the last specified number of days in storage. https://goaccess.io/man
            - PROCESSING_THREADS=1 #optional - This parameter sets the number of concurrent processing threads in the program's execution, affecting log data analysis, typically adjusted based on CPU cores. Default is 1. https://goaccess.io/man
        volumes:
        - /path/to/host/nginx/logs:/opt/log
        - /path/to/host/custom:/opt/custom #optional, required if using log_type = CUSTOM

If you have permission issues, you can add PUID and PGID with the correct user id that has read access to the log files.

version: '3.3'
services:
    goaccess:
        image: 'xavierh/goaccess-for-nginxproxymanager:latest'
        container_name: goaccess
        restart: always
        ports:
            - '7880:7880'
        environment:
            - PUID=0
            - PGID=0
            - TZ=America/New_York         
            - SKIP_ARCHIVED_LOGS=False #optional
            - DEBUG=False #optional
            - BASIC_AUTH=False #optional
            - BASIC_AUTH_USERNAME=user #optional
            - BASIC_AUTH_PASSWORD=pass #optional   
            - EXCLUDE_IPS=127.0.0.1 #optional - comma delimited 
            - LOG_TYPE=NPM #optional - more information below
            - ENABLE_BROWSERS_LIST=True #optional - more information below
            - CUSTOM_BROWSERS=Kuma:Uptime,TestBrowser:Crawler #optional - comma delimited, more information below
            - HTML_REFRESH=5 #optional - Refresh the HTML report every X seconds. https://goaccess.io/man
            - KEEP_LAST=30 #optional - Keep the last specified number of days in storage. https://goaccess.io/man
            - PROCESSING_THREADS=1 #optional - This parameter sets the number of concurrent processing threads in the program's execution, affecting log data analysis, typically adjusted based on CPU cores. Default is 1. https://goaccess.io/man
        volumes:
        - /path/to/host/nginx/logs:/opt/log
        - /path/to/host/custom:/opt/custom #optional, required if using log_type = CUSTOM

Additional environment information

-e LOG_TYPE=

• Options:

  • CUSTOM

  • this feature will load your own configuration

  • an additional volume must be included

    • /path/to/host/custom:/opt/custom

  • volume should include

    • goaccess.conf
      • this is your custom config
      • container will exit if no file is found
      • leave the default port number at 7890

  • environment parameters that will not work and will be ignored

    • SKIP_ARCHIVED_LOGS
    • EXCLUDE_IPS

  • NPM (default if variable is empty or not included)

  • the following file(s) are read and parsed.

    • proxy-host-*_access.log.gz
    • proxy-host-*_access.log
    • proxy*host-*.log

  • NPM+R

  • a second instance of GOACCESS is created

  • append "/redirection" to the url to access the instance, for example http://localhost:7880/redirection/

  • the following file(s) are read and parsed:

    • redirection*host-*.log*.gz
    • redirection*host-*.log
    • fallback_access.log*.gz
    • fallback_access.log
    • dead-host*.log*.gz
    • dead-host*.log

  • NPM+ALL

  • a second and third instance of GOACCESS are created

    • append "/redirection" to the url to access the instance, for example http://localhost:7880/redirection/
    • the following file(s) are read and parsed.
      • redirection*host-.log.gz
      • redirection*host-*.log
    • append "/error" to the url to access the instance, for example http://localhost:7880/error/
    • the following file(s) are read and parsed.
      • *_error.log*.gz
      • *_error.log
    • "error" log files sometimes have inconsistent log types and there isn't a way to process these. GoAccess does process files that have at least 1 error log in the files in the correct format. Viewing the docker container logs will tell you which files have been skipped.

  • TRAEFIK

  • environment parameters that will not work and will be ignored

    • SKIP_ARCHIVED_LOGS

  • the following file(s) are read and parsed.

    • access.log

  • You can also set the following environment variables yourself to override the defaults

    • TIME_FORMAT
    • DATE_FORMAT
    • LOG_FORMAT

    FOR EXAMPLE
    ---------------------------------
    TIME_FORMAT=%H:%M:%S
    DATE_FORMAT=%Y-%m-%d
    LOG_FORMAT='{"ClientHost": "%h", "ClientUsername": "%e", "DownstreamContentSize": "%b", "Duration": "%n", "DownstreamStatus": "%s", "RequestAddr": "%v", "RequestMethod": "%m", "RequestPath": "%U", "RequestProtocol": "%H", "StartUTC": "%dT%t.%^", "request_User-Agent": "%u"}'

  • NCSA_COMBINED

  • environment parameters that will not work and will be ignored

    • SKIP_ARCHIVED_LOGS

  • by default the following file(s) are read and parsed.

    • *.log

  • NGINX_ACCESS

  • the following file(s) are read and parsed.

    • access.log

LOG FORMATS

NPM PROXY LOG FORMAT

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %^ %^ %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] [Sent-to %^] "%u" "%R"

NPM REDIRECTION LOG FORMAT

time-format %T
date-format %d/%b/%Y
log_format [%d:%t %^] %s - %m %^ %v "%U" [Client %h] [Length %b] [Gzip %^] "%u" "%R"

NPM ERROR LOG FORMAT

time-format %T
date-format %Y/%m/%d
log_format %d %t %^: %v, %^: %h, %^ %v %^"%r" %^

TRAEFIK ACCESS LOG FORMAT

time-format %T
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u" %Lm"

NGINX ACCESS LOG FORMAT

time-format %T
date-format %d/%b/%Y
log-format %h %^[%d:%t %^] "%r" %s %b "%R" "%u"

Possible/Known Issues

• A lot of CPU Usage and 10000 request every second in webUI
  • https://github.com/xavier-hernandez/goaccess-for-nginxproxymanager/issues/38
• If your using NPM to proxy the container you need to turn on websockets support
  • https://github.com/xavier-hernandez/goaccess-for-nginxproxymanager/issues/69
• LOG_TYPE=NPM+ALL
  • "error" log files sometimes have inconsistent log types and there isn't a way to process these. GoAccess does process files that have at least 1 error log in the files in the correct format. Viewing the docker container logs will tell you which files have been skipped.
• Debug=True
  • The version of this application, (GOAN vX), does not get displayed in the left side toolbar on purpose. In debug mode I don't want many customizations.

Thanks

To https://github.com/GregYankovoy for the inspiration, and for their nginx.conf :)

Users:

• Just5KY
• martadinata666

Disclaimer

This product includes GeoLite2 data created by MaxMind, available from https://www.maxmind.com.