search

allinurl / goaccess

GoAccess is a real-time web log analyzer and interactive viewer that runs in a terminal in *nix systems or through your browser.
https://goaccess.io
MIT License
18.28k stars 1.1k forks source link

Log format configuration help #1856

Open hopewise opened 4 years ago

hopewise commented 4 years ago

I am passing a custom error message into access.log , how can I use goaccess to create report of those error messages?

like for example:

X.X.X.X - - [26/Jul/2020:11:51:58 +0000] "POST /v1/app/ping HTTP/1.1" 403 43 "-" "-" "0.072" "Error message 1" X.X.X.X - - [26/Jul/2020:11:51:58 +0000] "POST /v1/app/ping HTTP/1.1" 403 43 "-" "-" "0.072" "Error message 2" X.X.X.X - - [26/Jul/2020:11:51:58 +0000] "POST /v1/app/ping HTTP/1.1" 403 43 "-" "-" "0.072" "Error message 3" etc ..

So, can I for example create a report to count how many times Error message 1 occurred? including IP address, time, etc.. ? How?

allinurl commented 4 years ago

Sorry for the delay on my response. This should do it:

goaccess access.log --log-format='%h %^[%d:%t %^] "%r" %s %b "%R" "%u" "%T" "%v"' --date-format=%d/%b/%Y --time-format=%T
maverickGeek commented 2 years ago

The common format I see in apache errors logs is: [Thu Sep 08 03:13:19.350456 2022] [proxy_fcgi:error] [pid 19907:tid 139944144115456] [client X.X.X.X:49972] Error message

But I keep getting the data format is incorrect error, for the following: %^[%d %t%^] %^[proxy_fcgi:error%^] %^[pid 3105:tid 139630105073408%^] %^[client %h%^] Date: %a %b %d Time: %T %Y

allinurl commented 2 years ago

@maverickGeek could you please share a few more lines from your error log, including the message. Feel free to replace the ip with a local IP, e.g., 192.168.0.1. Thanks

maverickGeek commented 2 years ago

Local dev error log example:

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 172.18.0.3. Set the 'ServerName' directive globally to suppress this message
[Thu Sep 08 03:22:36.644614 2022] [mpm_prefork:notice] [pid 1] AH00163: Apache/2.4.52 (Debian) PHP/7.4.27 configured -- resuming normal operations
[Thu Sep 08 03:22:36.644668 2022] [core:notice] [pid 1] AH00094: Command line: 'apache2 -D FOREGROUND'
[Thu Sep 08 04:26:25.897780 2022] [mpm_prefork:notice] [pid 1] AH00170: caught SIGWINCH, shutting down gracefully

Production server example:

[Wed Sep 07 23:44:11.838524 2022] [proxy_fcgi:error] [pid 3105:tid 139630105073408] [client 10.10.10.10:38776] AH01071: Got error 'PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payments_on_order function is deprecated since version 2.6.0.PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payment_failure_on_order function is deprecated since version 2.6.0.', referer: https://www.google.com/wp-admin/admin-ajax.php
[Wed Sep 07 23:59:09.979785 2022] [autoindex:error] [pid 5525:tid 139629864879872] [client 10.10.10.10:48018] AH01276: Cannot serve directory /var/www/vhosts/google.com/httpdocs/wp-admin/css/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,index.shtml) found, and server-generated directory index forbidden by Options directive, referer: google.com
[Thu Sep 08 00:00:02.562511 2022] [proxy_fcgi:error] [pid 5525:tid 139629848094464] (104)Connection reset by peer: [client 10.10.10.10:48364] AH01075: Error dispatching request to :, referer: https://googleads.g.doubleclick.net/
[Thu Sep 08 00:00:07.126429 2022] [ssl:warn] [pid 19800:tid 139944546777216] AH01909: RSA certificate configured for google.com:443 does NOT include an ID which matches the server name
[Thu Sep 08 00:00:07.155156 2022] [ssl:warn] [pid 19800:tid 139944546777216] AH01909: RSA certificate configured for google.com:443 does NOT include an ID which matches the server name
[Thu Sep 08 00:00:12.347291 2022] [proxy_fcgi:error] [pid 19806:tid 139944186078976] [client 10.10.10.10:48462] AH01071: Got error 'PHP message: RedisException: socket error on read socket in /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php:523\nStack trace:\n#0 /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php(523): Redis->ping()\n#1 /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php(220): WP_Object_Cache->__construct()\n#2 /var/www/vhosts/google.com/httpdocs/wp-includes/load.php(729): wp_cache_init()\n#3 /var/www/vhosts/google.com/httpdocs/wp-settings.php(131): wp_start_object_cache()\n#4 /var/www/vhosts/google.com/httpdocs/wp-config.php(150): require_once('/var/www/vhosts...')\n#5 /var/www/vhosts/google.com/httpdocs/wp-load.php(50): require_once('/var/www/vhosts...')\n#6 /var/www/vhosts/google.com/httpdocs/wp-admin/admin-ajax.php(22): require_once('/var/www/vhosts...')\n#7 {main}', referer: https://www.google.com/wp-admin/admin-ajax.php
[Thu Sep 08 00:00:16.172888 2022] [proxy_fcgi:error] [pid 19804:tid 139944278431488] [client 10.10.10.10:48456] AH01071: Got error 'PHP message: RedisException: socket error on read socket in /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php:523\nStack trace:\n#0 /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php(523): Redis->ping()\n#1 /var/www/vhosts/google.com/httpdocs/wp-content/object-cache.php(220): WP_Object_Cache->__construct()\n#2 /var/www/vhosts/google.com/httpdocs/wp-includes/load.php(729): wp_cache_init()\n#3 /var/www/vhosts/google.com/httpdocs/wp-settings.php(131): wp_start_object_cache()\n#4 /var/www/vhosts/google.com/httpdocs/wp-config.php(150): require_once('/var/www/vhosts...')\n#5 /var/www/vhosts/google.com/httpdocs/wp-load.php(50): require_once('/var/www/vhosts...')\n#6 /var/www/vhosts/google.com/httpdocs/wp-blog-header.php(13): require_once('/var/www/vhosts...')\n#7 /var/www/vhosts/google.com/httpdocs/index.php(17): require('/var/www/vhosts...')\n#8 {main}'
[Thu Sep 08 00:20:09.778846 2022] [proxy_fcgi:error] [pid 19806:tid 139944144115456] [client 10.10.10.10:32860] AH01071: Got error 'PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payment_failure_on_order function is deprecated since version 2.6.0.', referer: https://www.google.com/wp-admin/admin-ajax.php
[Thu Sep 08 00:45:58.457958 2022] [proxy_fcgi:error] [pid 19907:tid 139944194471680] [client 10.10.10.10:45062] AH01071: Got error 'PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payment_failure_on_order function is deprecated since version 2.6.0.', referer: https://www.google.com/wp-admin/admin-ajax.php
[Thu Sep 08 01:03:05.334496 2022] [proxy_fcgi:error] [pid 19907:tid 139944278431488] [client 10.10.10.10:52442] AH01071: Got error 'PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payment_failure_on_order function is deprecated since version 2.6.0.', referer: https://www.google.com/wp-admin/admin-ajax.php
[Thu Sep 08 01:20:15.819717 2022] [proxy_fcgi:error] [pid 19804:tid 139944118937344] [client 10.10.10.10:60438] AH01068: Got bogus version 108, referer: https://www.google.com/wp-admin/admin.php?page=cartflows&path=flows&paged=2
[Thu Sep 08 01:20:15.819754 2022] [proxy_fcgi:error] [pid 19804:tid 139944118937344] (22)Invalid argument: [client 10.10.10.10:60438] AH01075: Error dispatching request to :, referer: https://www.google.com/wp-admin/admin.php?page=cartflows&path=flows&paged=2
[Thu Sep 08 01:58:20.846639 2022] [proxy_fcgi:error] [pid 15847:tid 139944118937344] [client 10.10.10.10:49008] AH01071: Got error 'PHP message: The WC_Abstract_Legacy_Order::get_product_from_item function is deprecated since version 4.4.0. Replace with $item->get_product().PHP message: The WC_Subscriptions_Manager::process_subscription_payments_on_order function is deprecated since version 2.6.0.', referer: https://www.google.com/wp-admin/admin-ajax.php
[Thu Sep 08 03:13:19.350456 2022] [proxy_fcgi:error] [pid 19907:tid 139944144115456] [client 10.10.10.10:49972] AH01071: Got error 'PHP message: PHP Fatal error:  Uncaught Exception: Unable to change subscription status to "pending-cancel". in /var/www/vhosts/google.com/httpdocs/wp-content/plugins/woocommerce-subscriptions/vendor/woocommerce/subscriptions-core/includes/class-wc-subscription.php:430\nStack trace:\n#0 /var/www/vhosts/google.com/httpdocs/wp-content/plugins/cb-142969-cancel-subscription/class-rdcancelmodal.php(989): WC_Subscription->update_status()\n#1 /var/www/vhosts/google.com/httpdocs/wp-includes/class-wp-hook.php(307): RDCancelModal->ajax_cancel_subscription()\n#2 /var/www/vhosts/google.com/httpdocs/wp-includes/class-wp-hook.php(331): WP_Hook->apply_filters()\n#3 /var/www/vhosts/google.com/httpdocs/wp-includes/plugin.php(476): WP_Hook->do_action()\n#4 /var/www/vhosts/google.com/httpdocs/wp-admin/admin-ajax.php(187): do_action()\n#5 {main}\n  thrown in /var/www/vhosts/google.com/httpdocs/wp-content/plugins/woocommerce-subsc...', referer: https://www.google.com/my-account/view-subscription/638297/
allinurl commented 2 years ago

Looks like there's no structure or consistency on the output so I'm not sure it's worth parsing it, but you can give it a shot and get a few pieces with:

goaccess error.log --log-format='[%d %t.%^] [%e] [%^] [%^ %h:%^] %v: %U\n' --date-format='%a %b %d' --time-format=%T --http-protocol=no --http-method=no --ignore-panel=OS --ignore-panel=BROWSERS --ignore-panel=REFERRING_SITES --ignore-panel=REQUESTS_STATIC --ignore-panel=NOT_FOUND --ignore-panel=STATUS_CODES --date-spec=min --hour-spec=min

2022-09-12 19-27

maverickGeek commented 1 year ago

@allinurl thank you for the help! But yes, after trying it on a few logs, it wasn't useful. Is there a format that you recommend for apache error and access logs? I have control over two servers that I work on frequently, where I can update the default log format.

allinurl commented 1 year ago

@maverickGeek For the access log I like to keep it as:

LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\" %D %{Content-Type}o %{SSL_PROTOCOL}x %{SSL_CIPHER}x" vhost_combined_tls

For error log, I tend to keep the error as the last field

ErrorLogFormat "[%{u}t] [%-m:%l] [pid %P:tid %T] %7F: %E: [client\ %a] %{Referer}i %M%"