search

Azure / azure-quickstart-templates

Azure Quickstart Templates
https://aka.ms/azqst
MIT License
14.06k stars 16.12k forks source link

application-gateway-logviewer-goaccess - Error Parsing Log Entries #8550

Open davehcdn opened 3 years ago

davehcdn commented 3 years ago

application-gateway-logviewer-goaccess

Issue

It seems like the template deployed, but no report.html was produced. I navigated to /var/log/azure/Microsoft.Azure.Networking.ApplicationGateway.LogProcessor/application.log where it looks like there is an error every time that the Microsoft.Azure.Networking.ApplicationGateway.LogProcessor attempts to Parse lines from the log file.

It appears that the Processor is able to go and fetch the files successfully, considering it is having problems parsing real data. I assume that there must be an issue with the custom log format developed in the .zip? My WAF is Azure V2.

Error Example

2020-11-25 15:21:41,512 INFO [6] ?.? - ProcessBlobCache: BlobContext: Microsoft.Azure.Networking.ApplicationGateway.LogProcessor.CloudBlobContext 2020-11-25 15:21:41,513 INFO [5] ?.? - PostBlobData: Starting Processing File: REMOVED 2020-11-25 15:21:41,568 ERROR [5] ?.? - PostBlobData: Error while processing line : { "timeStamp": "2020-11-25T15:00:04+00:00", "resourceId": "/SUBSCRIPTIONS/sanitized/RESOURCEGROUPS/MPPCCORENETRG/PROVIDERS/MICROSOFT.NETWORK/APPLICATIONGATEWAYS/MPPCWAFGW", "listenerName": "sanitized", "ruleName": "sanitized", "backendPoolName": "sanitized", "backendSettingName": "sanitized", "operationName": "ApplicationGatewayAccess", "category": "ApplicationGatewayAccessLog", "properties": {"instanceId":"appgw_3","clientIP":"sanitized","clientPort":sanitized,"httpMethod":"GET","originalRequestUriWithArgs":"\/available","requestUri":"\/available","requestQuery":"","userAgent":"Mozilla\/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit\/537.36 (KHTML, like Gecko) Chrome\/87.0.4280.66 Safari\/537.36","httpStatus":200,"httpVersion":"HTTP\/2.0","receivedBytes":300,"sentBytes":663,"timeTaken":0.008,"transactionId":"sanitized","sslEnabled":"on","sslCipher":"ECDHE-RSA-AES128-GCM-SHA256","sslProtocol":"TLSv1.2","sslClientVerify":"NONE","sslClientCertificateFingerprint":"","sslClientCertificateIssuerName":"","serverRouted":"sanitized:443","serverStatus":"200","serverResponseLatency":"0.008","originalHost":"sanitized","host":"sanitized"}} 2020-11-25 15:21:41,575 ERROR [5] ?.? - PostBlobData: Caught Exception: System.NullReferenceException: Object reference not set to an instance of an object. at Microsoft.Azure.Networking.ApplicationGateway.LogProcessor.CustomLogProcessor.ProcessLog(String logLine, Boolean& transactionComplete) in E:\src\networkmonitoring\Tools\src\AppGatewayLogProcessor\Services\CustomLogProcessor.cs:line 88 at Microsoft.Azure.Networking.ApplicationGateway.LogProcessor.Program.PostBlobData(Object state) in E:\src\networkmonitoring\Tools\src\AppGatewayLogProcessor\Program.cs:line 251

Reproduce:

I think that if you deploy the template yourself using the instructions provided (be specific in the SAS token section) you will encounter the same issue, seeing as there is nothing special with my WAF.

yovko123 commented 3 years ago

We're experiencing the same issue. Did you manage to resolve it?

davehcdn commented 3 years ago

We're experiencing the same issue. Did you manage to resolve it?

Nope. 🦗🦗. Still unassigned.

btrimmell-mtmrecognition commented 1 year ago

Anyone find a solution for this - I just tried it and the AGW logs are getting pulled but GoAccess does not like the log format

btrimmell-mtmrecognition commented 1 year ago

I found a solution for me it is a bad custom log format that is in the deployment script. They have this --log-format='"%dT%tZ"{%^:"%h",%^:"%m",%^:"%U",%^:"%q",%^:"%u",%^:"%s",%^:"%H",%^:"%b",%^:"%T",%^:%v}' but the datetime output but the process like this 2022-11-23T05:14:08+00:00 so the format is nto right I changed mine to --log-format='"%dT%t+00:00"{%^:"%h",%^:"%m",%^:"%U",%^:"%q",%^:"%u",%^:"%s",%^:"%H",%^:"%b",%^:"%T",%^:%v}' and all is good.

My next questionis why have they not updated this tmeplate to deploy a more current version of GoAccess not one that is years old - 1.2 released in 2017?