mlr: syntax error: unwrapped double quote at line 49078 for puppet-top-api-calls.sh

jordi-garcia commented 3 years ago

Describe the Bug

The script is expecting a set of data formatted in a particular way e.g. IP , Date, - , ... and when there is a block of data missing it will try to make transformations on the data probably based on positioning and throw an exception. The above error was produced as there was at least a line with a missing IP.

Expected Behavior

Script could validate data formatted as expected and provide option on the fly or less preferable ignore the line - bit dangerous as one may not have visibility of how much data has been left out in the analysis

Steps to Reproduce

puppetserver-access log reference

For example,

Bad reference

- - [29/Mar/2021:01:25:43 +0200] "GET /packages/2019.8.1/debian-8-amd64/dists/jessie/puppet6/binary-amd64/Packages.gz HTTP/1.1" 416 0 "-" "Debian APT-CURL/1.0 (1.0.9.8.6)" 1 - -

versus good reference

10.0.0.238 - - [29/Mar/2021:01:25:43 +0200] "GET /puppet/v3/file_metadata/modules/supervisor/supervisord.conf?links=manage&checksum_type=md5&source_permissions=ignore&environment=production HTTP/1.1" 200 365 "-" "Puppet/6.17.0 Ruby/2.5.8-p224 (x86_64-linux)" 4 - 2

our file had 10 lines where the IP was missing. We opted for ignoring those lines by adding | grep -E -v "^ -" to line 69 of puppet-top-api-calls.sh to read "${read_cmd[@]}" "$file" | grep -E -v "^ -" \

Environment

Version [e.g. 1.27.0]
Platform [e.g. Ubuntu 18.04]

Additional Context

Add any other context about the problem here.

m0dular commented 3 years ago

Do you have a support script with these logs? I don't think I've ever seen PE log a non-standard line to the access logs, so that's interesting.

This might be a pretty large ask. We expect the logs to be in common log format, and I don't think I've encountered a case where they weren't. Unless there's a CLF regex we can easily use, it may be a lot of work for not a lot of gain.

jordi-garcia commented 3 years ago

Adrian,

Box folder (https://puppetlabs.zendesk.com/agent/apps/box) for Smartbit BVBA - HQ folder 43787 puppetserver.log file from support script, for context, doesn't let me attach it here unfortunately.

I was thinking more about a simple level of validation that it is repeatable across logs. Also, I could have a go at it for you too. Best regards,

Jordi Garcia Senior DevOps Engineer @.***

On Wed, Mar 31, 2021 at 4:41 PM Adrian Parreiras Horta < @.***> wrote:

Do you have a support script with these logs? I don't think I've ever seen PE log a non-standard line to the access logs, so that's interesting.

This might be a pretty large ask. We expect the logs to be in common log format, and I don't think I've encountered a case where they weren't. Unless there's a CLF regex we can easily use, it may be a lot of work for not a lot of gain.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/puppetlabs/support-script-tooling/issues/3#issuecomment-811166265, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARXHGR6KHZSBY7X62BPRJ5DTGM7CBANCNFSM42EXILHQ .

m0dular commented 3 years ago

Interesting, HTTP 416 looks like an odd return code that I've never seen in PE logs.

I'd love to have more people contribute. What kind of validation were you thinking? We may be able to use a regex from here, if it works and is portable. https://www.oreilly.com/library/view/regular-expressions-cookbook/9781449327453/ch07s12.html

m0dular commented 3 years ago

I was running into this more often than I thought, so I added a fix in this PR. @jordi-garcia could you try it out?

https://github.com/puppetlabs/support-script-tooling/pull/6

m0dular commented 3 years ago

Fixed in #6 :+1:

puppetlabs / support-script-tooling