running error with FILTER

[codeooo]

hi,

I used the below command line to parse log files:

   haproxy_log_analysis -l sh35.log -f slow_requests[100]

the command returned the message:

Traceback (most recent call last):
  File "/usr/bin/haproxy_log_analysis", line 11, in <module>
    sys.exit(console_script())
  File "/usr/lib/python2.7/site-packages/haproxy/main.py", line 289, in console_script
    main(arguments)
  File "/usr/lib/python2.7/site-packages/haproxy/main.py", line 274, in main
    for command in args['commands']:
TypeError: 'NoneType' object is not iterable

my haproxy logfile example:

Sep 30 03:20:19 localhost haproxy[116914]: 116.236.250.43:11298 [30/Sep/2016:03:20:19.296] sg-in server_sg/sg185 0/0/1/25/+26 200 +210 - - --NI 1/1/1/1/0 0/0 {sg.wxmgo.com.cn|282|} {
Apache-Coyote/1.1||} "POST /service-gate/gate/invoke HTTP/1.1"

please help me to resolve the problem, thank you.

haproxy_log_analysis version is 2.0

[gforcada]

@codeooo you are missing a command option, i.e.:

haproxy_log_analysis -l sh35.log -f slow_requests[100] -c print

This will filter as you like and print the full line, if you only want to count how many lines match your filter use counter instead.

[novferdianto]

Hi, I'm always getting empty response when running this script

My haproxy log look like this: Oct 17 23:59:59 localhost.localdomain haproxy[407393]: 180.243.52.146:14480 [17/Oct/2016:23:59:59.662] ft_web http_main/ws_main 14/0/0/248/262 200 23091 - - --VN 2641/687/10/5/0 0/0 {Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.143 Safari/537.36|example.com} "GET /hello.jpg HTTP/1.1"

when I run "haproxy_log_analysis -l test3.txt -c counter_invalid" it returns "command: counter_invalid" ========================" "1"

when I run "haproxy_log_analysis -l test3.txt -c counter" it returns "command: counter" "================" "0"

and when I run "haproxy_log_analysis -l test3.txt -c status_codes_counter" it returns "command: status_codes_counter" "=============================" "defaultdict(<type 'int'>, {})"

I'm using the latest haproxy_log_analysis

[gforcada]

@novferdianto if counter_invalid commands gives you 1 and you only have one line on the log file it means that, for whatever reason, haproxy_log_analysis was not able to parse that line.

[novferdianto]

hi @gforcada, i have a huge log on haproxy, i just send you 1 line. but when i run counter, i always getting 0 result and when i run counter_invalid i got many result as same as total line on my haproxy log

[gforcada]

@novferdianto yes I did understand that, the problem is that the format your log lines are is different from what haproxy_log_analysis expects, thus it can not parse your lines.

Would it be possible to reformat your log lines to something that haproxy_log_lines accepts? I would love to have more time to work on haproxy_log_analysis to make it more flexible, but so far I did not have nor I expect to have

If you look at the tests folder there are some minimal examples of parseable log lines, you could play with your log line and see what needs to be changed so that haproxy_log_analysis can parse it and then make the same changes on all the files

I know is cumbersome, but is the only thing I can do for you right now :-(

[gforcada]

@novferdianto it's your lucky day. Out of curiosity I just checked with haproxy_log_analysis what was wrong with your log line. I kept firing haproxy_log_analysis -l FILENAME -c counter while changing suspicious characters/parts of the log line you provided (thanks for that!) and after 5 minutes I could see that it was totally trivial your problem :-)

So without further ado: haproxy_log_analysis 2.0.1 is out get it while is hot! :-)

[novferdianto]

Hi @gforcada , Cool !.. This night I will try the new one.. 👍 👍