Bug on whitelist generation with flat file

[forting]

This is the bug:

root@machine:/var/opt/nxtool-ng# python nxtool.py --whitelist --flat-file /var/log/nginx/site/error.log 

[+] ['2017/06/16 11:43:01 [error] 8090#0: *717302 access forbidden by rule, client: 1.2.3.4, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 11:43:01 [error] 8090#0: *717302 access forbidden by rule, client: 1.2.3.4, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] ['2017/06/16 12:10:04 [error] 8765#0: *717762 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:10:04 [error] 8765#0: *717762 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] ['2017/06/16 12:10:15 [error] 8764#0: *717763 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /ecrire/ HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:10:15 [error] 8764#0: *717763 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /ecrire/ HTTP/1.1", host: "site"

[+] ['2017/06/16 12:14:32 [error] 8798#0: *718044 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"\n is an invalid extlog or nxlog, string "ip=" not found.'] while parsing 2017/06/16 12:14:32 [error] 8798#0: *718044 access forbidden by rule, client: 5.6.7.8, server: site, request: "GET /spip.php?page=login&url=ecrire%2F HTTP/1.1", host: "site"

[+] Generating Google analytics rules
[+] Generating Image 1002 rules
[+] Generating array-like variable name rules
Traceback (most recent call last):
  File "nxtool.py", line 145, in <module>
    sys.exit(main())
  File "nxtool.py", line 130, in main
    rules = module.generate_whitelist(source, whitelist)
  File "/var/opt/nxtool-ng/nxtool/whitelists_generators/__init__.py", line 4, in wrapper
    return func(provider, wl)
  File "/var/opt/nxtool-ng/nxtool/whitelists_generators/array_like_variables_names.py", line 35, in generate_whitelist
    variables = provider.get_top('var_name')
  File "/var/opt/nxtool-ng/nxtool/log_providers/flat_file.py", line 63, in get_top
    for key, value in collections.Counter(values).most_common(10):
  File "/usr/lib/python2.7/collections.py", line 453, in __init__
    self.update(iterable, **kwds)
  File "/usr/lib/python2.7/collections.py", line 534, in update
    for elem in iterable:
  File "/var/opt/nxtool-ng/nxtool/log_providers/flat_file.py", line 62, in <genexpr>
    values = (log[field] for log in self.__get_filtered_logs())
KeyError: 'var_name'

error.txt

[sabban]

Can you provide us an (anonymized) logfile that triggers this bug ?

Thanks.

[forting]

It's in the joined file error.txt

Thanks Manuel

[sabban]

nxapi-ng have to be synced in the same time as nxtool-ng. Thanks for the report.