Error importing logs into Elastic on Centos 7

[Keithsc]

Hi I am attempting to get our logs into an ElasticSearch V5 server but am getting the following error. could you help ?

nxtool-ng-master]# python ./nxtool.py --flat-file /root/Z.log --elastic-dest
Traceback (most recent call last):
  File "./nxtool.py", line 153, in <module>
    sys.exit(main())
  File "./nxtool.py", line 129, in main
    destination.insert([log])
  File "/root/nxtool-ng-master/nxtool/log_providers/__init__.py", line 53, in insert
    self.nList.extend(obj)
AttributeError: 'Elastic' object has no attribute 'nList'

Thanks Keith

[sabban]

Thank you for your report, it should be corrected by my last commit.

[jvoisin]

@sabban Did you prove it with tests? :>

[sabban]

@jvoisin Yes, i could reproduce it, and squash it. What I can't understand is how I let this bug reach the repo in the first place...

[Keithsc]

Hi, Almost there I think, well I am getting a different error anyway.

ElasticSearch 127.0.0.1:9200 = "lucene_version" : "5.5.4"

[elastic] host = 127.0.0.1:9200 use_ssl = false index = nxapi version = 5

python ./nxtool.py --flat-file /root/Z.log --elastic-dest Traceback (most recent call last): File "./nxtool.py", line 153, in sys.exit(main()) File "./nxtool.py", line 130, in main destination.stop() File "/root/nxtool-ng-master/nxtool/log_providers/init.py", line 65, in stop self.commit() File "/root/nxtool-ng-master/nxtool/log_providers/elastic.py", line 196, in commit self.client.bulk(body=items) File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 785, in bulk doc_type, '_bulk'), params=params, body=self._bulk_body(body)) File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 110, in perform_request self._raise_error(response.status, raw_data) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 114, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception', u'Malformed action/metadata line [3], expected START_OBJECT or END_OBJECT but found [VALUE_STRING]')

Thanks again for you help. Keith

On 08/06/17 12:07, Manuel Sabban wrote:

Thank you for your report, it should be corrected by my last commit.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/nbs-system/nxtool-ng/issues/29#issuecomment-307072269, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFUXkCxKQuTkmG3-LwzdgwmBw8DOhtOhks5sB9XhgaJpZM4Nz3zW.

---

This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit http://support.nhs.net/joiningnhsmail

---

[sabban]

I suspect an ES version problem. I tried to make our code more ES-version agnostic. Can you try our fix_es branch I just created ?

Thank you for your testing.

[Keithsc]

The fix_es branch seems to work, It managed to use file source and use ES as a destination without any errors and Iit also seems to work using ES as a source.

python ./nxtool.py --flat-file /root/Z.log --elastic-dest python ./nxtool.py --elastic-source

I have another few ES instances I will try this with later. It seems to work on v5.5.4 but later I will try v6.5.0 and v5.5.2 and report back.

Thanks Keith.

On 08/06/17 16:02, Manuel Sabban wrote:

I suspect an ES version problem. I tried to make our code more ES-version agnostic. Can you try our fix_es branch I just created ?

Thank you for your testing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/nbs-system/nxtool-ng/issues/29#issuecomment-307130987, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFUXkD6x521bbW8AUzEPpPX6KGpH6DmAks5sCAztgaJpZM4Nz3zW.

---

This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit http://support.nhs.net/joiningnhsmail

---

[Keithsc]

The es_fix branch appears to work with v5.5.2 as a source and destination but v6.5.0 fails with I try to use ES as a source.

In the config.cfg file I tried setting version = 5 and version = 6. but they both failed.

[root@unassigned nxtool-ng-fix_es]# python ./nxtool.py --elastic-source Traceback (most recent call last): File "./nxtool.py", line 153, in sys.exit(main()) File "./nxtool.py", line 149, in main print(printers.print_generic(source.get_results())) File "/root/nxtool-ng-fix_es/nxtool/printers.py", line 17, in print_generic for item in it: File "/usr/lib/python2.7/site-packages/elasticsearch_dsl/search.py", line 664, in scan self._params File "/usr/lib/python2.7/site-packages/elasticsearch/helpers/init.py", line 279, in scan request_timeout=request_timeout, kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/utils.py", line 69, in _wrapped return func(*args, params=params, **kwargs) File "/usr/lib/python2.7/site-packages/elasticsearch/client/init.py", line 539, in search doc_type, '_search'), params=params, body=body) File "/usr/lib/python2.7/site-packages/elasticsearch/transport.py", line 327, in perform_request status, headers, data = connection.perform_request(method, url, params, body, ignore=ignore, timeout=timeout) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/http_urllib3.py", line 110, in perform_request self._raise_error(response.status, raw_data) File "/usr/lib/python2.7/site-packages/elasticsearch/connection/base.py", line 114, in _raise_error raise HTTP_EXCEPTIONS.get(status_code, TransportError)(status_code, error_message, additional_info) elasticsearch.exceptions.RequestError: TransportError(400, u'illegal_argument_exception', u'No search type for [scan]')

On 08/06/17 16:02, Manuel Sabban wrote:

I suspect an ES version problem. I tried to make our code more ES-version agnostic. Can you try our fix_es branch I just created ?

Thank you for your testing.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://github.com/nbs-system/nxtool-ng/issues/29#issuecomment-307130987, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AFUXkD6x521bbW8AUzEPpPX6KGpH6DmAks5sCAztgaJpZM4Nz3zW.

---

This message may contain confidential information. If you are not the intended recipient please inform the sender that you have received the message in error before deleting it. Please do not disclose, copy or distribute information in this e-mail or take any action in reliance on its contents: to do so is strictly prohibited and may be unlawful.

Thank you for your co-operation.

NHSmail is the secure email and directory service available for all NHS staff in England and Scotland NHSmail is approved for exchanging patient data and other sensitive information with NHSmail and other accredited email services. NHSmail provides an email address for your career in the NHS and can be accessed anywhere For more information and to find out how you can switch, visit http://support.nhs.net/joiningnhsmail

---

[sabban]

Hi, The 6.x releases aren't mentioned in the compatibility list of python-elasticsearch which is used by elasticsearch_dsl, I have no clue to solve the issue right now, but I can try to improve things a little later, stay tuned.

Thank your for your testing

[jvoisin]

The (last and recent) release of elasticearch_dsl only supports Elastic up to 5.3, while the last release is actually 5.4.

Why are you using something that isn't released yet :'(

I don't think that there is much that we can do, except marking it play nice with the last Elastic version :/

[sabban]

By the way, your issue is related to display content in ES. The whitelist generation may work even if display don't: python ./nxtool.py --elastic-source --whitelist may work.