search

apache / kibble-scanners

Apache Kibble Scanners
https://kibble.apache.org/
Apache License 2.0
16 stars 11 forks source link

Difficulty setting-up Kibble Scanners #3

Closed ghost closed 4 years ago

ghost commented 4 years ago

Hello,

I've been able to set-up the Kibble Server and Elastic Search on 2 different Ubuntu Machines and a Windows 10 machine by closely following the docs, however I am experiencing a no-joy moment setting-up kibble-scanners.

  1. First issue, the bttf build fails:
cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt
Collecting bttf
  Using cached bttf-0.1.tar.gz (11 kB)
Requirement already satisfied: certifi in /usr/lib/python3/dist-packages (from -r requirements.txt (line 2)) (2019.11.28)
Requirement already satisfied: elasticsearch in /usr/local/lib/python3.8/dist-packages (from -r requirements.txt (line 3)) (7.6.0)
Requirement already satisfied: psutil in /home/cos/.local/lib/python3.8/site-packages (from -r requirements.txt (line 4)) (5.7.0)
Requirement already satisfied: python-dateutil in /usr/lib/python3/dist-packages (from -r requirements.txt (line 5)) (2.7.3)
Requirement already satisfied: requests in /usr/lib/python3/dist-packages (from -r requirements.txt (line 6)) (2.22.0)
Requirement already satisfied: urllib3>=1.21.1 in /usr/lib/python3/dist-packages (from elasticsearch->-r requirements.txt (line 3)) (1.25.8)
Building wheels for collected packages: bttf
  Building wheel for bttf (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-pnzdwahd
       cwd: /tmp/pip-install-z2997c34/bttf/
  Complete output (37 lines):
  /tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
    speedups = Feature(
  /usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
    _setup_distribution = dist = klass(attrs)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.8
  creating build/lib.linux-x86_64-3.8/bttf
  copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf
  running egg_info
  writing bttf.egg-info/PKG-INFO
  writing dependency_links to bttf.egg-info/dependency_links.txt
  writing top-level names to bttf.egg-info/top_level.txt
  reading manifest file 'bttf.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'bttf.egg-info/SOURCES.txt'
  copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf
  copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf
  copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf
  running build_ext
  building 'bttf._bttf' extension
  creating build/temp.linux-x86_64-3.8
  creating build/temp.linux-x86_64-3.8/bttf
  x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o
  In file included from bttf/_bttf.c:2:
  bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
      5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
        | ^~~~~~~~~~~~~~~~~~~~~
  In file included from /usr/include/python3.8/unicodeobject.h:1037,
                   from /usr/include/python3.8/Python.h:97,
                   from bttf/_bttf.c:1:
  /usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
   1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
        |                       ^~~~~~~~~~~~~~~~~~~~~
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  ----------------------------------------
  ERROR: Failed building wheel for bttf
  Running setup.py clean for bttf
Failed to build bttf
Installing collected packages: bttf
    Running setup.py install for bttf ... error
    ERROR: Command errored out with exit status 1:
     command: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf
         cwd: /tmp/pip-install-z2997c34/bttf/
    Complete output (37 lines):
    /tmp/pip-install-z2997c34/bttf/setup.py:15: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
      speedups = Feature(
    /usr/lib/python3.8/distutils/core.py:108: DistDeprecationWarning: Features are deprecated and will be removed in a future version. See https://github.com/pypa/setuptools/issues/65.
      _setup_distribution = dist = klass(attrs)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.8
    creating build/lib.linux-x86_64-3.8/bttf
    copying bttf/__init__.py -> build/lib.linux-x86_64-3.8/bttf
    running egg_info
    writing bttf.egg-info/PKG-INFO
    writing dependency_links to bttf.egg-info/dependency_links.txt
    writing top-level names to bttf.egg-info/top_level.txt
    reading manifest file 'bttf.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'bttf.egg-info/SOURCES.txt'
    copying bttf/_bttf.c -> build/lib.linux-x86_64-3.8/bttf
    copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.8/bttf
    copying bttf/typehacks.h -> build/lib.linux-x86_64-3.8/bttf
    running build_ext
    building 'bttf._bttf' extension
    creating build/temp.linux-x86_64-3.8
    creating build/temp.linux-x86_64-3.8/bttf
    x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.8 -c bttf/_bttf.c -o build/temp.linux-x86_64-3.8/bttf/_bttf.o
    In file included from bttf/_bttf.c:2:
    bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
        5 | _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
          | ^~~~~~~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.8/unicodeobject.h:1037,
                     from /usr/include/python3.8/Python.h:97,
                     from bttf/_bttf.c:1:
    /usr/include/python3.8/cpython/unicodeobject.h:1218:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
     1218 | PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
          |                       ^~~~~~~~~~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /usr/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"'; __file__='"'"'/tmp/pip-install-z2997c34/bttf/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-9r9h__jy/install-record.txt --single-version-externally-managed --user --prefix= --compile --install-headers /home/cos/.local/include/python3.8/bttf Check the logs for full command output.
  1. And then:
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
  File "src/kibble-scanner.py", line 180, in <module>
    main()
  File "src/kibble-scanner.py", line 136, in main
    for org in broker.organisations():
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 339, in organisations
    res = self.DB.search(
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 90, in search
    return self.ES.search(
  File "/usr/local/lib/python3.8/dist-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
TypeError: search() got an unexpected keyword argument '_source_include'
  1. First issue, the bttf build fails:
cos@ubuntu:~/temp/kibble-scanners$ pip3 install -r requirements.txt
Collecting bttf (from -r requirements.txt (line 1))
  Using cached https://files.pythonhosted.org/packages/e0/33/adb62781dcbc9124f0a58ca15d3aaa3bdb5968863e354a62bbd52ea8eb88/bttf-0.1.tar.gz
Collecting certifi (from -r requirements.txt (line 2))
  Using cached https://files.pythonhosted.org/packages/57/2b/26e37a4b034800c960a00c4e1b3d9ca5d7014e983e6e729e33ea2f36426c/certifi-2020.4.5.1-py2.py3-none-any.whl
Collecting elasticsearch (from -r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/cc/cf/7973ac58090b960857da04add0b345415bf1e1741beddf4cbe136b8ad174/elasticsearch-7.6.0-py2.py3-none-any.whl
Collecting psutil (from -r requirements.txt (line 4))
Collecting python-dateutil (from -r requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/d4/70/d60450c3dd48ef87586924207ae8907090de0b306af2bce5d134d78615cb/python_dateutil-2.8.1-py2.py3-none-any.whl
Collecting requests (from -r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/1a/70/1935c770cb3be6e3a8b78ced23d7e0f3b187f5cbfab4749523ed65d7c9b1/requests-2.23.0-py2.py3-none-any.whl
Collecting urllib3>=1.21.1 (from elasticsearch->-r requirements.txt (line 3))
  Using cached https://files.pythonhosted.org/packages/e1/e5/df302e8017440f111c11cc41a6b432838672f5a70aa29227bf58149dc72f/urllib3-1.25.9-py2.py3-none-any.whl
Collecting six>=1.5 (from python-dateutil->-r requirements.txt (line 5))
  Using cached https://files.pythonhosted.org/packages/65/eb/1f97cb97bfc2390a276969c6fae16075da282f5058082d4cb10c6c5c1dba/six-1.14.0-py2.py3-none-any.whl
Collecting chardet<4,>=3.0.2 (from requests->-r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/bc/a9/01ffebfb562e4274b6487b4bb1ddec7ca55ec7510b22e4c51f14098443b8/chardet-3.0.4-py2.py3-none-any.whl
Collecting idna<3,>=2.5 (from requests->-r requirements.txt (line 6))
  Using cached https://files.pythonhosted.org/packages/89/e3/afebe61c546d18fb1709a61bee788254b40e736cff7271c7de5de2dc4128/idna-2.9-py2.py3-none-any.whl
Building wheels for collected packages: bttf
  Running setup.py bdist_wheel for bttf ... error
  Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmp3sczfwj7pip-wheel- --python-tag cp36:
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-3.6
  creating build/lib.linux-x86_64-3.6/bttf
  copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf
  running egg_info
  writing bttf.egg-info/PKG-INFO
  writing dependency_links to bttf.egg-info/dependency_links.txt
  writing top-level names to bttf.egg-info/top_level.txt
  reading manifest file 'bttf.egg-info/SOURCES.txt'
  reading manifest template 'MANIFEST.in'
  writing manifest file 'bttf.egg-info/SOURCES.txt'
  copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf
  copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf
  copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf
  running build_ext
  building 'bttf._bttf' extension
  creating build/temp.linux-x86_64-3.6
  creating build/temp.linux-x86_64-3.6/bttf
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o
  In file included from bttf/_bttf.c:2:0:
  bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
   _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
   ^~~~~~~~~~~~~~~~~~~~~
  In file included from /usr/include/python3.6m/Python.h:89:0,
                   from bttf/_bttf.c:1:
  /usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
   PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
                         ^~~~~~~~~~~~~~~~~~~~~
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

  ----------------------------------------
  Failed building wheel for bttf
  Running setup.py clean for bttf
Failed to build bttf
Installing collected packages: bttf, certifi, urllib3, elasticsearch, psutil, six, python-dateutil, chardet, idna, requests
  Running setup.py install for bttf ... error
    Complete output from command /usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=:
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-3.6
    creating build/lib.linux-x86_64-3.6/bttf
    copying bttf/__init__.py -> build/lib.linux-x86_64-3.6/bttf
    running egg_info
    writing bttf.egg-info/PKG-INFO
    writing dependency_links to bttf.egg-info/dependency_links.txt
    writing top-level names to bttf.egg-info/top_level.txt
    reading manifest file 'bttf.egg-info/SOURCES.txt'
    reading manifest template 'MANIFEST.in'
    writing manifest file 'bttf.egg-info/SOURCES.txt'
    copying bttf/_bttf.c -> build/lib.linux-x86_64-3.6/bttf
    copying bttf/bytesformat.h -> build/lib.linux-x86_64-3.6/bttf
    copying bttf/typehacks.h -> build/lib.linux-x86_64-3.6/bttf
    running build_ext
    building 'bttf._bttf' extension
    creating build/temp.linux-x86_64-3.6
    creating build/temp.linux-x86_64-3.6/bttf
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.6m -c bttf/_bttf.c -o build/temp.linux-x86_64-3.6/bttf/_bttf.o
    In file included from bttf/_bttf.c:2:0:
    bttf/bytesformat.h:5:1: error: static declaration of ‘_PyUnicode_FormatLong’ follows non-static declaration
     _PyUnicode_FormatLong(PyObject *val, int alt, int prec, int type)
     ^~~~~~~~~~~~~~~~~~~~~
    In file included from /usr/include/python3.6m/Python.h:89:0,
                     from bttf/_bttf.c:1:
    /usr/include/python3.6m/unicodeobject.h:2307:23: note: previous declaration of ‘_PyUnicode_FormatLong’ was here
     PyAPI_FUNC(PyObject*) _PyUnicode_FormatLong(PyObject *, int, int, int);
                           ^~~~~~~~~~~~~~~~~~~~~
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-mno2wh38/bttf/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-gge6tlz0-record/install-record.txt --single-version-externally-managed --compile --user --prefix=" failed with error code 1 in /tmp/pip-build-mno2wh38/bttf/
  1. And then:
cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
/usr/lib/python3/dist-packages/requests/__init__.py:80: RequestsDependencyWarning: urllib3 (1.25.9) or chardet (3.0.4) doesn't match a supported version!
  RequestsDependencyWarning)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
  File "src/kibble-scanner.py", line 180, in <module>
    main()
  File "src/kibble-scanner.py", line 136, in main
    for org in broker.organisations():
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 345, in organisations
    'match_all': {}
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 94, in search
    body = body
  File "/home/cos/.local/lib/python3.6/site-packages/elasticsearch/client/utils.py", line 92, in _wrapped
    return func(*args, params=params, headers=headers, **kwargs)
TypeError: search() got an unexpected keyword argument '_source_include'

Any ideas?

Humbedooh commented 4 years ago

For the first issue, I think you can safely remove bttf from the requirements.txt, things should work without it. I'll see if we can just remove that altogether soon.

For the second issue, could you please show me the output of pip3 show elasticsearch ? It sounds like the ElasticSearch pip module might be an incompatible version. If you see a >= 7.0 version here, please try: pip3 install "elasticsearch<7" which will downgrade your elasticsearch client to an older version.

I'll look at whether we can dispense with the source_include altogether as well...

ghost commented 4 years ago

It turns out I was running with elasticsearch module version 7.6.0 . After downgrading it to version 6.8.1 I'm getting Invalid or missing API/ABI version in database kibble!

cos@ubuntu:~/temp/kibble-scanners$ curl -X GET "localhost:9200/?pretty"
{
  "name" : "ubuntu",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "6DoJlyXeQdqaGFQRMib71g",
  "version" : {
    "number" : "7.6.2",
    "build_flavor" : "oss",
    "build_type" : "deb",
    "build_hash" : "ef48eb35cf30adf4db14086e8aabd07ef6fb113f",
    "build_date" : "2020-03-26T06:34:37.794943Z",
    "build_snapshot" : false,
    "lucene_version" : "8.4.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}
cos@ubuntu:/var/www/kibble/setup$ sudo python3 setup.py
...
...
...
Okay, I got all I need, setting up Kibble...
Creating index kibble_api
Creating index kibble_ci_build
Creating index kibble_ci_queue
Creating index kibble_code_commit
Creating index kibble_code_commit_unique
Creating index kibble_code_modification
Creating index kibble_evolution
Creating index kibble_file_history
Creating index kibble_forum_post
Creating index kibble_forum_topic
Creating index kibble_ghstats
Creating index kibble_im_stats
Creating index kibble_im_ops
Creating index kibble_im_msg
Creating index kibble_issue
Creating index kibble_logstats
Creating index kibble_email
Creating index kibble_mailstats
Creating index kibble_mailtop
Creating index kibble_organisation
Creating index kibble_view
Creating index kibble_publish
Creating index kibble_source
Creating index kibble_stats
Creating index kibble_social_follow
Creating index kibble_social_followers
Creating index kibble_social_follower
Creating index kibble_social_person
Creating index kibble_uisession
Creating index kibble_useraccount
Creating index kibble_message
Creating index kibble_person
Indices created! {'acknowledged': True, 'shards_acknowledged': True, 'index': 'kibble_person'} 
Creating administrator account
Account created!
../api/yaml/kibble.yaml already exists! Writing to ../api/yaml/kibble.yaml.tmp instead
Writing Kibble config (../api/yaml/kibble.yaml.tmp)
All done, Kibble should...work now :)
cos@ubuntu:/var/www/kibble$ curl http://localhost:9200/_cat/indices
yellow open kibble_social_follower    6-dHZ1JhRVmJNBn8gkRVxw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_file_history       KTRE2H7dTS-YeZiQh9v3Lw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_source             THZYCPoDRCuIpN5uYsOjbA 5 1 1 0  5.9kb  5.9kb
yellow open kibble_view               y7u56OaCQImYuMIiCo6jBQ 5 1 0 0  1.3kb  1.3kb
yellow open kibble_social_followers   3mqxh2PeQvOugMNmnY2cAA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_mailtop            R0dLPudlQfar_yX8B758tg 5 1 0 0  1.3kb  1.3kb
yellow open kibble_im_msg             orKAoiVUTIGp-LRHuB9ppA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_mailstats          9qasGXX3QSmTTHDNSxbxDQ 5 1 0 0  1.3kb  1.3kb
yellow open kibble_evolution          9tF5HSy9QRiIQcj3VKvx5Q 5 1 0 0  1.3kb  1.3kb
yellow open kibble_code_modification  sH5BBhOZSMO5GHohVn-_gw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_logstats           KWre3KjYQMey0k8lwRurYg 5 1 0 0  1.3kb  1.3kb
yellow open kibble_api                rkubLmLKQFaAszjxMpIj_g 5 1 1 0  4.8kb  4.8kb
yellow open kibble_code_commit        P4WYrMzYQKCY-kOq3bn4IQ 5 1 0 0  1.3kb  1.3kb
yellow open kibble_im_stats           QLu91uZMSRmMbe_YvlMwXg 5 1 0 0  1.3kb  1.3kb
yellow open kibble_ci_build           2DD2ul39R0uEeyX6Gt7AHQ 5 1 0 0  1.3kb  1.3kb
yellow open kibble_forum_topic        -7bMi_PqRvGOsCumPj-ERw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_ghstats            W6Xp09CgTFualUkQShRbGA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_useraccount        EwbozIvrTrWST1oODz0yvA 5 1 1 0  7.6kb  7.6kb
yellow open kibble_ci_queue           _QDyVycES4iV06YiRDfVdQ 5 1 0 0  1.3kb  1.3kb
yellow open kibble_social_follow      v36Z9dBbQPqoQj2w_E2z4A 5 1 0 0  1.3kb  1.3kb
yellow open kibble_email              Xlf_A-JDRViul8xvcHxLug 5 1 0 0  1.3kb  1.3kb
yellow open kibble_publish            JHVAlU6cSLa9sjBH2AQ09A 5 1 0 0  1.3kb  1.3kb
yellow open kibble_message            pZXzm3k1QvqGPW0uAvCJLA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_organisation       Kt7tqlvSRi-dRXfP0kDbhg 5 1 1 0  5.6kb  5.6kb
yellow open kibble_person             yfuI2fGaRea7zmcAJpndoA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_stats              djtOMcM0QW6AAiXvpofHXA 5 1 0 0  1.3kb  1.3kb
yellow open kibble_forum_post         Cbt5WkALQ86aRN5WsAtZMw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_im_ops             37XncVh8QyiYPN8Tk9i-Bg 5 1 0 0  1.3kb  1.3kb
yellow open kibble_issue              ou5-56zeSg-tG8pOBeRolw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_social_person      XCrdhLaRR86miG7p9iBzMw 5 1 0 0  1.3kb  1.3kb
yellow open kibble_uisession          Bk6zyBIuR9C8QTUotolDQw 5 1 4 0 16.9kb 16.9kb
yellow open kibble_code_commit_unique TQJGIEp6T0mlqI4N91CBVA 5 1 0 0  1.3kb  1.3kb

And then:

cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Invalid or missing API/ABI version in database kibble! Please ensure the database has been primed by setup.py
Humbedooh commented 4 years ago

What is the output of curl http://localhost:9200/kibble_api/_doc/current?pretty ?

Humbedooh commented 4 years ago

the /_doc bit may be optional depending on the version of ES, not sure

ghost commented 4 years ago 
cos@ubuntu:/var/www/kibble/setup$ curl http://localhost:9200/kibble_api/_doc/current?pretty
{
  "_index" : "kibble_api",
  "_type" : "_doc",
  "_id" : "current",
  "_version" : 1,
  "_seq_no" : 0,
  "_primary_term" : 1,
  "found" : true,
  "_source" : {
    "apiversion" : "0.1.0",
    "dbversion" : 2
  }
}
Humbedooh commented 4 years ago

Could you try pulling in the last change in src/plugins/kibbleES.py and see what error it spits out when you run now?

Humbedooh commented 4 years ago

FWIW, the DB is the correct version, but there seems to be an issue with fetching the db document for tests

ghost commented 4 years ago

Here's what I get now:

cos@ubuntu:~/temp/kibble-scanners$ python3 src/kibble-scanner.py
[core]: Loaded plugins/scanners/git-sync v/0.1.2 (Sync plugin for Git repositories)
[core]: Loaded plugins/scanners/git-census v/0.1.0 (Census Scanner for Git)
[core]: Loaded plugins/scanners/git-sloc v/0.1.0 (SloC Counter for Git)
[core]: Loaded plugins/scanners/git-evolution v/0.1.0 (Git Evolution Scanner)
[core]: Loaded plugins/scanners/jira v/0.1.0 (Scanner for Atlassian JIRA)
[core]: Loaded plugins/scanners/ponymail v/0.1.0 (Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-tone v/0.1.0 (Tone/Mood Scanner plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/ponymail-kpe v/0.1.0 (Key Phrase Extraction plugin for Apache Pony Mail)
[core]: Loaded plugins/scanners/pipermail v/0.1.0 (Scanner for GNU Mailman Pipermail)
[core]: Loaded plugins/scanners/github-issues v/0.1.0 (Scanner for GitHub Issues)
[core]: Loaded plugins/scanners/bugzilla v/0.1.0 (Scanner for BugZilla)
[core]: Loaded plugins/scanners/gerrit v/0.1.1 (Scanner for Gerrit Code Review)
[core]: Loaded plugins/scanners/jenkins v/0.1.0 (Scanner for Jenkins CI)
[core]: Loaded plugins/scanners/buildbot v/0.1.0 (Scanner for Buildbot)
[core]: Loaded plugins/scanners/travis v/0.1.0 (Scanner for Travis CI)
[core]: Loaded plugins/scanners/discourse v/0.1.0 (Scanner for Discourse Forums)
[core]: Kibble Scanner v/0.1.0 starting
src/kibble-scanner.py:122: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  config = yaml.load(open(CONFIG_FILE))
[core]: Loaded YAML config from conf/config.yaml
[core]: Using direct ElasticSearch broker model
[core]: Connecting to ElasticSearch database at localhost:9200...
[core]: Connected!
[core]: This is a type-less DB, expanding database names instead.
[core]: We're using ES >= 7.x, NO DOC_TYPE!
Traceback (most recent call last):
  File "src/kibble-scanner.py", line 180, in <module>
    main()
  File "src/kibble-scanner.py", line 129, in main
    broker = plugins.brokers.kibbleES.Broker(config)
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 318, in __init__
    apidoc = es.get(index=es_config['database'], doc_type='api', id = 'current')['_source']
  File "/home/cos/temp/kibble-scanners/src/plugins/brokers/kibbleES.py", line 79, in get
    return self.ES.get(index = index+'_'+doc_type, id = id)
  File "/home/cos/.local/lib/python3.8/site-packages/elasticsearch/client/utils.py", line 84, in _wrapped
    return func(*args, params=params, **kwargs)
TypeError: get() missing 1 required positional argument: 'doc_type'
ghost commented 4 years ago

[core]: We're using ES >= 7.x, NO DOC_TYPE!

TypeError: get() missing 1 required positional argument: 'doc_type'

Humbedooh commented 4 years ago

Okay, so clearly an issue with the elasticsearch library then. Please try upgrading it to >7: pip3 install "elasticsearch>7<8"

Humbedooh commented 4 years ago

ES client 7.x should work now with yesterday's tweaks..

ghost commented 4 years ago

Everything seems peachy. Thank you @Humbedooh . :)