orlikoski / Skadi

Collect, Process, and Hunt with host based data from MacOS, Windows, and Linux
https://www.skadivm.com
GNU General Public License v3.0
489 stars 71 forks source link

Using docker-compose, yeti-beat keep on exiting #114

Closed mrezqi closed 5 years ago

mrezqi commented 5 years ago

Hi, I tried using docker-compose to get everything up and running.

I keep on running into this error: docker_yeti-beat_1 exited with code 1

Detailed logs below:

yeti-beat_1       | celery beat v4.2.1 (windowlicker) is starting.
yeti-beat_1       | Traceback (most recent call last):
yeti-beat_1       |   File "/usr/local/bin/celery", line 10, in <module>
yeti-beat_1       |     sys.exit(main())
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/__main__.py", line 16, in main
yeti-beat_1       |     _main()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 322, in main
yeti-beat_1       |     cmd.execute_from_commandline(argv)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 496, in execute_from_commandline
yeti-beat_1       |     super(CeleryCommand, self).execute_from_commandline(argv)))
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 275, in execute_from_commandline
yeti-beat_1       |     return self.handle_argv(self.prog_name, argv[1:])
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 488, in handle_argv
yeti-beat_1       |     return self.execute(command, argv)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/celery.py", line 420, in execute
yeti-beat_1       |     ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 279, in run_from_argv
yeti-beat_1       |     sys.argv if argv is None else argv, command)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 363, in handle_argv
yeti-beat_1       |     return self(*args, **options)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/base.py", line 238, in __call__
yeti-beat_1       |     ret = self.run(*args, **kwargs)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/bin/beat.py", line 109, in run
yeti-beat_1       |     return beat().run()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 81, in run
yeti-beat_1       |     self.start_scheduler()
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 100, in start_scheduler
yeti-beat_1       |     print(self.banner(service))
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 122, in banner
yeti-beat_1       |     c.reset(self.startup_info(service))),
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/apps/beat.py", line 132, in startup_info
yeti-beat_1       |     scheduler = service.get_scheduler(lazy=True)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/celery/beat.py", line 618, in get_scheduler
yeti-beat_1       |     lazy=lazy,
yeti-beat_1       |   File "/opt/yeti/core/scheduling.py", line 61, in __init__
yeti-beat_1       |     self.load_entries()
yeti-beat_1       |   File "/opt/yeti/core/scheduling.py", line 79, in load_entries
yeti-beat_1       |     self.loaded_entries = get_plugins()
yeti-beat_1       |   File "/opt/yeti/core/yeti_plugins.py", line 21, in get_plugins
yeti-beat_1       |     entry = obj.get_or_create(name=obj.default_values['name'])
yeti-beat_1       |   File "/opt/yeti/core/database.py", line 90, in get_or_create
yeti-beat_1       |     r = cls.objects(**select_dict).modify(upsert=True, **update_dict)
yeti-beat_1       |   File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 648, in modify
yeti-beat_1       |     raise OperationError(u'Update failed (%s)' % err)
yeti-beat_1       | mongoengine.errors.OperationError: Update failed (BSON field 'no_cursor_timeout' is an unknown field.)
mongodb           | 2019-07-09T13:14:44.533+0000 I  NETWORK  [conn59] end connection 172.25.0.14:54382 (17 connections now open)
mongodb           | 2019-07-09T13:14:44.534+0000 I  NETWORK  [conn60] end connection 172.25.0.14:54384 (16 connections now open)
docker_yeti-beat_1 exited with code 1
orlikoski commented 5 years ago

The issue, according to the log snippet, is that Yeti-beat is having an issue inside mongodb.

In order to fix that I recommend removing the mongodb and yeti-beat docker containers and volumes and try again. Here are some commands to do that. Please let me know what the results are. Also, what was the Skadi installation method (signed installer, OVA, Vagrant?)

Commands

cd /opt/Skadi/Docker
docker-compose down
docker system prune
docker volume prune
docker-compose up -d

If that doesn't work try changing the version of mongodb in the /opt/Skadi/Docker/docker-compose.yml file to the following

mongodb:
    image: mongo:4.0

then type

cd /opt/Skadi/Docker
docker-compose down
docker system prune
docker volume prune
docker-compose up -d
epicsilence99 commented 5 years ago

were you able to look at the steps @orlikoski outlined above and see if that resolved your issue @mrezqi ?

orlikoski commented 5 years ago

This has been fixed in Skadi 2019.4