search

yeti-platform / yeti

Your Everyday Threat Intelligence
https://yeti-platform.io/
Apache License 2.0
1.72k stars 288 forks source link

Custom Feed is not working as expected #344

Closed s-shubh closed 5 years ago

s-shubh commented 5 years ago

Environment

Question Answer
OS version CentOS7
Browser Chrome 56.0.2924.87

Hi Team,

I am having custom feed imput which is stored in my web server and I generally query that document for the feed input. Find the input file example ;

CNC 5.6.7.8 Tornode 5.7.8.9 BruteForcer 6.7.8.9 & so on....

To process this type of data; I have created custom feed in Yeti and put in under plugins/feeds/public/ directory. Kindly find this feed.

import requests
from datetime import timedelta
import csv
import logging
from core.observables import Url, Ip
from core.feed import Feed
from core.errors import ObservableValidationError
from core.config.config import yeti_config
class ET_Blocklist(Feed):
        default_values = {
                "frequency": timedelta(hours=1),
                "name": "ET_Blocklist",
                "source": "http://172.16.0.28/fd",
                "description": "This feed contains custom ET ourput",
        }
        def update(self):
                for line in self.update_lines():
                        print(line)
                self.analyze(line)
        def analyze(self, line):
                if not line or line[0].startswith("#"):
                        return
                fields = line.split()
                context = {}
                ip = fields[1]
                context['ThreatCategory'] = fields[0]
                context['description'] = "Emerging Threat IP List: %s (%s)" % (
                context['name'], ip)
                context['source'] = self.name
                try:
                        ip = Ip.get_or_create(value=fields[1])
                        ip.add_context(context)
                        ip.add_source("feed")
                        ip.tag(['ET'])
                except ObservableValidationError as e:
                        logging.error(e)

Output(Custom Feed Name ET_Bocklist) is shown on Yeti Dataflows interface as shown in the below screenshot but while searching for an IP address such as 5.6.7.8 then it shows null output. find both the screenshots.

ip is now get shown

dataflows shown

Please help...!!

~Shubham

s-shubh commented 5 years ago

Due to Format error, feed is not shown in python indent. So please consider that format error.

tomchop commented 5 years ago

That's weird. Could you provide a sample of you feed's format?

Can you check the logs' output? There might be some useful information there.

Due to Format error, feed is not shown in python indent. So please consider that format error.

For the record, you can use backticks (``) to format code blocks on GitHub (click onedit` to see the changes I've made to your original post.

s-shubh commented 5 years ago

Thank you for your information. Which logs do you want from me ? standard linux or mongodb ?

tomchop commented 5 years ago

Try this journalctl -u yeti_feeds.service and systemctl status yeti_feeds.service

s-shubh commented 5 years ago

logs.zip

Find the attached output of the command "journalctl -u yeti_feeds.service"

--> Find the status of the command; "systemctl status yeti_feeds.service"

● yeti_feeds.service - Yeti workers - Feeds Loaded: loaded (/usr/lib/systemd/system/yeti_feeds.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-02-20 06:44:42 EST; 1 weeks 1 days ago Main PID: 25846 (celery) CGroup: /system.slice/yeti_feeds.service ├─25846 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25853 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25858 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25861 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25866 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25872 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25875 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25880 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge └─25885 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge

Feb 28 10:16:35 threatintel1 celery[25846]: [2019-02-28 10:16:35,053: WARNING/ForkPoolWorker-7] meguia.net,191.5.59.60,ns1.hospedaria.com.br|ns2.hospedaria.com.br|ns3.hospedaria.com.br|server1.hospedaria.com.br|server2.hospedaria.com.br,191.5.59.11|191.5.59.12|191.5.59.18|191.5.59.8|191.5.59.9,Master Indicator Feed for pykspa non-sinkholed domains,http://osint.bambenekconsulting.com/manual/pykspa.txt Feb 28 10:16:44 threatintel1 celery[25846]: [2019-02-28 10:16:44,616: WARNING/ForkPoolWorker-7] oagirkdsholapet.com,198.54.117.197|198.54.117.198|198.54.117.199|198.54.117.200,dns101.registrar-servers.com|dns102.registrar-servers.com,198.54.117.253|198.54.117.254,Master Indicator Feed for pykspa non-sinkholed domains,http://osint.bambenekconsulting.com/manual/pykspa.txt Feb 28 10:16:51 threatintel1 celery[25846]: [2019-02-28 10:16:51,529: WARNING/ForkPoolWorker-7] tmwkliqsuku.info,173.231.184.56,ns1.tmwkliqsuku.info|ns2.tmwkliqsuku.info|ns3.tmwkliqsuku.info|ns4.tmwkliqsuku.info,34.229.84.179|34.230.76.81|54.227.204.233|184.73.137.229,Master Indicator Feed for pykspa non-sinkholed domains,http://osint.bambenekconsulting.com/manual/pykspa.txt Feb 28 10:16:59 threatintel1 celery[25846]: [2019-02-28 10:16:59,371: WARNING/ForkPoolWorker-7] ceigqweqwaywiqgu.org,206.189.61.126,ns1.dynadot.com|ns2.dynadot.com,52.26.28.15|52.34.35.173|52.35.76.183|52.36.53.176|52.71.195.14|52.72.130.79|52.72.200.98|52.73.101.236,Master Indicator Feed for ramdo non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramdo.txt Feb 28 10:17:09 threatintel1 celery[25846]: [2019-02-28 10:17:09,181: WARNING/ForkPoolWorker-7] eiscawcsgiaemsco.org,35.170.58.11,ns-1454.awsdns-53.org|ns-1988.awsdns-56.co.uk|ns-273.awsdns-34.com|ns-846.awsdns-41.net,205.251.193.17|205.251.195.78|205.251.197.174|205.251.199.196,Master Indicator Feed for ramdo non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramdo.txt Feb 28 10:17:17 threatintel1 celery[25846]: [2019-02-28 10:17:17,131: WARNING/ForkPoolWorker-7] aafgcvjyvxlosy.com,46.165.254.199,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt Feb 28 10:17:21 threatintel1 celery[25846]: [2019-02-28 10:17:21,432: WARNING/ForkPoolWorker-7] aarbvsrdnhhidhwk.com,46.165.254.214,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt Feb 28 10:17:25 threatintel1 celery[25846]: [2019-02-28 10:17:25,555: WARNING/ForkPoolWorker-7] absqvhpldvsmclt.com,46.165.220.150,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt Feb 28 10:17:29 threatintel1 celery[25846]: [2019-02-28 10:17:29,545: WARNING/ForkPoolWorker-7] advjpbbhiwoccqa.com,46.165.220.147,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt Feb 28 10:17:33 threatintel1 celery[25846]: [2019-02-28 10:17:33,630: WARNING/ForkPoolWorker-7] aeblrkyvqmk.com,185.159.129.140,dns1.regway.com|dns2.regway.com|dns3.regway.com|dns4.regway.com,162.251.82.118|162.251.82.119|162.251.82.120|162.251.82.121|162.251.82.122|162.251.82.123|162.251.82.124|162.251.82.125|162.251.82.246|162.251.82.247|162.251.82.248|162.251.82.249|162.251.82.250|162.251.82.251|162.251.82.252|162.251.82.253,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt

s-shubh commented 5 years ago

Also find latest output of yeti_feeds.service

● yeti_feeds.service - Yeti workers - Feeds Loaded: loaded (/usr/lib/systemd/system/yeti_feeds.service; enabled; vendor preset: disabled) Active: active (running) since Wed 2019-02-20 06:44:42 EST; 1 weeks 1 days ago Main PID: 25846 (celery) CGroup: /system.slice/yeti_feeds.service ├─25846 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25853 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25858 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25861 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25866 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25872 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25875 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge ├─25880 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge └─25885 /usr/bin/python2 /usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge

Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] BruteForcer 6.70.8.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] Tornode 5.7.8.90 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] BruteForcer 6.7.80.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] Tornode 5.70.8.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] BruteForcer 16.7.8.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] Tornode 15.7.8.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] BruteForcer 16.7.8.9 Feb 28 10:19:00 threatintel1 celery[25846]: [2019-02-28 10:19:00,470: WARNING/ForkPoolWorker-2] Tor 4.5.6.7 Feb 28 10:19:06 threatintel1 celery[25846]: [2019-02-28 10:19:06,818: WARNING/ForkPoolWorker-7] aujastmvehxqmlbb.com,217.20.116.140,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt Feb 28 10:19:11 threatintel1 celery[25846]: [2019-02-28 10:19:11,121: WARNING/ForkPoolWorker-7] avpsjncogss.com,46.165.221.144,dns1.registrar-servers.com|dns2.registrar-servers.com,216.87.152.33|216.87.155.33,Master Indicator Feed for ramnit non-sinkholed domains,http://osint.bambenekconsulting.com/manual/ramnit.txt

tomchop commented 5 years ago

Can you share the format of the feed you're trying to ingest? There's most likely a problem with that.

s-shubh commented 5 years ago

I have already given you a format in my query.

Format is; "Threat Category" "IP Address"

Example Feed is; CNC 5.6.7.8 Tornode 5.7.8.9 BruteForcer 6.7.8.9 & so on....

tomchop commented 5 years ago

OK. At this point I would recommend this:

And provide the output of the celery worker

s-shubh commented 5 years ago

Hey, Thank you for your prompt suggestions. I have gone through steps those you recommended. But while starting celery worker , I got the below output with error.

/usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge

Traceback (most recent call last): File "/usr/bin/celery", line 11, in sys.exit(main()) File "/usr/lib/python2.7/site-packages/celery/main.py", line 16, in main _main() File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 322, in main cmd.execute_from_commandline(argv) File "/usr/lib/python2.7/site-packages/celery/bin/celery.py", line 496, in execute_from_commandline super(CeleryCommand, self).execute_from_commandline(argv))) File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 273, in execute_from_commandline argv = self.setup_app_from_commandline(argv) File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 479, in setup_app_from_commandline self.app = self.find_app(app) File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 501, in find_app return find_app(app, symbol_by_name=self.symbol_by_name) File "/usr/lib/python2.7/site-packages/celery/app/utils.py", line 359, in find_app sym = symbol_by_name(app, imp=imp) File "/usr/lib/python2.7/site-packages/celery/bin/base.py", line 504, in symbol_by_name return imports.symbol_by_name(name, imp=imp) File "/usr/lib/python2.7/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name module = imp(module_name, package=package, **kwargs) File "/usr/lib/python2.7/site-packages/celery/utils/imports.py", line 104, in import_from_cwd return imp(module, package=package) File "/usr/lib64/python2.7/importlib/init.py", line 37, in import_module import(name) ImportError: No module named core.config.celeryctl

tomchop commented 5 years ago

You need to be in the /opt/yeti directory for the command to work

s-shubh commented 5 years ago

Thank you for this help...Please find the output...

/usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge /usr/lib/python2.7/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

uid=uid, euid=euid, gid=gid, egid=egid,

-------------- celery@feeds v4.2.1 (windowlicker) ---- * ----- --- * -- Linux-3.10.0-862.9.1.el7.x86_64-x86_64-with-centos-7.5.1804-Core 2019-02-28 11:49:31 -- - **** ---

[2019-02-28 11:49:42,338: WARNING/ForkPoolWorker-7] CNC 5.6.7.8 [2019-02-28 11:49:42,338: WARNING/ForkPoolWorker-7] Tornode 5.7.8.9 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] BruteForcer 6.7.8.9 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] CNC 5.6.7.8 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] CNC 5.6.7.9 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] CNC 5.6.7.10 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] CNC 5.6.7.11 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] CNC 5.6.7.12 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] Tornode 5.7.8.10 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] BruteForcer 6.7.8.10 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] Tornode 5.7.80.9 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] BruteForcer 6.70.8.9 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] Tornode 5.7.8.90 [2019-02-28 11:49:42,339: WARNING/ForkPoolWorker-7] BruteForcer 6.7.80.9 [2019-02-28 11:49:42,340: WARNING/ForkPoolWorker-7] Tornode 5.70.8.9 [2019-02-28 11:49:42,340: WARNING/ForkPoolWorker-7] BruteForcer 16.7.8.9 [2019-02-28 11:49:42,340: WARNING/ForkPoolWorker-7] Tornode 15.7.8.9 [2019-02-28 11:49:42,340: WARNING/ForkPoolWorker-7] BruteForcer 16.7.8.9 [2019-02-28 11:49:42,340: WARNING/ForkPoolWorker-7] Tor 4.5.6.7

tomchop commented 5 years ago

Hm looks like your split isn't working as intended. Can you print context and print fields before the try / except block and relaunch the celery workers?

s-shubh commented 5 years ago

I have added print context and 'print fields' before try and run the celery workers but got the same output.

tomchop commented 5 years ago

So what's happening is that an error in your code is likely preventing the observables from being stored. Try removing the try / except statements and see if that changes anything?

s-shubh commented 5 years ago

Do you want me to remove full block of try/except statements ? I have removed all their block but still could not get printed observable.

tomchop commented 5 years ago

something like this

                context['ThreatCategory'] = fields[0]
                context['description'] = "Emerging Threat IP List: %s (%s)" % (context['name'], ip)
                context['source'] = self.name
                print context
                print fields
                ip = Ip.get_or_create(value=fields[1])
                ip.add_context(context)
                ip.add_source("feed")
                ip.tag(['ET'])
tomchop commented 5 years ago

Also make sure to remove any generated pyc files in your workspace.

$ find . -name "*.pyc" -exec rm {} \;

s-shubh commented 5 years ago

Thank you for your continuous help... I have tried all these but it still shows the same output while initiating workers.

s-shubh commented 5 years ago

Can we make any other changes to get this worked ?

tomchop commented 5 years ago

Ugh, I think I found your bug. There's an indentation problem in the update function:

        def update(self):
                for line in self.update_lines():
                        print(line)
                self.analyze(line)

You should call analyze for every line, and in this case you're calling it from outside the loop.

s-shubh commented 5 years ago

So we need to run 'update()' function like this: 

def update(self):
          for line in self.update_lines():
                 print(line)
                 self.analyze(line)

Am I correct ?

tomchop commented 5 years ago

Yes. You can also remove the print statement

s-shubh commented 5 years ago

I got this working but found some error related to 'name'.

/usr/bin/celery -A core.config.celeryctl.celery_app worker -Ofair -c 8 -Q feeds -n feeds --purge

/usr/lib/python2.7/site-packages/celery/platforms.py:796: RuntimeWarning: You're running the worker with superuser privileges: this is
absolutely not recommended!

Please specify a different user using the --uid option.

User information: uid=0 euid=0 gid=0 egid=0

  uid=uid, euid=euid, gid=gid, egid=egid,

 -------------- celery@feeds v4.2.1 (windowlicker)
---- **** -----
--- * ***  * -- Linux-3.10.0-862.9.1.el7.x86_64-x86_64-with-centos-7.5.1804-Core 2019-02-28 12:37:53
-- * - **** ---
- ** ---------- [config]
- ** ---------- .> app:         yeti:0x7fc8c37d7e90
- ** ---------- .> transport:   redis://127.0.0.1:6379/0
- ** ---------- .> results:     disabled://
- *** --- * --- .> concurrency: 8 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** -----
 -------------- [queues]
                .> feeds            exchange=feeds(direct) key=feeds

[2019-02-28 12:38:00,081: WARNING/ForkPoolWorker-7] CNC 5.6.7.8
[2019-02-28 12:38:00,081: ERROR/ForkPoolWorker-7] Traceback (most recent call last):
  File "/opt/yeti/core/feed.py", line 42, in update_feed
    f.update()
  File "/opt/yeti/plugins/feeds/public/ET_blocklist.py", line 20, in update
    self.analyze(line)
  File "/opt/yeti/plugins/feeds/public/ET_blocklist.py", line 30, in analyze
    context['description'] = "Emerging Threat IP List: %s (%s)" % (context['name'], ip)
KeyError: 'name'

[2019-02-28 12:38:00,081: ERROR/ForkPoolWorker-7] ERROR updating feed: 'name'
tomchop commented 5 years ago

Looks like context['name'] is not set.

I'm closing this issue out, make sure you've ironed out remaining bugs and if things are still not working feel free to reopen.

s-shubh commented 5 years ago

Thank you for the continuous help... Just little guidance. I just changed context name to context description but still got the same output. Let me know if you can help in this. 

    ```
    fields = line.split()
            context = {}
            ip = fields[1]
            context['ThreatCategory'] = fields[0]
            context['description'] = "Emerging Threat IP List: %s (%s)" % (context['ThreatCategory'], ip)
            context['source'] = self.name
            ip = Ip.get_or_create(value=fields[1])
            ip.add_context(context)
            ip.add_source("feed")
            ip.tag(['ET'])


Got the same output.
s-shubh commented 5 years ago

Thank you for your help... I appreciate constant output. It's get worked...