search

yeti-platform / yeti

Your Everyday Threat Intelligence
https://yeti-platform.io/
Apache License 2.0
1.74k stars 287 forks source link

Exception: ScheduleEntry.Feed matching query does not exist. #263

Closed fr33tux closed 6 years ago

fr33tux commented 6 years ago

Description

Feeds frozen and Exception: ScheduleEntry.Feed matching query does not exist. spamming in the logfile.

Environment

Question Answer
Git commit b24bf3de5ffd15b23255c759b3a7e1264223d1fe
OS version Ubuntu 16.04
Browser Firefox stable (60.0.2)
MongoDB v3.6.5

Steps to Reproduce

Unsure precisely. After the Yeti instance has been running for several days, the feeds get freezed while updating. Restarting the yeti_feeds service doesn't help, and journalctl -u yeti_feeds.service outputs the exception below.

Expected behavior

The feeds complete their update with success, and no exception thrown on the logs.

Actual behavior

Jun 10 17:34:03 ip-10-132-0-175 celery[9618]: [2018-06-10 17:34:03,699: ERROR/ForkPoolWorker-10] Task core.feed.update_feed[3a88b184-b02b-41b8-8602-5d34c26f8fd4] raised unexpected: DoesNotExist('ScheduleEntry.Fe
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]: Traceback (most recent call last):
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 374, in trace_task
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:     R = retval = fun(*args, **kwargs)
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:   File "/usr/local/lib/python2.7/dist-packages/celery/app/trace.py", line 629, in __protected_call__
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:     return self.run(*args, **kwargs)
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:   File "/data/yeti/core/feed.py", line 35, in update_feed
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:     Feed.objects.get(id=feed_id).name))
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:   File "/usr/local/lib/python2.7/dist-packages/mongoengine/queryset/base.py", line 276, in get
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]:     raise queryset._document.DoesNotExist(msg)
Jun 10 17:34:03 ip-10-132-0-175 celery[9618]: Exception: ScheduleEntry.Feed matching query does not exist.
tomchop commented 6 years ago

Is this happening on a new install? or did you upgrade with a git pull?

fr33tux commented 6 years ago

It's happening on a new install. Not immediately, but after a few days (no git pull between the good state and the issue).

tomchop commented 6 years ago

This is strange. Can you send me a dump of your schedule_entry mongo collection?

fr33tux commented 6 years ago

It's fixed by manually unlocking the feeds and relaunch the worker (restarting the worker alone didn't help):

db.schedule_entry.update({lock: true}, {$set :{lock:false}}, {multi:true})

I'll share the collection when it happens again.

tomchop commented 6 years ago

Ah, thanks! We've encountered this problem before, and we have code that's supposed to fix it. Please let me know if you see this again. We're planning to make the scheduling more reliable in v2 :)

s-shubh commented 5 years ago

I got the same error. Feeds are not synced even tested. Please help with configs.

sebdraven commented 5 years ago

Before launching feeds, you have to launch the webserver and log to the web UI and in a second way you launch feeds

s-shubh commented 5 years ago

I can initiate feeds but in the logs I got the same error.

s-shubh commented 5 years ago

even my database file located under ..yeti/storage/ has also been shown updated months ago.

s-shubh commented 5 years ago

[root@ssmachine1 yeti]# python testfeeds.py plugins/feeds/public/dynamic_dns.py DEBUG:root:Scheduler started Traceback (most recent call last): File "testfeeds.py", line 19, in f = Feed.objects.get(name=name) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 276, in get raise queryset._document.DoesNotExist(msg) core.feed.DoesNotExist: ScheduleEntry.Feed matching query does not exist.

Got the same error even though fixing mongodb unlocking manually... I want to check custom script if the current default script gets found working properly...

Please help...!

tomchop commented 5 years ago

Can you please run db.schedule_entry.find() in a mongo shell and past the output here?

s-shubh commented 5 years ago

Find the output over here....

{ "_id" : ObjectId("5c6d1baf9e4bbc58dbf3686a"), "_cls" : "ScheduleEntry.Export", "name" : "TestExport", "enabled" : true, "description" : "Test description", "frequency" : 3600, "lock" : false, "include_tags" : [ ObjectId("5c6d1bafbbb230e2db4528f1"), ObjectId("5c6d1bafbbb230e2db4528f3") ], "exclude_tags" : [ ], "ignore_tags" : [ ], "output_dir" : "exports", "acts_on" : "Url", "template" : ObjectId("5c6d1baf9e4bbc58dbf36869"), "status" : "OK", "hash_md5" : "a8fa01a481aa0bc647b08e756572bfc0", "last_run" : ISODate("2019-02-20T15:44:38.103Z") } { "_id" : ObjectId("5c6d1dd4bbb230e2db452912"), "_cls" : "ScheduleEntry.ScheduledAnalytics.ExpireTags", "name" : "ExpireTags", "enabled" : true, "lock" : false, "frequency" : 43200, "description" : "Expires tags in observables" } { "_id" : ObjectId("5c6d1dd5bbb230e2db452922"), "_cls" : "ScheduleEntry.ScheduledAnalytics.TagLogic", "name" : "TagLogic", "enabled" : true, "lock" : false, "frequency" : 1800, "description" : "Processes some tagging logic", "status" : "OK", "last_run" : ISODate("2019-02-20T16:14:40.276Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db452927"), "_cls" : "ScheduleEntry.Feed.AsproxTracker", "name" : "AsproxTracker", "enabled" : false, "lock" : false, "source" : "http://atrack.h3x.eu/api/asprox_full_csv.php", "frequency" : 3600, "description" : "This feed contains known Asprox C2 servers", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:38.059Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db452928"), "_cls" : "ScheduleEntry.Feed.BambenekOsintIpmaster", "name" : "BambenekOsintIpmaster", "enabled" : false, "lock" : false, "source" : "http://osint.bambenekconsulting.com/feeds/c2-masterlist.txt", "frequency" : 3600, "description" : "Master Feed of known, active and non-sinkholed C&Cs indicators (Bambenek)", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:38.085Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db452929"), "_cls" : "ScheduleEntry.Feed.CybercrimeAtmosTracker", "name" : "CybercrimeAtmosTracker", "enabled" : false, "lock" : false, "source" : "http://cybercrime-tracker.net/ccam_rss.php", "frequency" : 3600, "description" : "CyberCrime Atmos Tracker - Latest 20 Atmos binaries", "last_run" : ISODate("2019-02-20T15:44:37.923Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db45292a"), "_cls" : "ScheduleEntry.Feed.CybercrimePonyTracker", "name" : "CybercrimePonyTracker", "enabled" : false, "lock" : false, "source" : "http://cybercrime-tracker.net/ccpm_rss.php", "frequency" : 3600, "description" : "CyberCrime Pony Tracker - Latest 20 CnC URLS", "last_run" : ISODate("2019-02-20T15:44:38.211Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db45292c"), "_cls" : "ScheduleEntry.Feed.DynamicDomains", "name" : "DynamicDomains", "enabled" : true, "lock" : false, "source" : "http://mirror1.malwaredomains.com/files/dynamic_dns.txt", "frequency" : 86400, "description" : "Malwaredomains.com Dynamic Domains list", "status" : "OK", "last_run" : ISODate("2019-02-20T11:32:12.809Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db45292d"), "_cls" : "ScheduleEntry.Feed.FeodoTracker", "name" : "FeodoTracker", "enabled" : false, "lock" : false, "source" : "https://feodotracker.abuse.ch/feodotracker.rss", "frequency" : 3600, "description" : "Feodo Tracker RSS Feed. This feed shows the latest twenty Feodo C2 servers which Feodo Tracker has identified.", "status" : "ERROR updating feed: Space required after the Public Identifier, line 1, column 47", "last_run" : ISODate("2019-02-20T15:44:37.671Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db45292f"), "_cls" : "ScheduleEntry.Feed.HostsFileEXP", "name" : "HostsFileEXP", "enabled" : false, "lock" : false, "source" : "https://hosts-file.net/exp.txt", "frequency" : 14400, "description" : "Sites engaged in the housing, development or distribution of exploits, including but not limited to exploitation of browser, software (inclusive of website software such as CMS), operating system exploits aswell as those engaged in exploits via social engineering.", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.561Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452930"), "_cls" : "ScheduleEntry.Feed.HostsFileFSA", "name" : "HostsFileFSA", "enabled" : false, "lock" : false, "source" : "https://hosts-file.net/fsa.txt", "frequency" : 14400, "description" : "Sites engaged in the selling or distribution of bogus or fraudulent applications and/or provision of fraudulent services.", "status" : "Updating...", "last_run" : ISODate("2019-02-20T15:44:37.571Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452937"), "_cls" : "ScheduleEntry.Feed.MispFeed", "name" : "MispFeed", "enabled" : false, "lock" : false, "source" : "MISP", "frequency" : 3600, "description" : "Parses events from a given MISP instance", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.720Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db45293b"), "_cls" : "ScheduleEntry.Feed.TorExitNodes", "name" : "TorExitNodes", "enabled" : false, "lock" : false, "source" : "https://www.dan.me.uk/tornodes", "frequency" : 3600, "description" : "Tor exit nodes", "last_run" : ISODate("2019-02-20T15:44:37.931Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452934"), "_cls" : "ScheduleEntry.Feed.MalwareDomainList", "name" : "MalwareDomainList", "enabled" : false, "lock" : false, "source" : "http://www.malwaredomainlist.com/hostslist/mdl.xml", "frequency" : 3600, "description" : "MalwareDomainList update. This feed shows the latest urls which have been added to our list.", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.808Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452935"), "_cls" : "ScheduleEntry.Feed.MalwareDomainsDotCom", "name" : "MalwareDomainsDotCom", "enabled" : false, "lock" : false, "source" : "http://mirror1.malwaredomains.com/files/domains.txt", "frequency" : 7200, "description" : "Malwaredomains.com Blacklist", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.927Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452936"), "_cls" : "ScheduleEntry.Feed.MalwareConfig", "name" : "MalwareConfig", "enabled" : false, "lock" : false, "source" : "https://malwareconfig.com/static/C2_24.csv", "frequency" : 3600, "description" : "MalwareConfig offers list of submitted files along extracted c2 servers and malware family.", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.795Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452938"), "_cls" : "ScheduleEntry.Feed.OpenPhish", "name" : "OpenPhish", "enabled" : false, "lock" : false, "source" : "https://openphish.com/feed.txt", "frequency" : 14400, "description" : "OpenPhish community feed. Contains a list of possible Phishing URLs.", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.572Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452939"), "_cls" : "ScheduleEntry.Feed.PhishTank", "name" : "PhishTank", "enabled" : false, "lock" : false, "source" : "http://data.phishtank.com/data/online-valid.csv", "frequency" : 14400, "description" : "PhishTank community feed. Contains a list of possible Phishing URLs.", "last_run" : ISODate("2019-02-20T15:44:37.579Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db45293a"), "_cls" : "ScheduleEntry.Feed.RansomwareTracker", "name" : "RansomwareTracker", "enabled" : false, "lock" : false, "source" : "http://ransomwaretracker.abuse.ch/feeds/csv/", "frequency" : 1200, "description" : "Ransomware Tracker offers various types of blocklists that allows you to block Ransomware botnet C&C traffic.", "last_run" : ISODate("2019-02-20T16:24:39.537Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db45293c"), "_cls" : "ScheduleEntry.Feed.UrlHaus", "name" : "UrlHaus", "enabled" : false, "lock" : false, "source" : "https://urlhaus.abuse.ch/downloads/csv/", "frequency" : 1200, "description" : "URLhaus is a project from abuse.ch with the goal of sharing malicious URLs that are being used for malware distribution.", "last_run" : ISODate("2019-02-20T16:24:39.462Z") }

{ "_id" : ObjectId("5c6d1dd7bbb230e2db45293d"), "_cls" : "ScheduleEntry.Feed.VirusTotalHunting", "name" : "VirusTotalHunting", "enabled" : false, "lock" : false, "source" : "https://www.virustotal.com/intelligence/hunting/", "frequency" : 300, "description" : "Feed of hunting of VirusTotal", "last_run" : ISODate("2019-02-20T16:29:41.463Z") } { "_id" : ObjectId("5c6d1dd8bbb230e2db45293e"), "_cls" : "ScheduleEntry.Feed.VXVaultUrl", "name" : "VXVaultUrl", "enabled" : false, "lock" : false, "source" : "http://vxvault.net/URL_List.php", "frequency" : 3600, "description" : "VXVault Community URL list.", "last_run" : ISODate("2019-02-20T15:44:37.704Z") } { "_id" : ObjectId("5c6d1dd8bbb230e2db45293f"), "_cls" : "ScheduleEntry.Feed.ZeusTrackerBinaries", "name" : "ZeusTrackerBinaries", "enabled" : false, "lock" : false, "source" : "https://zeustracker.abuse.ch/monitor.php?urlfeed=binaries", "frequency" : 3600, "description" : "This feed shows the latest 50 ZeuS binaries URLs.", "last_run" : ISODate("2019-02-20T15:44:38.016Z") } { "_id" : ObjectId("5c6d1dd8bbb230e2db452940"), "_cls" : "ScheduleEntry.Feed.ZeusTrackerConfigs", "name" : "ZeusTrackerConfigs", "enabled" : false, "lock" : false, "source" : "https://zeustracker.abuse.ch/monitor.php?urlfeed=configs", "frequency" : 3600, "description" : "This feed shows the latest 50 ZeuS config URLs.", "last_run" : ISODate("2019-02-20T15:44:37.774Z") } { "_id" : ObjectId("5c6d1dd8bbb230e2db452941"), "_cls" : "ScheduleEntry.Feed.ZeusTrackerDropzones", "name" : "ZeusTrackerDropzones", "enabled" : false, "lock" : false, "source" : "https://zeustracker.abuse.ch/monitor.php?urlfeed=dropzones", "frequency" : 3600, "description" : "Shows the latest 50 ZeuS dropzone URLs.", "last_run" : ISODate("2019-02-20T15:44:37.958Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db452926"), "_cls" : "ScheduleEntry.Feed.ET_Blocklist", "name" : "ET_Blocklist", "enabled" : true, "lock" : false, "source" : "http://172.16.0.28/fd", "frequency" : 3600, "description" : "This feed contains custom ET ourput", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:38.093Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452933"), "_cls" : "ScheduleEntry.Feed.MalcodeBinaries", "name" : "MalcodeBinaries", "enabled" : false, "lock" : false, "source" : "http://malc0de.com/rss/", "frequency" : 3600, "description" : "Updated Feed of Malicious Executables", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.684Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db45292e"), "_cls" : "ScheduleEntry.Feed.HostsFileEMD", "name" : "HostsFileEMD", "enabled" : false, "lock" : false, "source" : "https://hosts-file.net/emd.txt", "frequency" : 14400, "description" : "Sites engaged in malware distribution.", "status" : "Updating...", "last_run" : ISODate("2019-02-20T15:44:37.561Z") } { "_id" : ObjectId("5c6d1dd6bbb230e2db45292b"), "_cls" : "ScheduleEntry.Feed.CybercrimeTracker", "name" : "CybercrimeTracker", "enabled" : false, "lock" : false, "source" : "http://cybercrime-tracker.net/rss.xml", "frequency" : 3600, "description" : "CyberCrime Tracker - Latest 20 CnC URLS", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.728Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452932"), "_cls" : "ScheduleEntry.Feed.HybridAnalysis", "name" : "HybridAnalysis", "enabled" : false, "lock" : false, "source" : "https://www.hybrid-analysis.com/feed?json", "frequency" : 300, "description" : "Hybrid Analysis Public Feeds", "last_run" : ISODate("2019-02-20T16:29:41.486Z") } { "_id" : ObjectId("5c6d1dd5bbb230e2db45291f"), "_cls" : "ScheduleEntry.ScheduledAnalytics.PropagateBlocklist", "name" : "PropagateBlocklist", "enabled" : true, "lock" : false, "frequency" : 3600, "description" : "Propagates blocklist from URLs to hostnames", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:37.765Z") } { "_id" : ObjectId("5c6d1dd5bbb230e2db452920"), "_cls" : "ScheduleEntry.ScheduledAnalytics.ResolveHostnames", "name" : "ResolveHostnames", "enabled" : true, "lock" : false, "frequency" : 3600, "description" : "Resolves hostnames and extracts subdomains", "status" : "OK", "last_run" : ISODate("2019-02-20T15:44:38.074Z") } { "_id" : ObjectId("5c6d1dd7bbb230e2db452931"), "_cls" : "ScheduleEntry.Feed.HostsFilePSH", "name" : "HostsFilePSH", "enabled" : false, "lock" : false, "source" : "https://hosts-file.net/psh.txt", "frequency" : 14400, "description" : "Domains associated to phishing attempts.", "last_run" : ISODate("2019-02-20T15:44:37.562Z") }

s-shubh commented 5 years ago

Again got the same error....

[root@ssmachine1 yeti]# python2.7 testfeeds.py plugins/feeds/public/hostsfile_fsa.py DEBUG:root:Scheduler started Traceback (most recent call last): File "testfeeds.py", line 19, in f = Feed.objects.get(name=name) File "/usr/lib/python2.7/site-packages/mongoengine/queryset/base.py", line 276, in get raise queryset._document.DoesNotExist(msg) core.feed.DoesNotExist: ScheduleEntry.Feed matching query does not exist.

It seems testfeeds.py is not working along with mongoengine queryset.

kittrCZ commented 1 year ago

Hi @s-shubh, have you ever been able to figure out the root cause of this issue? I'm experiencing it now after adding a new rule to the container. I have seen the exact same issue, but on the latest version of Yet, so I'm wondering what I'm doing wrong :/

Thanks for the help!

Docdez commented 5 months ago

For Note (In old versions, like 1.8.5)

That problem appears when we triggers Сelery workers restart (feeds etc.). The worker subprocess does not reload nested modules in the plugin package and the module items still use old schedule entity classes (mongoengine Documents), which caches all available class based handlers (DocumentInstance.objects). After that the subprocess run the scheduled task which try to get the handler from the cache but new schedule entity class are used (without anything).

Also Celery has problem with restart all worker subprocess (only one in time).