Open AndreC10002 opened 4 years ago
And the JSON file of an event:
{ "Event": { "extends_uuid": "", "publish_timestamp": 1577336929, "info": "Active DGA for Bamital", "threat_level_id": 1, "published": true, "analysis": 1, "date": "2019-12-25", "uuid": "5e04405d-af64-4215-828c-0af3ac110002", "timestamp": 1577336927, "Orgc": { "uuid": "5df12de7-b018-4f25-ac49-04bdac110002", "name": "My ORG" }, "Tag": [ { "colour": "#fbff30", "name": "dga" }, { "colour": "#FFC000", "name": "tlp:amber" }, { "colour": "#075200", "name": "admiralty-scale:source-reliability=\"b\"" }, { "colour": "#0eb100", "name": "admiralty-scale:information-credibility=\"1\"" }, { "colour": "#0088cc", "name": "misp-galaxy:botnet=\"Bamital\"" } ], "Attribute": [ { "type": "domain", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc", "uuid": "5e04405d-0d54-4d02-b1b7-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "domain|ip", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc|175.126.123.219", "uuid": "5e04405d-2e54-429e-bc66-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "ip-dst", "comment": "", "to_ids": true, "category": "External analysis", "value": "175.126.123.219", "uuid": "5e04405d-d8a0-49b9-b35a-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "comment", "comment": "", "to_ids": true, "category": "External analysis", "value": "Bamital", "uuid": "5e04405d-dd40-400f-bd8c-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "datetime", "comment": "", "to_ids": true, "category": "Other", "value": "2019-12-25T00:00:00", "uuid": "5e04405d-312c-42f9-bbb6-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false } ] } }
I think the problem is due to the fact your manifest file is a list of dictionaries, when is should just be a dictionary.
Your manifest file should look like that:
{
"5e04405b-20f4-4f12-a6ba-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336925
},
"5e04405d-af64-4215-828c-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-25",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336927
},
"5e044060-5090-48af-a0d7-0af3ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-26",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336930
},
"5e044062-b804-4ac4-9df4-0eb6ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"name": "dga",
"colour": "#fbff30"
}, {
"name": "tlp:amber",
"colour": "#FFC000"
}, {
"name": "admiralty-scale:source-reliability=\"b\"",
"colour": "#075200"
}, {
"name": "admiralty-scale:information-credibility=\"1\"",
"colour": "#0eb100"
}, {
"name": "misp-galaxy:botnet=\"Bamital\"",
"colour": "#0088cc"
}],
"info": "Active DGA for Bamital",
"date": "2019-12-27",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1577336932
}
}
I ran the feed generator locally and the manifest file was as expected, so I'm not sure why you have an incorrect file on your end. Can you re-try with the latest version of PyMISP from the repository>
I see. Not sure why that happened, but with the latest version of PyMISP the manifest.json looks OK. Does it look OK to you? But the error persists:
{
"5df941c1-e490-4032-a8b5-0061ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Pony",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616388
},
"5df941c4-fa90-4d05-9be5-021bac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616390
},
"5df941c7-240c-4555-ab12-0145ac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616393
},
"5df941ca-a82c-4774-ab9e-005eac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616396
},
"5df941cc-979c-447e-b859-021fac110002": {
"Orgc": {
"uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
"name": "My ORG"
},
"Tag": [{
"colour": "#641cd9",
"name": "c2_address"
}, {
"colour": "#FFC000",
"name": "tlp:amber"
}, {
"colour": "#075200",
"name": "admiralty-scale:source-reliability=\"b\""
}, {
"colour": "#0eb100",
"name": "admiralty-scale:information-credibility=\"1\""
}, {
"colour": "#004f4f",
"name": "rsit:malicious-code=\"c2-server\""
}],
"info": "Command and Control for Lokibot",
"date": "2019-12-16",
"analysis": 1,
"threat_level_id": 1,
"timestamp": 1576616398
}
}
The importing server still shows:
==> error.log <==
2020-01-02 17:05:42 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59659')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59659')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
I'm running Python 3.8.1 and PyMISP 2.4.119.1. The servers (both exporting and importing) run MISP v2.4.119.
I'm probably missing the point here, but now I don't see any structural difference between my manifest.json and yours. And, because my understanding of how MISP ingests feeds is very limited, I can't spot the reason for the error.
Again, sorry for the multiple messages but I'm writing as I try to debug it.
This is odd, I replaced the manifest.json file with yours and expected the importing server to produce a different error, something related to it now finding the corresponding JSON files with events or inconsistency with hashes.json. Instead, I got exactly the same error:
2020-01-02 17:32:08 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59661')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59661')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
OK, I found the problem. And it is very odd and not related to PyMISP, but to MISP itself.
The feed was hosted in an URL like this:
https://www.example.com/misp/bla:c2_address
That when MISP URL encodes, becomes:
https://www.example.com/misp/bla%3ac2_address
If I host it at:
https://www.example.com/misp/c2_address
It works! Therefore, I believe the problem is with the ':' character or its URL encoded version '%3a'. I'm not sure, but I'm relieved it works.
What is the best way to file this as a MISP bug?
Thank you very much for your assistance with this matter!
Oh wow, nice catch. Let me loop @iglocska in for that one, because I'm unsure about the way forward, and if/how we can fix it.
I still face a similiar error:
` 2023-01-30 18:52:42 Warning: Could not add event '5a5df804-acb5-4fd3-8c76-6982e5e1ce75' from feed 66: 1900 2023-01-30 18:52:42 Warning: Could not add event 'ca0e87d9-b850-404d-8b17-e51d2e2b717e' from feed 66: 1901 2023-01-30 18:52:42 Warning: Could not add event '83b4018e-1f45-48c4-908b-2ef8d2e1db0f' from feed 66: 1902 2023-01-30 18:52:42 Warning: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66: 1903 2023-01-30 18:52:43 Error: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66. [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'Purchase order.exe' for key 'value' Stack Trace:
`
Moreover, I face another problem probably linked to that:
The previous issue has been experienced both with simple workers and simplebackgroundjob
I don't think it is the same issue. But it is most probably not PyMISP related (please tell me if I'm wrong). Can you open an issue in the MISP repository?
We are successfully generating our feeds using feed-generate.py but when we add them to a different MISP instance, the following message is presented on debug logs:
==> error.log <== 2019-12-30 17:46:11 Error: Could not add event '0' from feed XX. [InvalidArgumentException] Given event UUID '0' is invalid. Stack Trace:
0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSo
cket))
1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Arra
y, Array)
2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Ar
ray, '59652')
3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Arra
y, '59652')
4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
10 {main}
The only file downloaded is the manifest.json and its JSON structure is correct. I can't see in it any UUID as '0', our org UUID is correct.
Could this be that the manifest.json file generated by feed-generate.py is incompatible with the latest version of MISP?
Snippet of a generated manifest.json file:
Thank you!