search

MISP / PyMISP

Python library using the MISP Rest API
Other
444 stars 279 forks source link

Error: Could not add event '0' from feed XX - incompatible manifest.json file? #509

Open AndreC10002 opened 4 years ago

AndreC10002 commented 4 years ago

We are successfully generating our feeds using feed-generate.py but when we add them to a different MISP instance, the following message is presented on debug logs:

==> error.log <== 2019-12-30 17:46:11 Error: Could not add event '0' from feed XX. [InvalidArgumentException] Given event UUID '0' is invalid. Stack Trace:

0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSo

cket))

1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Arra

y, Array)

2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Ar

ray, '59652')

3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Arra

y, '59652')

4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()

5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()

6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()

7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))

8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')

9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')

10 {main}

The only file downloaded is the manifest.json and its JSON structure is correct. I can't see in it any UUID as '0', our org UUID is correct.

Could this be that the manifest.json file generated by feed-generate.py is incompatible with the latest version of MISP?

Snippet of a generated manifest.json file:

[{
    "5e04405b-20f4-4f12-a6ba-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-26",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336925
    },
    "5e04405d-af64-4215-828c-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-25",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336927
    },
    "5e044060-5090-48af-a0d7-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-26",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336930
    },
    "5e044062-b804-4ac4-9df4-0eb6ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-27",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336932
    }
}]

Thank you!

AndreC10002 commented 4 years ago

And the JSON file of an event:

{ "Event": { "extends_uuid": "", "publish_timestamp": 1577336929, "info": "Active DGA for Bamital", "threat_level_id": 1, "published": true, "analysis": 1, "date": "2019-12-25", "uuid": "5e04405d-af64-4215-828c-0af3ac110002", "timestamp": 1577336927, "Orgc": { "uuid": "5df12de7-b018-4f25-ac49-04bdac110002", "name": "My ORG" }, "Tag": [ { "colour": "#fbff30", "name": "dga" }, { "colour": "#FFC000", "name": "tlp:amber" }, { "colour": "#075200", "name": "admiralty-scale:source-reliability=\"b\"" }, { "colour": "#0eb100", "name": "admiralty-scale:information-credibility=\"1\"" }, { "colour": "#0088cc", "name": "misp-galaxy:botnet=\"Bamital\"" } ], "Attribute": [ { "type": "domain", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc", "uuid": "5e04405d-0d54-4d02-b1b7-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "domain|ip", "comment": "", "to_ids": true, "category": "External analysis", "value": "003d179fbbf1bead22105705142d6db7.co.cc|175.126.123.219", "uuid": "5e04405d-2e54-429e-bc66-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "ip-dst", "comment": "", "to_ids": true, "category": "External analysis", "value": "175.126.123.219", "uuid": "5e04405d-d8a0-49b9-b35a-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "comment", "comment": "", "to_ids": true, "category": "External analysis", "value": "Bamital", "uuid": "5e04405d-dd40-400f-bd8c-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false }, { "type": "datetime", "comment": "", "to_ids": true, "category": "Other", "value": "2019-12-25T00:00:00", "uuid": "5e04405d-312c-42f9-bbb6-0af3ac110002", "timestamp": 1577336925, "disable_correlation": false } ] } }

Rafiot commented 4 years ago

I think the problem is due to the fact your manifest file is a list of dictionaries, when is should just be a dictionary.

Your manifest file should look like that:

{
    "5e04405b-20f4-4f12-a6ba-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-26",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336925
    },
    "5e04405d-af64-4215-828c-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-25",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336927
    },
    "5e044060-5090-48af-a0d7-0af3ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-26",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336930
    },
    "5e044062-b804-4ac4-9df4-0eb6ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "name": "dga",
            "colour": "#fbff30"
        }, {
            "name": "tlp:amber",
            "colour": "#FFC000"
        }, {
            "name": "admiralty-scale:source-reliability=\"b\"",
            "colour": "#075200"
        }, {
            "name": "admiralty-scale:information-credibility=\"1\"",
            "colour": "#0eb100"
        }, {
            "name": "misp-galaxy:botnet=\"Bamital\"",
            "colour": "#0088cc"
        }],
        "info": "Active DGA for Bamital",
        "date": "2019-12-27",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1577336932
    }
}

I ran the feed generator locally and the manifest file was as expected, so I'm not sure why you have an incorrect file on your end. Can you re-try with the latest version of PyMISP from the repository>

AndreC10002 commented 4 years ago

I see. Not sure why that happened, but with the latest version of PyMISP the manifest.json looks OK. Does it look OK to you? But the error persists:

{
    "5df941c1-e490-4032-a8b5-0061ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "colour": "#641cd9",
            "name": "c2_address"
        }, {
            "colour": "#FFC000",
            "name": "tlp:amber"
        }, {
            "colour": "#075200",
            "name": "admiralty-scale:source-reliability=\"b\""
        }, {
            "colour": "#0eb100",
            "name": "admiralty-scale:information-credibility=\"1\""
        }, {
            "colour": "#004f4f",
            "name": "rsit:malicious-code=\"c2-server\""
        }],
        "info": "Command and Control for Pony",
        "date": "2019-12-16",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1576616388
    },
    "5df941c4-fa90-4d05-9be5-021bac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "colour": "#641cd9",
            "name": "c2_address"
        }, {
            "colour": "#FFC000",
            "name": "tlp:amber"
        }, {
            "colour": "#075200",
            "name": "admiralty-scale:source-reliability=\"b\""
        }, {
            "colour": "#0eb100",
            "name": "admiralty-scale:information-credibility=\"1\""
        }, {
            "colour": "#004f4f",
            "name": "rsit:malicious-code=\"c2-server\""
        }],
        "info": "Command and Control for Lokibot",
        "date": "2019-12-16",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1576616390
    },
    "5df941c7-240c-4555-ab12-0145ac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "colour": "#641cd9",
            "name": "c2_address"
        }, {
            "colour": "#FFC000",
            "name": "tlp:amber"
        }, {
            "colour": "#075200",
            "name": "admiralty-scale:source-reliability=\"b\""
        }, {
            "colour": "#0eb100",
            "name": "admiralty-scale:information-credibility=\"1\""
        }, {
            "colour": "#004f4f",
            "name": "rsit:malicious-code=\"c2-server\""
        }],
        "info": "Command and Control for Lokibot",
        "date": "2019-12-16",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1576616393
    },
    "5df941ca-a82c-4774-ab9e-005eac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "colour": "#641cd9",
            "name": "c2_address"
        }, {
            "colour": "#FFC000",
            "name": "tlp:amber"
        }, {
            "colour": "#075200",
            "name": "admiralty-scale:source-reliability=\"b\""
        }, {
            "colour": "#0eb100",
            "name": "admiralty-scale:information-credibility=\"1\""
        }, {
            "colour": "#004f4f",
            "name": "rsit:malicious-code=\"c2-server\""
        }],
        "info": "Command and Control for Lokibot",
        "date": "2019-12-16",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1576616396
    },
    "5df941cc-979c-447e-b859-021fac110002": {
        "Orgc": {
            "uuid": "5df12de7-b018-4f25-ac49-04bdac110002",
            "name": "My ORG"
        },
        "Tag": [{
            "colour": "#641cd9",
            "name": "c2_address"
        }, {
            "colour": "#FFC000",
            "name": "tlp:amber"
        }, {
            "colour": "#075200",
            "name": "admiralty-scale:source-reliability=\"b\""
        }, {
            "colour": "#0eb100",
            "name": "admiralty-scale:information-credibility=\"1\""
        }, {
            "colour": "#004f4f",
            "name": "rsit:malicious-code=\"c2-server\""
        }],
        "info": "Command and Control for Lokibot",
        "date": "2019-12-16",
        "analysis": 1,
        "threat_level_id": 1,
        "timestamp": 1576616398
    }
}

The importing server still shows:

==> error.log <==
2020-01-02 17:05:42 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59659')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59659')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
AndreC10002 commented 4 years ago

I'm running Python 3.8.1 and PyMISP 2.4.119.1. The servers (both exporting and importing) run MISP v2.4.119.

AndreC10002 commented 4 years ago

I'm probably missing the point here, but now I don't see any structural difference between my manifest.json and yours. And, because my understanding of how MISP ingests feeds is very limited, I can't spot the reason for the error.

AndreC10002 commented 4 years ago

Again, sorry for the multiple messages but I'm writing as I try to debug it.

This is odd, I replaced the manifest.json file with yours and expected the importing server to produce a different error, something related to it now finding the corresponding JSON files with events or inconsistency with hashes.json. Instead, I got exactly the same error:

2020-01-02 17:32:08 Error: Could not add event '0' from feed 69.
[InvalidArgumentException] Given event UUID '0' is invalid.
Stack Trace:
#0 /var/www/MISP/app/Model/Feed.php(710): Feed->downloadAndParseEventFromFeed(Array, 0, Object(HttpSocket))
#1 /var/www/MISP/app/Model/Feed.php(412): Feed->__addEventFromFeed(Object(HttpSocket), Array, 0, Array, Array)
#2 /var/www/MISP/app/Model/Feed.php(787): Feed->downloadFromFeed(Array, Array, Object(HttpSocket), Array, '59661')
#3 /var/www/MISP/app/Console/Command/ServerShell.php(196): Feed->downloadFromFeedInitiator('69', Array, '59661')
#4 /var/www/MISP/app/Console/Command/AppShell.php(32): ServerShell->fetchFeed()
#5 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Job.php(199): AppShell->perform()
#6 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(278): Resque_Job->perform()
#7 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/lib/Resque/Worker.php(241): Resque_Worker->perform(Object(Resque_Job))
#8 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(109): Resque_Worker->work('5')
#9 /var/www/MISP/app/Vendor/kamisama/php-resque-ex/bin/resque(100): startWorker('default', 1, Object(MonologInit\MonologInit), '5')
#10 {main}
AndreC10002 commented 4 years ago

OK, I found the problem. And it is very odd and not related to PyMISP, but to MISP itself.

The feed was hosted in an URL like this:

https://www.example.com/misp/bla:c2_address

That when MISP URL encodes, becomes:

https://www.example.com/misp/bla%3ac2_address

If I host it at:

https://www.example.com/misp/c2_address

It works! Therefore, I believe the problem is with the ':' character or its URL encoded version '%3a'. I'm not sure, but I'm relieved it works.

What is the best way to file this as a MISP bug?

Thank you very much for your assistance with this matter!

Rafiot commented 4 years ago

Oh wow, nice catch. Let me loop @iglocska in for that one, because I'm unsure about the way forward, and if/how we can fix it.

valpet93 commented 1 year ago

I still face a similiar error:

` 2023-01-30 18:52:42 Warning: Could not add event '5a5df804-acb5-4fd3-8c76-6982e5e1ce75' from feed 66: 1900 2023-01-30 18:52:42 Warning: Could not add event 'ca0e87d9-b850-404d-8b17-e51d2e2b717e' from feed 66: 1901 2023-01-30 18:52:42 Warning: Could not add event '83b4018e-1f45-48c4-908b-2ef8d2e1db0f' from feed 66: 1902 2023-01-30 18:52:42 Warning: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66: 1903 2023-01-30 18:52:43 Error: Could not add event 'de55cf56-1a4d-4d46-954c-40f0f176d53e' from feed 66. [PDOException] SQLSTATE[23000]: Integrity constraint violation: 1062 Duplicate entry 'Purchase order.exe' for key 'value' Stack Trace:

0 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(502): PDOStatement->execute()

1 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(468): DboSource->_execute()

2 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Datasource/DboSource.php(1132): DboSource->execute()

3 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1942): DboSource->create()

4 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()

5 /var/www/MISP/app/Model/OverCorrelatingValue.php(50): Model->save()

6 /var/www/MISP/app/Model/Correlation.php(503): OverCorrelatingValue->block()

7 /var/www/MISP/app/Model/Attribute.php(470): Correlation->afterSaveCorrelation()

8 /var/www/MISP/app/Lib/Tools/BetterCakeEventManager.php(21): Attribute->afterSave()

9 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1970): BetterCakeEventManager->dispatch()

10 /var/www/MISP/app/Lib/cakephp/lib/Cake/Model/Model.php(1760): Model->_doSave()

11 /var/www/MISP/app/Model/Attribute.php(2518): Model->save()

12 /var/www/MISP/app/Model/MispObject.php(1134): Attribute->captureAttribute()

13 /var/www/MISP/app/Model/Event.php(3776): MispObject->captureObject()

14 /var/www/MISP/app/Model/Feed.php(1098): Event->_add()

15 /var/www/MISP/app/Model/Feed.php(693): Feed->__addEventFromFeed()

16 /var/www/MISP/app/Model/Feed.php(1182): Feed->downloadFromFeed()

17 /var/www/MISP/app/Console/Command/ServerShell.php(404): Feed->downloadFromFeedInitiator()

18 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/Shell.php(459): ServerShell->fetchFeed()

19 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(222): Shell->runCommand()

20 /var/www/MISP/app/Lib/cakephp/lib/Cake/Console/ShellDispatcher.php(66): ShellDispatcher->dispatch()

21 /var/www/MISP/app/Console/cake.php(45): ShellDispatcher::run()

22 {main}

`

Moreover, I face another problem probably linked to that: image

The previous issue has been experienced both with simple workers and simplebackgroundjob

Rafiot commented 1 year ago

I don't think it is the same issue. But it is most probably not PyMISP related (please tell me if I'm wrong). Can you open an issue in the MISP repository?