update_object() does not work with custom objects

TheDr1ver commented 3 years ago

Version: Latest (2.4.148 for both MISP and PyMISP) - I just updated and confirmed it was still a bug right before opening this issue.

Expected behavior:

Create a custom object and save to a MISP event
At a later point in time, fetch that object and add an attribute to it
Update that object using the PyMISP.update_object() method
Custom object is updated in MISP with the newly added attribute
Go about your merry way

Actual behavior:

Create a custom object and save to a MISP event
Fetch that object and add an attribute to it
Update that object using the PyMISP.update_object() method
Something went wrong (403): {'saved': False, 'name': 'Could not edit Object', 'message': 'Could not edit Object', 'url': '/objects/edit', 'errors': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx'}
Spend half a day wondering what you did wrong

Reproduction code:

from pymisp import PyMISP, MISPObject

misp = PyMISP('https://<your_url>/', '<your_api_key>')

# Get existing event and add newly created custom object (no template exists)

demo_event = misp.get_event('1234', pythonify=True)
cust_obj = MISPObject('custom-object')
cust_obj.add_attribute('custom-attribute', value='testing1', type='text', to_ids=False)
updated_object = demo_event.add_object(cust_obj, pythonify=True)
cust_obj_uuid = updated_object.uuid

updated_event = misp.update_event(demo_event, pythonify=True)

# So far so good...

# try updating newly created object

cust_obj = misp.get_object(cust_obj_uuid, pythonify=True)
cust_obj.add_attribute('another-custom-attribute', value='testing2', type='text', to_ids=False)

# ...still okay...

latest_obj = misp.update_object(cust_obj)

# Something went wrong (403): {'saved': False, 'name': 'Could not edit Object', 'message': 'Could not edit Object', 'url': '/objects/edit', 'errors': 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx'}

I hope that captures the problem efficiently.

The only instance in which I could get update_object() to work properly was when I was interacting with an object that was already a default MISP template.

Are we no longer allowed to update MISP objects without first uploading a corresponding template? Or am I missing something obvious?

Thanks in advance for your help!

Rafiot commented 3 years ago

That's a very weird error you're getting there, and your code should just work. When you say no longer did that code work before?

I'm investigating the issue now.

Rafiot commented 3 years ago

That's super weird, as it works for me (see commit referenced above).

The errors key you're getting is an actual UUID, right? And not xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxx?

TheDr1ver commented 3 years ago

Thanks for getting back to me so quickly.

RE: no longer working - looking at an older script of mine, it appears this method used to work, but to be perfectly honest, I haven't investigated it too closely to see if it's still working or not.

As for the UUID, I only changed it to X's as a redaction - in practice it's an actual (and valid) UUID.

I noticed in your test script you appear to be defining a custom object path:

new_object = MISPObject('overwrite_file', misp_objects_path_custom='tests/mispevent_testfiles') ... new_object.force_misp_objects_path_custom('tests/mispevent_testfiles', 'overwrite_file')

I have not been using a custom path in my snippet example as I didn't think it was explicitly required for an ad-hoc custom object. I also have not been calling force_misp_objects_path_custom() as this is the first time I've seen that function, and, as I mentioned, I have not been defining a custom object path.

If you create a custom object without defining a custom path and without calling the force_misp_objects_path_custom function does it still work? If you run the exact code I provided (with changing nothing but the API info and the hard-coded event ID) does it not throw the 403 error?

I'll continue to dig into it on my end to see if I can pinpoint the issue. Is there anywhere I can find a better error message than the 403 Could not edit Object message I'm currently getting? If I could pinpoint exactly what's throwing the error I might have an easier time figuring out what I'm doing wrong.

TheDr1ver commented 3 years ago

To provide some clarification on my previous post -

After further review, this method of updating an object worked previously. Going back through my existing scripts and notes, it appears it was last working with version 2.4.140 and broke after I updated on May 20th to 2.4.143 (so it could have broken in any version between 140 and 143).

Digging deeper into the audit logs I found the following info which shines a little more light on the issue and is likely related to the root of the problem:

{
    "meta-category": [
        "Meta-category cannot be empty."
    ],
    "description": [
        "Description cannot be empty."
    ],
    "template_uuid": [
        "Please provide a valid RFC 4122 UUID"
    ]
}

Should these fields be required for a custom ad-hoc object?

Does that mean every MISP object needs an explicitly defined template going forward? If not, what values should I set these to, as using my current method has each of them set to null by default.

Rafiot commented 3 years ago

oh, right, I see. The weird thing is that you seem to be able to create the event (with the custom object), but updating fails.

I'm investigating that, and will come with a solution asap.

Rafiot commented 3 years ago

Right, confirmed: creating the object doesn't require meta-category/description/template_uuid but update doesn't work. That should probably be fixed on MISP site.

In the meantime, I'd strongly recommend you to create a custom object template and use it. The main advantage is that you can version it, even if you don't share the template itself publicly.

TheDr1ver commented 3 years ago

Thanks so much for all your help.

I'm still reluctant to create and upload definition.json files for every custom object I want to use (there are probably a dozen or so objects I need, and it seems like a lot of steps if I decide I just want to add a single attribute type to a custom object one day).

That being said, once I created a definition and loaded it into MISP, everything worked as expected. For (mainly just my own) future reference, once the JSON definition was created and loaded for custom-object with attributes custom-attribute and another-custom-attribute the following snippet behaved as expected and allowed me to update an object:

from pymisp import PyMISP, MISPObject

misp = PyMISP('https://<your_url>/', '<your_api_key>')

# Get existing event and add newly created custom object (no template exists)

demo_event = misp.get_event('1234', pythonify=True)

raw_template = misp.get_raw_object_template('custom-object')
cust_obj = MISPObject('custom-object', misp_objects_template_custom=raw_template)

# cust_obj.add_attribute('custom-attribute', value='testing1', type='text', to_ids=False)
# note - type is not required if the MISPObject was loaded with an explicit template
cust_obj.add_attribute('custom-attribute', value='testing1', to_ids=False)

updated_object = demo_event.add_object(cust_obj, pythonify=True)
cust_obj_uuid = updated_object.uuid

updated_event = misp.update_event(demo_event, pythonify=True)

# So far so good...

# try updating newly created object

cust_obj = misp.get_object(cust_obj_uuid, pythonify=True)
cust_obj.add_attribute('another-custom-attribute', value='testing2', type='text', to_ids=False)

latest_obj = misp.update_object(cust_obj)

# Success!

One last question before I close this out - should I bother tracking down the inability to use update_object() on an ad-hoc custom object on the MISP side? Or do you think explicit templates will be a requirement for custom objects moving forward?

TheDr1ver commented 3 years ago

For what it's worth, I think this may have been the change that broke update_object()

https://github.com/MISP/MISP/commit/06cc58e16360ccce8df5020b536cc567d91a206c

Rafiot commented 3 years ago

Good catch, let's see what @mokaddem and @iglocska say, there might be a reason I'm not aware of to have that.

I'm a bit worried about the vast amount of custom objects you need, it might make sharing events complicated. Can you tell me more about your usecase? Maybe we can come with a solution that requires a finite amount of objects (and/or generating the templates on the fly, it is also possible with https://github.com/MISP/PyMISPObjectTemplates/)

TheDr1ver commented 3 years ago

I sent a DM on Twitter RE: my specific use case, as I'd rather not discuss it in this open forum. I'll be sure to take a look at generating templates on the fly with PyMISPObjectTemplates in the meantime - thanks for the suggestion!

Rafiot commented 3 years ago

You're welcome, and let us know if you have further questions!

MISP / PyMISP

update_object() does not work with custom objects #776