Helpers to spoof orgc - Githubissues

Rafiot commented 6 years ago

[ ] Add class to represent an org if needed
[ ] Add helpers to pass the required parameters

@iglocska what makes a valid orgc object? I'm assuming UUID, but is there other parameters that the helper must require?

iglocska commented 6 years ago

name + uuid, that's it afaik.

github-germ commented 6 years ago

MariaDB [misp]> desc organisations;
+----------------------+--------------+------+-----+---------+----------------+
| Field                | Type         | Null | Key | Default | Extra          |
+----------------------+--------------+------+-----+---------+----------------+
| id                   | int(11)      | NO   | PRI | NULL    | auto_increment |
| name                 | varchar(255) | NO   | MUL | NULL    |                |
| date_created         | datetime     | NO   |     | NULL    |                |
| date_modified        | datetime     | NO   |     | NULL    |                |
| description          | text         | YES  |     | NULL    |                |
| type                 | varchar(255) | YES  |     | NULL    |                |
| nationality          | varchar(255) | YES  |     | NULL    |                |
| sector               | varchar(255) | YES  |     | NULL    |                |
| created_by           | int(11)      | NO   |     | 0       |                |
| uuid                 | varchar(40)  | YES  | MUL | NULL    |                |
| contacts             | text         | YES  |     | NULL    |                |
| local                | tinyint(1)   | NO   |     | 0       |                |
| restricted_to_domain | text         | YES  |     | NULL    |                |
| landingpage          | text         | YES  |     | NULL    |                |
+----------------------+--------------+------+-----+---------+----------------+

iglocska commented 6 years ago

Definitely don't need all of those. Only name and uuid are synced.

Rafiot commented 6 years ago

So, the helpers are here, this is how to use it:

from pymisp import MISPEvent, MISPOrganisation

existing_event = MISPEvent()
existing_event.load('mispevent.json')
existing_event.orgc.name = 'Orgname'
existing_event.orgc.id = 123
existing_event.orgc.uuid = '43d4b157-681a-4d97-9c31-c3c9bb82b129'

# OR (assuming new_event is a object of type MISPEvent you just created)

misp_org = MISPOrganisation()
misp_org.name = 'Orgname'
misp_org.id = 123
misp_org.uuid = '43d4b157-681a-4d97-9c31-c3c9bb82b129'

new_event.orgc = misp_org

github-germ commented 6 years ago

OK, I copied your mispevent.py locally in site-packages, and had to modify your example above a little to make it run. But perhaps I misunderstood some things. When I created the event and updated it with the desired orgc it's not being changed.

...
try:
    # create event
    event = misp.new_event(
        distribution     = 0,        # our org only
        threat_level_id  = 2,        # medium
        analysis         = 0,        # initial
        info             = 'testing new_event mbg@att.com',
        date             = None,
        published        = False,
        orgc_id          = 2,        # TIARE -- gets set to 21 :-(
        org_id           = 21,       # iSIGHT
        sharing_group_id = None)

    sys.stdout.write('new_event() returned %s\n' % (str(event)))
except Exception as e:
    sys.stderr.write('new_event() exception: %s\n' % (str(e)))
    sys.exit(1)
try:
    # now update orgc
    misp_orgc                   = MISPOrganisation()
    misp_orgc.name              = 'TIARE'
    misp_orgc.id                = 2
    misp_orgc.uuid              = '5a6a4161-b2e0-46b6-b12e-1bca0a6b037c'

    # 1. without dict() update_event() throws 
    #    TypeError: Object of type 'MISPOrganisation' is not JSON serializable
    # 2. event['Event'].Orgc as I thought from your example no good
    event['Event']['Orgc']      = dict(misp_orgc) 

    # without adding a second to timestamp update_event() fails with
    # 'Event could not be saved: Event in the request not newer than the local copy.'
    event['Event']['timestamp'] = str(int(event['Event']['timestamp'])+1)

    updated_event  = misp.update_event(event['Event']['id'], event['Event'])
    sys.stdout.write('update_event() returned (secs=%.1f): %s\n' % (str(updated_event)))
except Exception as e:
    sys.stderr.write('update_event() exception: %s\n' % (str(e)))
    sys.exit(2)

Running as user in org_id=21


new_event() returned: {'Event': {'id': '12221', 'orgc_id':
'21', 'org_id': '21', 'date': '2018-06-18', 'threat_level_id': '2',
'info': 'testing new_event mbg@att.com', 'published': False, 'uuid':
'5b282087-0c34-44a1-bc50-6a54496a006a', 'attribute_count': '0',
'analysis': '0', 'timestamp': '1529356423', 'distribution': '0',
'proposal_email_lock': False, 'locked': False, 'publish_timestamp':
'0', 'sharing_group_id': '0', 'disable_correlation': False,
'extends_uuid': '', 'Org': {'id': '21', 'name': 'iSIGHT', 'uuid':
'5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'Orgc': {'id': '21', 'name':
'iSIGHT', 'uuid': '5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'Attribute':
[], 'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object':
[]}}
update_event() returned: {'Event': {'id': '12221',
'orgc_id': '21', 'org_id': '21', 'date': '2018-06-18',
'threat_level_id': '2', 'info': 'testing new_event mbg@att.com',
'published': False, 'uuid': '5b282087-0c34-44a1-bc50-6a54496a006a',
'attribute_count': '0', 'analysis': '0', 'timestamp': '1529356424',
'distribution': '0', 'proposal_email_lock': False, 'locked': False,
'publish_timestamp': '0', 'sharing_group_id': '0',
'disable_correlation': False, 'extends_uuid': '', 'Org': {'id': '21',
'name': 'iSIGHT', 'uuid': '5aec9d46-e47c-49f2-aace-06480a670c7f'},
'Orgc': {'id': '21', 'name': 'iSIGHT', 'uuid':
'5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'Attribute': [],
'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object': []}}

Rafiot commented 6 years ago

Instead of calling misp.new_event, can you create it locally (with a MISPEvent), set all the keys, and finally send the event created in memory with misp.add?

github-germ commented 6 years ago

If this is what you meant, then it's still not working. Otherwise, please tell me more. Thanks!!

          ...
    56 try:
    57     start = timer()
    58     sys.stdout.write('call new_event()\n')
    59     event = MISPEvent()
    60     event.distribution     = 0        # our org only
    61     event.threat_level_id  = 2        # medium
    62     event.analysis         = 0        # initial
    63     event.info             = 'testing new_event mbg@att.com'
    64     event.date             = None
    65     event.published        = False
    66     event.orgc_id          = 2        # TIARE
    67     event.orgc_name        = 'TIARE'
    68     event.orgc_uuid        = '5a6a4161-b2e0-46b6-b12e-1bca0a6b037c'
    69     event.org_id           = 21       # iSIGHT
    70     event.sharing_group_id = None
    71
    72     sys.stdout.write('new_event() returned (secs=%.1f): %s\n'
    73                      % (timer()-start, str(event)))
    74 except Exception as e:
    75     sys.stderr.write('new_event() exception (secs=%.1f): %s\n'
    76                      % (timer()-start, str(e)))
    77     sys.exit(6)
    78
    79 try:
    80     # now add the event with the desired org and orgc
    81     start = timer()
    82     sys.stdout.write('call add_event()\n')
    83     added_event  = misp.add_event(event)
    84
    85     sys.stdout.write('\nadd_event() returned (secs=%.1f): %s\n'
    86                      % (timer()-start, str(added_event)))
    87 except Exception as e:
    88     sys.stderr.write('add_event() exception (secs=%.1f): %s\n'
    89                      % (timer()-start, str(e)))
    90     sys.exit(7)
    91
    92 sys.exit(0)

Output


call new_event()
new_event() returned (secs=0.0): <MISPEvent(info=testing new_event
mbg@att.com)
call add_event()

add_event() returned (secs=0.1): {'Event': {'id': '12245', 'orgc_id': '21', 'org_id': '21', 'date': '2018-06-19', 'threat_level_id': '2', 'info': 'testing new_event mbg@att.com', 'published': False, 'uuid': '5b28f6e3-bf8c-40fb-97a9-58fa496a006a', 'attribute_count': '0', 'analysis': '0', 'timestamp': '1529411299', 'distribution': '0', 'proposal_email_lock': False, 'locked': False, 'publish_timestamp': '0', 'sharing_group_id': '0', 'disable_correlation': False, 'extends_uuid': '', 'Org': {'id': '21', 'name': 'iSIGHT', 'uuid': '5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'Orgc': {'id': '21', 'name': 'iSIGHT', 'uuid': '5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'Attribute': [], 'ShadowAttribute': [], 'RelatedEvent': [], 'Galaxy': [], 'Object': []}}

Rafiot commented 6 years ago

No, I meant that:

#!/usr/bin/env python3                                                                                                               
# -*- coding: utf-8 -*-                                                                                                              

from pymisp import PyMISP                                                                                                            
from keys import misp_url, misp_key                                                                                                  

from pymisp import MISPEvent, MISPOrganisation                                                                                       

event = MISPEvent()                                                                                                                  
event.distribution = 0        # our org only                                                                                         
event.threat_level_id = 2        # medium                                                                                            
event.analysis = 0        # initial                                                                                                  
event.info = 'testing new_event'                                                                                                     
event.published = False                                                                                                              

misp_org = MISPOrganisation()                                                                                                        
misp_org.name = 'bazbaz'                                                                                                             
misp_org.id = 15                                                                                                                     
misp_org.uuid = '5888a98d-a7e8-4183-94bb-4d19950d210f'                                                                               
event.Orgc = misp_org                                                                                                                

misp = PyMISP(misp_url, misp_key)                                                                                                    

misp.add_event(event)

And it is working.

github-germ commented 6 years ago

Yes, indeed that works. Thanks.

Question: Will this only work with add_event for a MISPEvent, i.e. no go with new_event since it's signature does not include a MISPOrganisation?

Rafiot commented 6 years ago

new_event creates an event immediately on the instance with only the meta information associated to the event (info, threat level, ..), add_event adds a full new event on the instance, with everything represented in a MISPEvent (including attributes, objects, and Orgc)

github-germ commented 6 years ago

@Rafiot

QUESTION: Now that I will get orgc set the way we need with new_event as you've explained, what's the best method to change orgc for existing events?

I believe this might not possible with PyMISP as it stands. Therefore, will doing a SQL update to the db suffice as a one time revision when we deploy our new PySight.py version?, e.g.

update events set orgc_id=2 where org_id=21 and orgc_id=21;

github-germ commented 6 years ago

@Rafiot -- Sorry, still a problem here... Changing orgc works well. HOWEVER, PyMISP.search_index(org=str) is not matching as it should, i.e. it appears that org is matching on the event's orgc and not org.

MariaDB [misp]> select id,org_id,orgc_id from events where id=14069;
+-------+--------+---------+
| id    | org_id | orgc_id |
+-------+--------+---------+
| 14069 |     21 |       2 |
+-------+--------+---------+
1 row in set (0.00 sec)

MariaDB [misp]>

(Pdb) ev = misp_instance.search_index(eventid=14069)
[20180706-16:51:20 DEBUG (139699026233152)] Starting new HTTPS connection (1): localhost
[20180706-16:51:20 DEBUG (139699026233152)] https://localhost:443 "POST /events/index HTTP/1.1" 200 1127
(Pdb) print(len(ev['response']))
1
(Pdb) print(ev['response'][0]['org_id'])
21
(Pdb) print(ev['response'][0]['orgc_id'])
2
(Pdb) ev1=misp_instance.search_index(org='21')
[20180706-16:53:06 DEBUG (139699026233152)] Starting new HTTPS connection (1): localhost
[20180706-16:53:06 DEBUG (139699026233152)] https://localhost:443 "POST /events/index HTTP/1.1" 200 2
(Pdb) print(len(ev1['response']))
0
(Pdb) ev2=misp_instance.search_index(org='2')
[20180706-16:53:47 DEBUG (139699026233152)] Starting new HTTPS connection (1): localhost
[20180706-16:53:48 DEBUG (139699026233152)] https://localhost:443 "POST /events/index HTTP/1.1" 200 3587513
(Pdb) print(len(ev2['response']))
3027
(Pdb)

github-germ commented 6 years ago

OK, I think I was confused by the fact that the arg to search_index is called org but it is actually matching on orgc which is the org creator.

github-germ commented 6 years ago

Now that I understand orgc to be org creator, help get this puzzle are lined up please:

USE CASE

take a feed from an outside source and ingest into MISP via a PyMISP client app
want the orgc to be the org that represents the outside source: call this OrgOut
want the org to be an org inside our team of analysts: call this OrgIn

QUESTIONS A. should the org of the MISP user running the PyMISP client ingestion app be OrgOut or OrgIn? B. can the MISP user of the app then set the MISPOrganisation in the MISPEvent.Org to the org of the OrgIn?

Rafiot commented 6 years ago

A. should be in OrgIn B. the Org of OrgIn is set by the server and you cannot force it.

Important note: the user running the ingestion has to be either a sync user, or a site admin If not, the MISP instance will ignore orgc.

github-germ commented 6 years ago

@Rafiot ... OK, those are the settings I was testing with. Thx for confirming. There is still an issue...

Creating the new event via PyMISP.add_event passing a constructed MISPEvent including with MISPEvent.Orgcset to OrgOut works well creating an event with org set to OrgIn, and orgc set to OrgOut as desired.

However, when there's a need to add more attributes to that now pre-existing event where the code gets the pre-existing event into a MISPEvent and calls add_attribute as needed and then calls PyMISP.update_event with that pre-existing id and the revised MISPEvent, the response is this error:


{
'name': 'Edit event failed.',
'message': 'Error',
'url': '/events/edit/14167',
'errors':
[
'Event could not be saved: The user used to edit the event is not authorised to do so.
This can be caused by the user not being of the same organisation as the original
creator of the event whilst also not being a site administrator.'
]
}

Rafiot commented 6 years ago

oh, this is interesting. @iglocska is it expected?

Rafiot commented 6 years ago

So this is where the exception comes from: https://github.com/MISP/MISP/blob/8d567782d9110c978ef12ce0331e8f11b80d1553/app/Model/Event.php#L2756

It seems that the event has to be "locked". Not sure what that mean.

Rafiot commented 6 years ago

Okay, so the reason it fails is because MISP locks this event as it doesn't comes from a synchronisation.

The way to solve this issue is to create the event with the locked field set to True.

#!/usr/bin/env python3                                                                                                               
# -*- coding: utf-8 -*-                                                                                                              

from pymisp import PyMISP                                                                                                            
from keys import misp_url, misp_key                                                                                                  

from pymisp import MISPEvent, MISPOrganisation                                                                                       

event = MISPEvent()                                                                                                                  
event.distribution = 0        # our org only                                                                                         
event.threat_level_id = 2        # medium                                                                                            
event.analysis = 0        # initial                                                                                                  
event.locked = True   # <====== This will allow you to update the event later on
event.info = 'testing new_event'                                                                                                     
event.published = False                                                                                                              

misp_org = MISPOrganisation()                                                                                                        
misp_org.name = 'bazbaz'                                                                                                             
misp_org.id = 15                                                                                                                     
misp_org.uuid = '5888a98d-a7e8-4183-94bb-4d19950d210f'                                                                               
event.Orgc = misp_org                                                                                                                

misp = PyMISP(misp_url, misp_key)                                                                                                    

misp.add_event(event)

github-germ commented 6 years ago

Cool. Was just going to try that your post came in. 👍 I assume no need to do some sort of unlock when done? I'll report back on this attempt. Thanks!

(btw, sorry if I was deemed a PITA on gitter earlier -- is pinging there on a issue is considered bad behavior?)

github-germ commented 6 years ago

Perhaps my test is invalid.

The event exists so load it into a MISPEvent via:

e = mispInstance.get(existingEventId)
mispEvent = MISPEvent()
mispEvent.from_dict(**e['Event'])
...
# all the other logic i had before including setting mispEvent.Orgc
...
mispEvent.locked = True
addedMispEvent = mispInstance.update_event(
event_id = mispEvent['id'],
event    = mispEvent)

addedMispEvent contains the same error dict as above.

...WAIT... Are you saying when the event is first created via add_event that locked should be set True? If that's the case, I will need to flip events.locked in the db for all the ones already created (or is there a better way?). And it's OK to simply leave all these events locked forever?

github-germ commented 6 years ago

OK, I tested again doing this first on the existing event about to be added to:

MariaDB [misp]> update events set locked=1 where id=14169;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

MariaDB [misp]> select id,orgc_id,org_id,locked from events where id=14169;
+-------+---------+--------+--------+
| id    | orgc_id | org_id | locked |
+-------+---------+--------+--------+
| 14169 |      21 |      2 |      1 |
+-------+---------+--------+--------+
1 row in set (0.00 sec)

MariaDB [misp]>

And the update_event() succeeded.

So...

Set locked=True every time I create an event from this client ingestion app?
Leave it locked forever?
Will having it locked prevent any other transactions on those events, e.g. can an analysts edit that event in the WebUI?

github-germ commented 6 years ago

chatted on gitter with @iglocska

Answers to https://github.com/MISP/PyMISP/issues/239#issuecomment-403859424

yes
yes
no

Transcript:

@adulau Orgc is the external creator of the premium feed we are ingesting, Org is our internal Analysts org.

github-germ @github-germ 13:08 does anyone know the meaning of an event being "locked"?

Andras Iklody @iglocska 13:08 yes

github-germ @github-germ 13:08 ok, thx. ha, what's it mean?

Andras Iklody @iglocska 13:08 locked = 0 means event was created on this instance and will reject remote sync edits

github-germ @github-germ 13:09 and in the context of setting orgc locked needs to be 1 because...?

Andras Iklody @iglocska 13:10 it's MISP's anti tampering protection

github-germ @github-germ 13:10 because by setting orgc it must be an event not from this instance in general?

Andras Iklody @iglocska 13:10 events that were synced (locked = 1) can be updated by sync users events that are local (locked = 0) will block edits by 3rd parties

github-germ @github-germ 13:10 ok, in my use case where i have my own ingestion app pulling from an external feed, this appears to make sense

Andras Iklody @iglocska 13:11 in that case you need to set locked = 1

github-germ @github-germ 13:11 yup, learned that today. and it's cool to leave it as 1. so i think that issue is now complete. nice.

Andras Iklody @iglocska 13:11 perfect ;) but best to use your own orgc

github-germ @github-germ 13:12 btw, was that documented somewhere. as a newbie, want tomake sure other than reading code that i am not missing places to read

Andras Iklody @iglocska 13:12 even if you pull remote feeds think in the rfc

github-germ @github-germ 13:12 yes, using our own created orgc i grab the org details from get_organisation() and then use that to mod Orgc to be an org we created to represent the source of that external data

Andras Iklody @iglocska 13:15 generally this is basically spoofing the orgc and not the cleanest way of handling it - you are encoding the data in MISP so it should be your event. The way we handle it for feeds: We pull them as CIRCL and mark the source with a tag such as source:vendor_name_feed alternatively, if you really want to stick to the organisations approach you could create a user for each of those organisations if you are just using them to spoof orgs and use their respective API keys

github-germ @github-germ 13:16 i was following the design that our analyst team wanted. i also tag it

Andras Iklody @iglocska 13:16 to create the data so for each feed's org create an automation user use that to push the data

github-germ @github-germ 13:16 yes did that

Andras Iklody @iglocska 13:16 = no spoofing needed the spoofing is only needed if the user that is pushing the data has a different org than the orgc of the event if you use the users of the individual orgs, you don't even have to set the orgc as it will be auto-assigned

github-germ @github-germ 13:17 ok, but they want orgc to be the xternal feed and org to be internal analyst owners

Andras Iklody @iglocska 13:17 ah ok. in that case spoofing it is

github-germ @github-germ 13:17 bingo

Andras Iklody @iglocska 13:17 just use the locked field as mentioned

github-germ @github-germ 13:18 yes, that did the trick

Andras Iklody @iglocska 13:18 perfect ;)

github-germ @github-germ 13:18 and setting to locked doesn't prevent any local transaction either API or WebUI on that event, right?

Andras Iklody @iglocska 13:18 locked = 1 is more permissive, however, only organsiations of the orgc org can modify it (+ site admins) _

github-germ commented 6 years ago

@Rafiot -- you may close this issue. Nice to have the secret potion. Thx for all your help @Rafiot and @iglocska !!

Rafiot commented 6 years ago

\o/

garrit-schroeder commented 5 years ago

Hello, i have a question regarding this issue. Is it possible to spoof the owner org as well?

Cheers

Rafiot commented 5 years ago

No, it's not. The owner org is always overwritten by the one owning the apikey.

garrit-schroeder commented 5 years ago

Tanks for the quick reply. So that means if an event has creator org 1 and owner orc 2. A user of org 1 can not see an even if distribution level is set to organization only?

Rafiot commented 5 years ago

Yes, that's correct.

MISP / PyMISP

Helpers to spoof orgc #239