why isn't change_comment() working for me?

github-germ commented 5 years ago

i'm trying to use change_comment for the first time. it's not working, i.e. the attribute in the event where i apply the change when observed in the WebUI has not changed, nor if i retrieve the attribute again via PyMISP as below:

also, why did change_comment seemingly return all of my events -- notice the len is 4917 below.

here's an excerpt from a python debugger session:

(Pdb) print(newComment)
score:[5.7,2018081615],[7.8,2018081615]
(Pdb) print(attr['uuid'])
47fcf8f0-63af-4297-b21c-a6af79f2a7bb
(Pdb) print(attr)
{'id': '1031069', 'event_id': '14959', 'object_id': '0', 'object_relation': None, 'category': 'Network activity', 'type': 'ip-dst', 'to_ids': True, 'uuid': '47fcf8f0-63af-4297-b21c-a6af79f2a7bb', 'timestamp': '1534432498', 'distribution': '5', 'sharing_group_id': '0', 'comment': 'score:[5.7,2018081615]', 'deleted': False, 'disable_correlation': False, 'value': '1.0.128.131', 'Tag': [{'id': '1101', 'name': 'X-Force-Inactive', 'colour': '#ff0000', 'exportable': True, 'org_id': False, 'hide_tag': False, 'user_id': '0'}, {'id': '964', 'name': 'misp:threat-level="no-risk"', 'colour': '#b3b3b3', 'exportable': True, 'org_id': False, 'hide_tag': False, 'user_id': '0'}]}
(Pdb) print(attr['comment'])
score:[5.7,2018081615]
(Pdb) rc = misp.change_comment(attr['uuid'], newComment)
(Pdb) print(len(rc['response']))
4917
(Pdb) print(rc['response'][0])
{'id': '14962', 'org_id': '2', 'date': '2018-08-16', 'info': 'IBM X-Force 20180816 Malware', 'uuid': '5b759520-c3c4-4862-90dc-5d75496a006a', 'published': True, 'analysis': '0', 'attribute_count': '5', 'orgc_id': '22', 'timestamp': '1534432544', 'distribution': '0', 'sharing_group_id': '0', 'proposal_email_lock': False, 'locked': True, 'threat_level_id': '2', 'publish_timestamp': '1534432544', 'disable_correlation': False, 'extends_uuid': '', 'Org': {'id': '2', 'name': 'TIARE', 'uuid': '5a6a4161-b2e0-46b6-b12e-1bca0a6b037c'}, 'Orgc': {'id': '22', 'name': 'X-Force', 'uuid': '5afc45ad-54d4-47ec-9bb7-52bc0a670c7f'}, 'EventTag': [{'id': '50235', 'event_id': '14962', 'tag_id': '944', 'Tag': {'id': '944', 'name': 'X-Force', 'colour': '#a31d96'}}]}
(Pdb) revisedAttr = misp.search(controller='attributes', uuid='47fcf8f0-63af-4297-b21c-a6af79f2a7bb')
(Pdb) print(revisedAttr)
{'response': {'Attribute': [{'id': '1031069', 'event_id': '14959', 'object_id': '0', 'object_relation': None, 'category': 'Network activity', 'type': 'ip-dst', 'to_ids': True, 'uuid': '47fcf8f0-63af-4297-b21c-a6af79f2a7bb', 'timestamp': '1534432498', 'distribution': '5', 'sharing_group_id': '0', 'comment': 'score:[5.7,2018081615]', 'deleted': False, 'disable_correlation': False, 'value': '1.0.128.131', 'Tag': [{'id': '1100', 'name': 'X-Force-Active', 'colour': '#00ff00', 'exportable': True, 'org_id': False, 'hide_tag': False, 'user_id': '0'}, {'id': '967', 'name': 'misp:threat-level="high-risk"', 'colour': '#d60000', 'exportable': True, 'org_id': False, 'hide_tag': False, 'user_id': '0'}]}]}}
(Pdb)
(Pdb) print(revisedAttr['response']['Attribute'][0]['comment'])
score:[5.7,2018081615]
(Pdb)

github-germ commented 5 years ago

@Rafiot -- should I be changing an attribute comment another way? Thx!!

github-germ commented 5 years ago

On Gitter...

Andras Iklody @iglocska 08:30
change_comment() is deprecated, please use update_attribute()
...
misp.update_attribute(attr['uuid'], {'comment': newComment})

github-germ commented 5 years ago

@Rafiot ... using update_attribute did not revise the comment either :-(

it returned a response with a long list of events. no error. i validated that there was no change by looking in the db.

github-germ commented 5 years ago

also tried passing a MISPAttribute to update_attribute and that doesn't work either.

github-germ commented 5 years ago

none of these 4 attempts work...

def reviseAttrComment(misp, attr, score, timestamp):
    '''
    TBD
    currently not working. see https://github.com/MISP/PyMISP/issues/260
    '''
    newComment = getComment(
        score        = score,
        timestamp    = timestamp,
        priorComment = attr['comment'])

    # 1
    #mispAttr = MISPAttribute()
    #mispAttr.from_dict(**attr)
    #r = misp.update_attribute(attr['uuid'], mispAttr)

    # 2
    #r = misp.update_attribute(attr['uuid'], {'comment': newComment})

    # 3
    #r = misp.update_attribute(attr['id'], {'comment': newComment})

    # 4
    r = misp.update_attribute(int(attr['id']), {'comment': newComment})

    if not 'response' in r or not r['response']:
        sys.stderr.write('reviseAttrComment failed for attr="%s" (id=%s), newComment="%s"'
                % (attr['value'], attr['id'], newComment))

github-germ commented 5 years ago

@Rafiot -- what next on this one? thx...

Rafiot commented 5 years ago

This one works for me: misp.update_attribute(194573, {'comment': 'bar'})

Are you sure you have the permission? Also, this call returns an attribute, not a full event.

github-germ commented 5 years ago

Are you sure you have the permission? Also, this call returns an attribute, not a full event.

The same code creates the attribute in a prior execution. If it was a permission issue, what can I look at to know?

iglocska commented 5 years ago

If you use the same key and curl - does the same action work?

Post to:

/attributes/edit/[attribute_id]

Headers:

  Authorization: your_key
  Content-type: application/json
  Accept: application/json

Payload:

{"comment":"foo"}

On Thu, Aug 23, 2018 at 1:23 PM github-germ notifications@github.com wrote:

Are you sure you have the permission? Also, this call returns an attribute, not a full event.

The same code creates the attribute in a prior execution. If it was a permission issue, what can I look at to know?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/MISP/PyMISP/issues/260#issuecomment-415380993, or mute the thread https://github.com/notifications/unsubscribe-auth/ADf6wH4eCIDC8v5x7p6v1R3PsKvRfotwks5uTpC8gaJpZM4WAJip .

github-germ commented 5 years ago

OK, here are test results for python and curl attempts to revise an attribute comment: Both fail to update, i.e. no change. The python return from update_attribute is a list of all events. I include the transcripts for both along with the apache log entry.

The user running the script that creates the event with this attribute and attempts to revise the comment is a member of org_id=2 named TIARE, which is a Sync user as tagger (tagger and tag editor enabled). The curl also runs as that same user to attempt the comment revision.

You may recall that we create events with a different org and orgc. This event is created with org set to org_id=2 named TIARE, and orgc set to org_id=22 named X-Force.

PYTHON

$ fold -w 60 -s github.pymisp.260.issue
mg973b@misp-vm:~/src/misp/xforce python -m pdb
./xforceIngest.py data/ip.delete.delta.csv
> /home/mg973b/src/misp/xforce/xforceIngest.py(23)<module>()
-> '''
(Pdb) b 267
Breakpoint 1 at
/home/mg973b/src/misp/xforce/xforceIngest.py:267
(Pdb) r

revise eventId=15063 "IBM X-Force IP Spam"
..
update_event for cat=Spam
  eventId=15063 "IBM X-Force IP Spam"
    attr=2: active=0 + inactive=2
>
/home/mg973b/src/misp/xforce/xforceIngest.py(267)reviseAttrC
omment()
-> r = misp.update_attribute(attr['uuid'], {'comment':
newComment})
(Pdb) print(attr)
{'id': '1039219', 'event_id': '15063', 'object_id': '0',
'object_relation': None, 'category': 'Network activity',
'type': 'ip-dst', 'to_ids': True, 'uuid':
'61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5', 'timestamp':
'1534881201', 'distribution': '5', 'sharing_group_id': '0',
'comment': 'score:[5.7,2018082119]', 'deleted': False,
'disable_correlation': False, 'value': '1.0.128.131'}
(Pdb) newComment
'score:[5.7,2018082119],[1.2,2018082416]'
(Pdb) list
255 def reviseAttrComment(misp, attr, score, timestamp):
256     '''
257     TBD
258     currently not working. 
see https://github.com/MISP/PyMISP/issues/260
259     '''
260     newComment = getComment(
261         score        = score,
262         timestamp    = timestamp,
263         priorComment = attr['comment'])
264
265
266     # 1 -- my orig code
267 B-> r = misp.update_attribute(attr['uuid'],
{'comment': newComment})
268
(Pdb) n
>
/home/mg973b/src/misp/xforce/xforceIngest.py(280)reviseAttrC
omment()
-> if not 'response' in r or not r['response']:
(Pdb) len(r['response'])
5021
(Pdb) print(r['response'][0])
{'id': '15116', 'org_id': '2', 'date': '2018-08-23',
'info': 'Indicator Report: ATMOS Activity Report (Aug 21,
2018) PySight 18-00013734', 'uuid':
'5b7e80b9-caf8-4a8b-bc73-5d75496a006a', 'published': True,
'analysis': '0', 'attribute_count': '16', 'orgc_id': '21',
'timestamp': '1535017252', 'distribution': '0',
'sharing_group_id': '0', 'proposal_email_lock': False,
'locked': True, 'threat_level_id': '2',
'publish_timestamp': '1535017252', 'disable_correlation':
False, 'extends_uuid': '', 'Org': {'id': '2', 'name':
'TIARE', 'uuid': '5a6a4161-b2e0-46b6-b12e-1bca0a6b037c'},
'Orgc': {'id': '21', 'name': 'iSIGHT', 'uuid':
'5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'EventTag':
[{'id': '50787', 'event_id': '15116', 'tag_id': '335',
'Tag': {'id': '335', 'name': 'iSight', 'colour':
'#00ace6'}}, {'id': '50788', 'event_id': '15116', 'tag_id':
'337', 'Tag': {'id': '337', 'name': 'tlp:amber', 'colour':
'#FFC000'}}, {'id': '50789', 'event_id': '15116', 'tag_id':
'338', 'Tag': {'id': '338', 'name':
'veris:discovery_method="Prt - monitoring service"',
'colour': '#a0fe74'}}, {'id': '50790', 'event_id': '15116',
'tag_id': '339', 'Tag': {'id': '339', 'name':
'veris:actor:external:variety="Organized crime"', 'colour':
'#731fdd'}}, {'id': '50791', 'event_id': '15116', 'tag_id':
'340', 'Tag': {'id': '340', 'name':
'veris:action:malware:variety="C2"', 'colour': '#c50918'}}]}
(Pdb) print(r['response'][-1])
{'id': '1989', 'org_id': '2', 'date': '2018-04-12', 'info':
'Indicator Report: Gandcrab Activity Report (Apr 11, 2018)
pySightSight 18-00006073', 'uuid':
'5acfbef6-c1c8-4b88-9b6a-3fb9496a006a', 'published': True,
'analysis': '0', 'attribute_count': '2', 'orgc_id': '21',
'timestamp': '1525964867', 'distribution': '0',
'sharing_group_id': '0', 'proposal_email_lock': False,
'locked': True, 'threat_level_id': '2',
'publish_timestamp': '1523491200', 'disable_correlation':
False, 'extends_uuid': '', 'Org': {'id': '2', 'name':
'TIARE', 'uuid': '5a6a4161-b2e0-46b6-b12e-1bca0a6b037c'},
'Orgc': {'id': '21', 'name': 'iSIGHT', 'uuid':
'5aec9d46-e47c-49f2-aace-06480a670c7f'}, 'EventTag':
[{'id': '8545', 'event_id': '1989', 'tag_id': '279', 'Tag':
{'id': '279', 'name': 'europol-event:brute-force-attempt=',
'colour': '#3F63AD'}}, {'id': '8546', 'event_id': '1989',
'tag_id': '274', 'Tag': {'id': '274', 'name':
'osint:lifetime=ephemeral', 'colour': '#DADCF0'}}, {'id':
'8547', 'event_id': '1989', 'tag_id': '275', 'Tag': {'id':
'275', 'name': 'osint:source-type=block-or-filter-list',
'colour': '#DADCF0'}}, {'id': '8548', 'event_id': '1989',
'tag_id': '3', 'Tag': {'id': '3', 'name': 'tlp:white',
'colour': '#ffffff'}}, {'id': '8549', 'event_id': '1989',
'tag_id': '7', 'Tag': {'id': '7', 'name': 'Type:OSINT',
'colour': '#1eed40'}}, {'id': '8550', 'event_id': '1989',
'tag_id': '276', 'Tag': {'id': '276', 'name':
'inthreat:event-src="feed-osint"', 'colour': '#DADCF0'}},
{'id': '8551', 'event_id': '1989', 'tag_id': '1', 'Tag':
{'id': '1', 'name':
'osint:source-type="block-or-filter-list"', 'colour':
'#004f89'}}, {'id': '32796', 'event_id': '1989', 'tag_id':
'335', 'Tag': {'id': '335', 'name': 'iSight', 'colour':
'#00ace6'}}]}
(Pdb)
(Pdb) sresult
{'response': {'Attribute': [{'id': '1039219', 'event_id':
'15063', 'object_id': '0', 'object_relation': None,
'category': 'Network activity', 'type': 'ip-dst', 'to_ids':
True, 'uuid': '61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5',
'timestamp': '1534881201', 'distribution': '5',
'sharing_group_id': '0', 'comment':
'score:[5.7,2018082119]', 'deleted': False,
'disable_correlation': False, 'value': '1.0.128.131'}]}}
(Pdb) 
$ 
$ 
/attributes/61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5 HTTP/1.1"
302 - "-" "PyMISP 2.4.93 - Python 3.6.4"
$

CURL

$ cat comment.json
{"comment":"foo"}
$ 
$ sh -x ./xforce-update-attr.curl
+ curl -v -k --header 'Authorization: <KEY>' --header 'Accept: application/json' --header 'Content-Type: application/json' -X POST --data @comment.json https://localhost/attributes/61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5
* About to connect() to localhost port 443 (#0)
*   Trying ::1...
* Connected to localhost (::1) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*   subject: E=misp@local.net,CN=redacted...
*   start date: May 21 14:11:11 2018 GMT
*   expire date: May 18 14:11:11 2028 GMT
*   common name: 135.91.54.39
*   issuer: E=misp@local.net,CN=redacted...
> POST /attributes/61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5 HTTP/1.1
> User-Agent: curl/7.29.0
> Host: localhost
> Authorization: <KEY>
> Accept: application/json
> Content-Type: application/json
> Content-Length: 17
>
* upload completely sent off: 17 out of 17 bytes
< HTTP/1.1 302 Found
< Date: Fri, 24 Aug 2018 16:52:25 GMT
< Server: Apache/2.4.6 (Red Hat Enterprise Linux) OpenSSL/1.0.1e-fips mod_wsgi/4.6.2 Python/3.6
< X-Powered-By: PHP/7.1.8
< Set-Cookie: CAKEPHP=ndcqi6arcs3e7vt55fgksluo6s; expires=Sat, 25-Aug-2018 16:52:25 GMT; Max-Age=86400; path=/; secure; HttpOnly
< Location: https://localhost/
< Transfer-Encoding: chunked
< Content-Type: application/json; charset=UTF-8
<
* Connection #0 to host localhost left intact
$ 
$ tail -1 /etc/httpd/logs/misp.local_access.log | fold -w60 -s
::1 - - [24/Aug/2018:16:52:25 +0000] "POST
/attributes/61b97d7a-b13a-4579-bac9-4a6d8ae3ffd5 HTTP/1.1"
302 - "-" "curl/7.29.0"
$

github-germ commented 5 years ago

@iglocska -- curl also fails to update attr comment -- see above https://github.com/MISP/PyMISP/issues/260#issuecomment-41582488

let me know what to explore next. THANKS!!!

github-germ commented 5 years ago

@Rafiot + @iglocska -- would be helpful to make progress on this issue. Thx!

github-germ commented 5 years ago

Being that I need to finish this feature and move forward in our project, it would really helpful if a solution to updating attr comments could be found. Thx a lot!

iglocska commented 5 years ago

Just tested it via curl with both an admin and a normal api enabled user (edited message as it said sync user here before which is incorrect - it was a normal publisher user). Works in both cases. curl command user:

root@iglocska-ThinkPad-T480:/var/www/MISP# curl -v -k --header 'Authorization: my_api_key' --header 'Accept: application/json' --header 'Content-Type: application/json' -X POST --data '{"comment": "foo2"}' http://localhost:5001/attributes/5b7c01b5-ee10-4825-bb7d-2f6f4df74d74

github-germ commented 5 years ago

@iglocska -- thx for testing! My environment is slightly different. My user is 'sync as tagger', and my org and orgc differ. Not sure if that's related to the problem. Have you had a chance to look at the details I posted in https://github.com/MISP/PyMISP/issues/260#issuecomment-415824889

Much appreciated. I can run more tests, grab more debug, etc. with your direction.

iglocska commented 5 years ago

Ah ok, you're trying to edit an event that you didn't create. That is not supported for any function not used by the sync mechanism as it's exclusively meant to be used for synchronisation. Again, as discussed before, if you want to use separate orgc_ids for different events internally - that's fine, but you'll have to use the appropriate users to edit them (= same orgc_id).

github-germ commented 5 years ago

Why can I update the event iteself, and tags for the same attr within that event running the same process, same code, same user?

github-germ commented 5 years ago

Were there changes made to support this setup originally as in https://github.com/MISP/PyMISP/issues/239 that need to be applied for update_attribute?

iglocska commented 5 years ago

You can update the event itself because you go through a function that is used by the sync (/events/edit). The attribute level edit is never used by the sync. Basically you can abuse the side effect of the sync mechanism to edit events via /events/edit with a sync user, but that's about it.

github-germ commented 5 years ago

Let me make sure I understand. If we need to arrange org/orgc differently as we do with our events that are ingested via PyMISP, then we cannot edit the comment of attributes within, but we can tag/untag those attributes? If this is true, then is our use case flawed or can editing attributes be enhanced to support this? Thx again!

iglocska commented 5 years ago

Again, let's go back to step one, but we've been over this last time:

You have data coming from different feed providers. You want to maintain their orgc - this is against the conventions in MISP as we've discussed, but it's fine, if that's what you want there's a way to do it.

Please make sure that you create a user corresponding to the organisations of each of those feeds. Make sure that you use the appropriate user when pushing data from one of your feeds (meaning the user associated with the feed, the orgc). You can easily use this user to update attributes in the event without any hurdles. No need to use sync users and abuse the sync mechanism.

github-germ commented 5 years ago

@iglocska ... I think I am following those "rules" for our use case. Are you saying that the root cause of my inability to change the attr comment is that the user has a sync role, and if I remove the sync role, I will be able to create/edit events and attributes within those events?

If so, here's the current user's role. What should I change?

screen shot 2018-08-29 at 9 27 17 am

iglocska commented 5 years ago

If you remove the sync role you will not be allowed to break the ownership of data like you do currently.

If I create an event, the orgc will be my user's org. I can edit it any way I want.
If I see an event that has a different orgc (= not my event), I cannot edit it.

Rule #2 has an exception for sync users using some API calls (/events/edit) to make the synchronisation mechanism work.

github-germ commented 5 years ago

If a sync user can edit an event, then why not allow that user to edit an attr contained in that event? Currently, that user can tag those same attributes.

If this cannot change, then is there alternative method to accomplish my use case? We want to store a history of the changes in score with that date (yyyymmddhh) for the life cycle of attributes received via an hourly feed, e.g. (5.7,2018010108),(3.4,2018042713), where we append as it changes. The attr.comment text (64k chars) seemed like a good place.

Open to other ideas. Thanks!!

rommelfs commented 5 years ago

I think that been answered in the last posts of @iglocska: it is an abuse of the sync mechanism.

The alternative way has also been outlined. Quote from earlier: "Please make sure that you create a user corresponding to the organisations of each of those feeds."

Please outline what is blocking you from doing it the correct way. Maybe we are missing an important aspect. Thanks.

github-germ commented 5 years ago

@iglocska + @rommelfs -- thanks to both of you for persisting through me being dense headed. indeed calling update_attribute as a user that's a member of the orgc that created the event and attribute within does work.

fyi prior references:

this now works for me. if i am still doing something "wrong" to get our desired use case, please let me know. THANKS AGAIN!

the TRICK is that with our use case (which i understand you deem unusual) is switching users. our initial ingestion and hourly updates are all done by one app: first creating and adding attributes, and then updating attributes and tags as needed as more info is aquired in a future hourly feed. hence, the TRICK is as follows (was confusing to me that i could tag/untag an attribute in an event created by a user in org, but to update the comment in that same attribute i need to be a user in orgc):

for the initial event_add and subsequent update_event that have add_attribute and tag/untag, the app talks to PyMISP as a user in org.
however, note that add_event creates with differing org (set to our internal analyst org name, i.e. "we" own this event even though it comes from a feed) and orgc (set to represent the feed creater, i.e. not "us").
the TRICK is then when there's a need to call update_attribute (which is via the uuid of the attribute, and not changed as part of a call to update_event) to revise a comment in a future feed update to an existing attr in an existing event that same single app needs to use a PyMISP connection for user in orgc (and not the usual one in org).

MISP / PyMISP

why isn't change_comment() working for me? #260