search

remg427 / misp42splunk

A Splunk app to use MISP in background
GNU Lesser General Public License v3.0
109 stars 30 forks source link

"last" parameter in command string does not work #23

Closed ghost closed 5 years ago

ghost commented 5 years ago

specifying an eventid does work, however when using the "last" parameter command the error of "object has no attribute keys" is displayed.

any ideas?

The below syntax throws error:

|mispgetioc last=1d

image

Thanks!!

ghost commented 5 years ago

search log error messages...

08-02-2018 17:51:37.609 INFO  ChunkedExternProcessor - Running process: /opt/splunk/bin/python /opt/splunk/etc/apps/misp42splunk/bin/mispgetioc.py
08-02-2018 17:51:37.802 INFO  DispatchThread - BatchMode: allowBatchMode: 0, conf(1): 1, timeline/Status buckets(0):300, realtime(0):0, report pipe empty(0):0, reqTimeOrder(0):0, summarize(0):0, statefulStreaming(0):0
08-02-2018 17:51:37.802 INFO  DispatchThread - Storing only 1000 events per timeline buckets due to limits.conf max_events_per_bucket setting.
08-02-2018 17:51:37.802 INFO  DispatchThread - required fields list to add to remote search = 
08-02-2018 17:51:37.802 INFO  DispatchCommandProcessor - summaryHash=NS3d9d854163f8f07a summaryId=D632C557-569F-4CF1-B118-2F0CECDDCA75_search_admin_NS3d9d854163f8f07a remoteSearch=
08-02-2018 17:51:37.816 INFO  DispatchThread - Setting summary_mode=NONE after optimization
08-02-2018 17:51:37.816 INFO  DispatchThread - SrchOptMetrics FinalEval=434
08-02-2018 17:51:37.816 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.816 INFO  UserManager - Done setting user context: admin -> admin
08-02-2018 17:51:37.817 INFO  UserManager - Unwound user context: admin -> admin
08-02-2018 17:51:37.817 INFO  DistributedSearchResultCollectionManager - Stream search: 
08-02-2018 17:51:37.817 INFO  DispatchThread - Disk quota = 10485760000
08-02-2018 17:51:37.817 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.817 INFO  UserManager - Done setting user context: NULL -> admin
08-02-2018 17:51:37.817 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.817 INFO  UserManager - Done setting user context: NULL -> admin
08-02-2018 17:51:37.817 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.817 INFO  UserManager - Done setting user context: NULL -> admin
08-02-2018 17:51:37.817 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.817 INFO  UserManager - Done setting user context: NULL -> admin
08-02-2018 17:51:37.817 INFO  UserManager - Setting user context: admin
08-02-2018 17:51:37.818 INFO  UserManager - Done setting user context: NULL -> admin
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr: AttributeError at "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/internals.py", line 541 : 'unicode' object has no attribute 'keys'
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr: Traceback:
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/search_command.py", line 780, in _process_protocol_v2
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:     self._execute(ifile, None)
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/reporting_command.py", line 108, in _execute
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:     SearchCommand._execute(self, ifile, getattr(self, self.phase))
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/search_command.py", line 848, in _execute
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:     self._record_writer.write_records(process(self._records(ifile)))
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/internals.py", line 522, in write_records
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:     write_record(record)
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:   File "/opt/splunk/etc/apps/misp42splunk/bin/splunklib/searchcommands/internals.py", line 541, in _write_record
08-02-2018 17:51:41.389 ERROR ChunkedExternProcessor - stderr:     self._fieldnames = fieldnames = list(record.keys())
ghost commented 5 years ago

update: found today that (h)ours and (m)inutes work; NOT (d)ays..

Not sure if it depends on MISP instance uptime or actual event data date fields.. (?)

Also noticed that when feeds are being fetched, the "last" command does not work.... This may be what my issue is as I test while a complete cache and fetch of all feeds are executing.

confirmation?

ghost commented 5 years ago

Update: all works just fine with MISP version latest commit - build 2.4.93

image

ghost commented 5 years ago

Its dead again in v.2.4.94.... Any ideas what changed thats killing the 'last" command with PyMISp ??

remg427 commented 5 years ago

I'll check over the weekend what's wrong

ghost commented 5 years ago

Thank you !!! Ive been digging and cant find it. I even looked at the MISP v.2.4.94 PyMISP changelog.....

ghost commented 5 years ago

Update: Well now it seems to work.

What changed....

Now I selected a whole bunch of feeds and enabled them with caching.

What makes the last parameter only pull data from specific feeds or is it a certain amount or kind of data?

remg427 commented 5 years ago

last parameter calls PyMISP function misp.search(last=l) On my test platform, I have no feeds only sync events the error comes from no result returned. I am going to improve error handling to return a message if there is no data (and if I have time I will improve debug mode)

ghost commented 5 years ago

Yes, I figured out when it had no results. and a long error I had to lessen the time duration for some reason.

Also, if search executed just fine with no results there was no error and no results.

The problem here is I think with how MISP handles data or what type of data is in certain events.

On Sat, Sep 1, 2018, 3:26 AM remg427 notifications@github.com wrote:

last parameter calls PyMISP function misp.search(last=l) the error comes from no esult returned. I am going to improve error handling to return a message if there is no data (and if I have time I will improve debug mode)

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/remg427/misp42splunk/issues/23#issuecomment-417842619, or mute the thread https://github.com/notifications/unsubscribe-auth/AOjHrm0s0cdV1wlC_jgHro3C7SZJ7lEiks5uWkStgaJpZM4VnNso .

remg427 commented 5 years ago

just commit a new version with improved logging when using mispgetioc. If there is an error in the Python3 section, it should return a dict with type:error value:an error message. Note that PyMISP should be version 2.4.95 or above. I still have to work a bit on getting the errors from python3 script back into splunk python script.