ZMQ Dispatcher Error: JSONDecodeError("Expecting value")

[faustus25]

Had the same issue with the zmq_subscriber script now the zmq_dispatcher.py one has this issue:

echo 'source /usr/local/src/misp-dashboard/DASHENV/bin/activate; /usr/bin/python3 /usr/local/src/misp-dashboard/zmq_dispatcher.py' | /bin/bash

Traceback (most recent call last):
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 299, in <module>
    main(args.sleeptime)
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 274, in main
    process_log(zmqName, content)
  File "/usr/local/src/misp-dashboard/zmq_dispatcher.py", line 254, in process_log
    jsonevent = json.loads(eventdata)
  File "/usr/lib/python3.5/json/__init__.py", line 319, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3.5/json/decoder.py", line 339, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/lib/python3.5/json/decoder.py", line 357, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

How are you running the "start_all.sh" script as virtualenv when passed to crontab at reboot or rc.local?

[mokaddem]

Of course, you need to be in a virtualenv. I usually start it manually but using rc.local should work.

[faustus25]

Can you specify how it runs in rc.local so I can test it (did you simply add the start_all script there)?

I notice that the start_all.sh script crashes (either zmq_subscriber or zmq_dispatcher with the error in my previous post) if the event has 1000+ attributes in it specifically if the event is from a Default or Custom Feed. Some of my Custom Feeds have 100,000+ attributes: e.g. hosts-file.net - hphost - malwarebytes feed 165060 attributes

[mokaddem]

Exactly: bash /var/www/misp-dashboard/start_all.sh Adding that much attributes should not be a problem, it will be slow but it should eventually work. Just a question, are you feeding the dashboard manually or are you simply letting MISP feed it through the standard ZMQ?

[faustus25]

MISP feed is automatically feeding to those events to the dashboard, it just doesn't handle the attribute heavy events well.

Latest issue I have running the zmq scripts:

redis.exceptions.ResponseError: MISCONF Redis is configured to save RDB snapshots, but is currently not able to persist on disk. Commands that may modify the data set are disabled. Please check Redis logs for details about the error.

Have you encountered this before?

Seen this as a possible workaround, any recommendations?:

redis-cli
CONFIG SET dir /tmp/some/directory/other/than/var
CONFIG SET dbfilename temp.rdb
BGSAVE

[mokaddem]

Not enough RAM or disk space? Are you running the dashboard inside the provided VM? Have you allocated enough memory?

[faustus25]

The error has not appeared since the redis-cli config change. Will close this for the moment and monitor for any reoccurence.

[faustus25]

Error reoccurred stopping zmq dispatcher and subscriber so disabled RDB snapshots

$ redis-cli
> config set stop-writes-on-bgsave-error no

This comes with a previso: "this is recommended only if you don't have RDB snapshots enabled or if you don't care about data persistence in the snapshots."

@mokaddem: Have you had a similar issue and remediated similarly?

[mokaddem]

Hello, Unfortunately, I never had this issue. But I never stressed the dashboard with that much data, especially if it runs on the VM.

[faustus25]

I will look into increasing the memory for the VM.

In terms of feeds for MISP (both default and custom) these are the top 5 which put a huge load on the Redis server and as I have as fixed events when the zmq_dispatcher has to run through all these attribute heavy fixed events the zmq scripts fail:

All Current Dominas OSINT Bambenek: 2+million attributes blocklist.de : 460,000+ attributes ci-badguys: 383,000+ attributes antisource: 297,000+ attributes Alienvault Rep: 209,000+ attributes

You must create a new event for each pull of these feeds or not have them enabled possibly?

Fixed events with a huge volume of attributes show as above may not be recommended for the dashboard, curious if others have the same experience.

[shaykster]

Have a similar error - anyone seen the issue below;

(DASHENV) misp@misp:~/misp-dashboard-master$ (DASHENV) misp@misp:~/misp-dashboard-master$ ./zmq_dispatcher.py Traceback (most recent call last): File "./zmq_dispatcher.py", line 299, in main(args.sleeptime) File "./zmq_dispatcher.py", line 274, in main process_log(zmqName, content) File "./zmq_dispatcher.py", line 256, in process_log dico_action[topic](zmq_name, jsonevent) File "./zmq_dispatcher.py", line 208, in handler_attribute trendings_helper.addTrendingCateg(categName, timestamp) File "/home/misp/misp-dashboard-master/helpers/trendings_helper.py", line 50, in addTrendingCateg self.addGenericTrending(self.keyCateg, categName, timestamp) File "/home/misp/misp-dashboard-master/helpers/trendings_helper.py", line 43, in addGenericTrending self.serv_redis_db.zincrby(keyname, to_save, 1) File "/home/misp/misp-dashboard-master/DASHENV/lib/python3.6/site-packages/redis/client.py", line 2281, in zincrby return self.execute_command('ZINCRBY', name, amount, value) File "/home/misp/misp-dashboard-master/DASHENV/lib/python3.6/site-packages/redis/client.py", line 755, in execute_command return self.parse_response(connection, command_name, **options) File "/home/misp/misp-dashboard-master/DASHENV/lib/python3.6/site-packages/redis/client.py", line 768, in parse_response response = connection.read_response() File "/home/misp/misp-dashboard-master/DASHENV/lib/python3.6/site-packages/redis/connection.py", line 638, in read_response raise response redis.exceptions.ResponseError: value is not a valid float (DASHENV) misp@misp:~/misp-dashboard-master$ (DASHENV) misp@misp:~/misp-dashboard-master$

And logs show this;

(DASHENV) misp@misp:/var/www/misp-dashboard/logs$ tail -f logs.log INFO:zmq_dispatcher:Handling attribute WARNING:helpers.geo_helper:can't resolve ip INFO:zmq_dispatcher:Handling attribute WARNING:helpers.geo_helper:can't resolve ip

[mokaddem]

Hello @shaykster , I think your issue is more related to this post #76. Can you try the fix provided there and comment the result on issue #76 ? Thanks!