kombu.exceptions.EncodeError: Object of type module is not JSON serializable

xiaohuajiao commented 6 years ago

When i try to run fexm fuzz ./example/crashing2.json or fexm fuzz ./example/top500.json, it reports following error.

Here is exception info, i print out the object which trigger this exception.

[2018-08-23 13:06:35,889: WARNING/MainProcess] <class 'tuple'>
[2018-08-23 13:06:35,890: WARNING/MainProcess] (('libgeotiff', 'pacmanfuzzer', '/media/vincent/VincentData/crashes/crashing', '/seeds', 1200, True, '1000+', False, {'seeds': '/seeds', 'out_dir': '/media/vincent/VincentData/crashes/crashing', 'build_folder': '/fexm/examples', 'fuzz_duration': 20, 'base_image': 'pacmanfuzzer', 'max_build_size': 1000000000000, 'max_size': 10000000000, 'use_asan': True, 'force': False, 'exec_timeout': '1000+', 'fuzzing_cores_per_binary': 7, 'fuzz_manager': 'pacman', 'name': 'package', 'packages_file': '/fexm/examples/crashing_packages3.txt', 'MAX_SIZE': 10000000000, 'USE_ASAN': True, 'qemu': False, 'json_name': 'crashing3.json', 'manager': <module 'fuzz_managers.pacman' from '/fexm/fuzz_managers/pacman.py'>, 'fuzz_func': functools.partial(<function fuzz at 0x7f8313d6dd90>, {'seeds': '/seeds', 'out_dir': '/media/vincent/VincentData/crashes/crashing', 'build_folder': '/fexm/examples', 'fuzz_duration': 20, 'base_image': 'pacmanfuzzer', 'max_build_size': 1000000000000, 'max_size': 10000000000, 'use_asan': True, 'force': False, 'exec_timeout': '1000+', 'fuzzing_cores_per_binary': 7, 'fuzz_manager': 'pacman', 'name': 'package', 'packages_file': '/fexm/examples/crashing_packages3.txt', 'MAX_SIZE': 10000000000, 'USE_ASAN': True, 'qemu': False, 'json_name': 'crashing3.json', 'manager': <module 'fuzz_managers.pacman' from '/fexm/fuzz_managers/pacman.py'>, 'fuzz_func': ...})}), {}, {'callbacks': None, 'errbacks': None, 'chain': None, 'chord': None})
[2018-08-23 13:06:35,896: WARNING/MainProcess] Exception in thread fuzz:
Traceback (most recent call last):
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 222, in dumps
    payload = encoder(data)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/utils/json.py", line 69, in dumps
    **dict(default_kwargs, **kwargs))
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)
TypeError: Object of type module is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/usr/lib/python3.6/threading.py", line 864, in run
    self._target(*self._args, **self._kwargs)
  File "/fexm/fuzz_managers/pacman.py", line 177, in fuzz
    return PacmanFuzzer(config).fuzz()
  File "/fexm/fuzz_managers/pacman.py", line 167, in fuzz
    results = jobs.apply_async()
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/canvas.py", line 978, in apply_async
    args=args, kwargs=kwargs, **options))
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/canvas.py", line 1054, in _apply_tasks
    **options)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/canvas.py", line 218, in apply_async
    return _apply(args, kwargs, **options)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/app/task.py", line 535, in apply_async
    **options
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/app/base.py", line 745, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/celery/app/amqp.py", line 552, in send_task_message
    **properties
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/messaging.py", line 169, in publish
    compression, headers)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/messaging.py", line 252, in _prepare
    body) = dumps(body, serializer=serializer)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 222, in dumps
    payload = encoder(data)
  File "/usr/lib/python3.6/contextlib.py", line 99, in __exit__
    self.gen.throw(type, value, traceback)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 54, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/vine/five.py", line 178, in reraise
    raise value.with_traceback(tb)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 50, in _reraise_errors
    yield
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/serialization.py", line 222, in dumps
    payload = encoder(data)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/utils/json.py", line 69, in dumps
    **dict(default_kwargs, **kwargs))
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/__init__.py", line 399, in dumps
    **kw).encode(obj)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/kombu/utils/json.py", line 59, in default
    return super(JSONEncoder, self).default(o)
  File "/home/vagrant/.virtualenvs/FExM/lib/python3.6/site-packages/simplejson/encoder.py", line 273, in default
    o.__class__.__name__)

viniul commented 6 years ago

Hi,

seems like the example folder contains some invalid json files. I will investigate and get back to you asap.

xiaohuajiao commented 6 years ago

in fuzz_manager/pacman.py

            tasks.append(run_eval.s(package, self.docker_image,
                                    os.path.realpath(os.path.join(os.getcwd() + "/", self.configuration_dir)),
                                    os.path.realpath(os.path.join(os.getcwd() + "/", self.seeds)), self.fuzz_duration,
                                    self.use_asan,
                                    self.exec_timeout, force_qemu, self.config_dict))

self.config_dict is not json serializable, because value of key manager is a module object. Take a deep look a run_eval definition, we may pass a tmp dict to run_eval, like this.

            tmp_dict = {"fuzzing_cores_per_binary":self.config_dict.get("fuzzing_cores_per_binary")}
            tasks.append(run_eval.s(package, self.docker_image,                                      
                                    os.path.realpath(os.path.join(os.getcwd() + "/", self.configuration_dir)),                                                                                              
                                    os.path.realpath(os.path.join(os.getcwd() + "/", self.seeds)), self.fuzz_duration,                    
                                    self.use_asan,                                                   
                                    self.exec_timeout, force_qemu, # self.config_dict
                                    tmp_dict                                                         
                                    ))

viniul commented 6 years ago

Hi,

the issue should be fixed in the current upstream branch. Thank you very much for your help.

Best, Vincent

fgsect / fexm

kombu.exceptions.EncodeError: Object of type module is not JSON serializable #2