USApplicationManager is not JSON serializable

[IshanMi]

I tried adding the celery.task decorator to the download_patents function in patent_fetcher.py, but ran into an error:

[2021-06-16 19:48:08,433] ERROR in app: Exception on /search/lens [GET]
Traceback (most recent call last):
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 42, in _reraise_errors
    yield
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 213, in dumps
    payload = encoder(data)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\utils\json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\__init__.py", line 398, in dumps
    return cls(
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\utils\json.py", line 58, in default
    return super().default(o)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 272, in default
    raise TypeError('Object of type %s is not JSON serializable' %
TypeError: Object of type USApplicationManager is not JSON serializable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\app.py", line 2447, in wsgi_app
    response = self.full_dispatch_request()
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\app.py", line 1952, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\app.py", line 1821, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\_compat.py", line 39, in reraise
    raise value
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\app.py", line 1950, in full_dispatch_request
    rv = self.dispatch_request()
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\flask\app.py", line 1936, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "C:\Users\ronal\PycharmProjects\reverse-image-search-patent\app.py", line 103, in patent_search
    drawing_files = download_patents.delay(patent_results, destination=dir_name)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\celery\app\task.py", line 421, in delay
    return self.apply_async(args, kwargs)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\celery\app\task.py", line 561, in apply_async
    return app.send_task(
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\celery\app\base.py", line 749, in send_task
    amqp.send_task_message(P, name, message, **options)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\celery\app\amqp.py", line 523, in send_task_message
    ret = producer.publish(
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\messaging.py", line 166, in publish
    body, content_type, content_encoding = self._prepare(
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\messaging.py", line 254, in _prepare
    body) = dumps(body, serializer=serializer)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 213, in dumps
    payload = encoder(data)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\contextlib.py", line 135, in __exit__
    self.gen.throw(type, value, traceback)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 46, in _reraise_errors
    reraise(wrapper, wrapper(exc), sys.exc_info()[2])
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\exceptions.py", line 21, in reraise
    raise value.with_traceback(tb)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 42, in _reraise_errors
    yield
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\serialization.py", line 213, in dumps
    payload = encoder(data)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\utils\json.py", line 68, in dumps
    return _dumps(s, cls=cls or _default_encoder,
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\__init__.py", line 398, in dumps
    return cls(
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 296, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 378, in iterencode
    return _iterencode(o, 0)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\kombu\utils\json.py", line 58, in default
    return super().default(o)
  File "C:\Users\ronal\Anaconda3\envs\reverse_image_search\lib\site-packages\simplejson\encoder.py", line 272, in default
    raise TypeError('Object of type %s is not JSON serializable' %
kombu.exceptions.EncodeError: Object of type USApplicationManager is not JSON serializable
127.0.0.1 - - [16/Jun/2021 19:48:08] "GET /search/lens?author=&status= HTTP/1.1" 500 -

[bbelderbos]

Yes this was because it seems you cannot pass a list of objects to a task (here a list of US patent objects): https://stackoverflow.com/questions/50666990/how-to-send-django-objects-to-celery-tasks https://stackoverflow.com/questions/49373825/kombu-exceptions-encodeerror-user-is-not-json-serializable

So I had to redefine the signature of the function a bit (not happy yet about it but hey)

More importantly I also needed to restructure the code, particularly moving app.py into src/ and making the app object more reusable. I will shoot you a PR and let you decide if you want to merge it or not.

Now the search returns much faster and Celery does the downloading asynchronously:

[bbelderbos]

I ran this like: celery -A src.patent_fetcher worker --loglevel=DEBUG

Code changes: https://github.com/IshanMi/reverse-image-search-patent/pull/13#pullrequestreview-697542122