Description

Using bravado's SwaggerClient in a multiprocessing protocol causes a RecursionError.

Minimal example

from multiprocessing import Pool

from bravado.client import SwaggerClient

API_DEFINITIONS = "https://klifs.net/swagger/swagger.json"
CLIENT = SwaggerClient.from_url(API_DEFINITIONS, config={"validate_responses": False})

pool = Pool(processes=2)
type_list = pool.map(type, [CLIENT, CLIENT])
print(type_list)
pool.close()
pool.join()

Error

Process ForkPoolWorker-2:
Traceback (most recent call last):
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/multiprocessing/pool.py", line 114, in worker
    task = get()
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/multiprocessing/queues.py", line 358, in get
    return _ForkingPickler.loads(res)
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/site-packages/bravado/client.py", line 170, in __getattr__
    return self._get_resource(item)    
return self._get_resource(item)
  File "/home/dominique/.local/miniconda/envs/kissim/lib/python3.8/site-packages/bravado/client.py", line 148, in _get_resource
    resource = self.swagger_spec.resources.get(item)

  # Repeating the line 170 and 148 errors over and over again

RecursionError: maximum recursion depth exceeded

Solution

Fix serialization errors in bravado through subclassing (defining __getstate__ and __setstate).

Todos

[x] Add SerializableSwaggerClient class (subclassing SwaggerClient and defining __getstate__ and __setstate__
[x] Move KLIFS_API_DEFINITIONS and KLIFS_CLIENT from utils to remote module; update all associated imports

Questions

[ ] Maybe this serialization protocol could be added directly to bravado's SwaggerClient? https://github.com/Yelp/bravado/issues/471

Status

[ ] Ready to go

codecov-io commented 3 years ago

Codecov Report

Merging #68 (a7f9637) into master (9eec84d) will decrease coverage by 0.42%. The diff coverage is 76.92%.

jaimergp commented 3 years ago

Thanks for the thorough description! 🚀

dominiquesydow commented 3 years ago

Can I go ahead and merge this PR, @jaimergp?