Uplink crashes on python3.9 when trying to serialize pydantic model

liiight commented 3 years ago

Describe the bug Uplink crashes on python3.9 when trying to serialize model

To Reproduce This is the uplink class and method:

@tpa_error
@uplink_retry
class TPAService(FlapiBase):
    @uplink.returns.json(key=('data', 'appInstances', 'items'))
    @uplink.json
    @uplink.post('app-governance-graphql/graphql')
    def get_app_instance(self, **body: uplink.Body) -> List[AppInstance]:
        """Send a query Query to Third Party App's GraphQL API"""

This is the crash:

Traceback (most recent call last):
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/tests/casb/sit/test_sit.py", line 98, in test_sit_spark_invalid_log_file
tenant_app = verify_tenant_app_exists_in_tpa(app_name=APP_FOR_SIT_SPARK, tenant_id=tenant_id)
File "</home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/decorator.py:decorator-gen-64>", line 2, in _
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/shield/utils/retrier.py", line 38, in retry_decorator
return retry_call(f, fargs, fkwargs, exceptions, tries, delay, max_delay, backoff, jitter, log)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/retry/api.py", line 101, in retry_call
return __retry_internal(partial(f, *args, **kwargs), exceptions, tries, delay, max_delay, backoff, jitter, logger)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/retry/api.py", line 33, in __retry_internal
return f()
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/tests/fixtures/tpa.py", line 140, in _
tenant_app_instance = tpa_service().get_tenant_app_instance(query=query, variables=variables)[0]
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/builder.py", line 95, in __call__
self._request_definition.define_request(request_builder, args, kwargs)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/commands.py", line 287, in define_request
self._method_handler.handle_builder(request_builder)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/decorators.py", line 62, in handle_builder
annotation.modify_request(request_builder)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/returns.py", line 64, in modify_request
converter = request_builder.get_converter(
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/helpers.py", line 96, in get_converter
return self._converter_registry[converter_key](*args, **kwargs)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/converters/__init__.py", line 54, in __call__
converter = self._converter_factory(*args, **kwargs)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/converters/__init__.py", line 114, in chain
converter = func(factory)(*args, **kwargs)
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/models.py", line 42, in __call__
if self._is_relevant(type_, *args, **kwargs):
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/models.py", line 34, in _is_relevant
return utils.is_subclass(
File "/home/jenkins/workspace/execute_shield_pull_request_sit_test_us1/.tox/py39/lib/python3.9/site-packages/uplink/utils.py", line 81, in is_subclass
return inspect.isclass(cls) and issubclass(cls, class_info)
File "/usr/lib/python3.9/abc.py", line 123, in __subclasscheck__
return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

Expected behavior Not to crash

Additional context This results from an apparant change in issubclass() in python 3.9. I debugged this in 3.6 (where it's working):

>>> cls
PyDev console: starting.
typing.List[shield.models.tpa.AppInstance]
>>> inspect.isclass(cls)
True
>>> issubclass(cls, class_info)
False

And 3.9:

>>> cls
PyDev console: starting.
typing.List[shield.models.tpa.AppInstance]
>>> inspect.isclass(cls)
True
>>> issubclass(cls, class_info)
Traceback (most recent call last):
  File "/Applications/PyCharm.app/Contents/plugins/python/helpers/pydev/_pydevd_bundle/pydevd_exec2.py", line 3, in Exec
    exec(exp, global_vars, local_vars)
  File "<input>", line 1, in <module>
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class

Apparently issubclass() does not consider this to be a class in python 3.9, haven't dug deep in its changelog to try and figure out why

liiight commented 3 years ago

This seems to be related to https://github.com/samuelcolvin/pydantic/issues/1298 Updating to latest version of pydantic and checking again

edit:

still occurs with pydantic==1.8.2 and uplink==0.9.4

liiight commented 3 years ago

I used this workaround to patch this in my testing suite:

@pytest.fixture(autouse=True)
def uplink_patch(monkeypatch, logger):
    def is_subclass_patch(cls, class_info):
        with suppress(TypeError):
            return inspect.isclass(cls) and issubclass(cls, class_info)
    logger.debug('Due to uplink issue https://github.com/prkumar/uplink/issues/234 we need to monkeypatch uplink')

    monkeypatch.setattr('uplink.utils.is_subclass', is_subclass_patch)

prkumar commented 2 years ago

Hi, @liiight! Sorry that I'm just getting around to this issue now. If this is still reproducible, could you provide a self-contained, simple example to help me reproduce. Something similar to this would be very helpful.

liiight commented 2 years ago

@prkumar this seems to be a little more complicated than I initially investigated, as a self contained example seems to work just fine:

import uplink
from pydantic import BaseModel

class HTTPBinData(BaseModel):
    foo: dict

class HTTPBinClient(uplink.Consumer):

    @uplink.returns.json(key=('json', 'data'))
    @uplink.json
    @uplink.post('/anything')
    def get_json(self, data: uplink.Field) -> list[HTTPBinData]:
        pass

c = HTTPBinClient(base_url='https://httpbin.org')
print(c.get_json(data=[{'foo': {}}]))

I believe this has something to do with pre-registered converters. I'll continue to investigate

bal-stan commented 1 year ago

Is there any progress on this?

Using:

Uplink 0.9.7
Pydantic 1.10.2
Python 3.10.4

I have a similar problem:

import typing as t

import typing_extensions as te
from pydantic import BaseModel
from uplink import Consumer, get

class Model(BaseModel):
    title: str
    description: str

class Client(Consumer):
    @get("some/endpoint")
    def get_data(self: te.Self) -> t.List[Model]:
        """List data."""

Then:

>>> a = Client("http://localhost:5000")
>>> a.get_data()
Traceback (most recent call last):
  File "pydantic/main.py", line 522, in pydantic.main.BaseModel.parse_obj
ValueError: dictionary update sequence element #0 has length 3; 2 is required

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/builder.py", line 106, in __call__
    return execution.start(
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 97, in start
    return self._io.execute(self)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 122, in execute
    return self._io.execute(executable)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 122, in execute
    return self._io.execute(executable)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 122, in execute
    return self._io.execute(executable)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/blocking_strategy.py", line 31, in execute
    return executable.execute()
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 93, in execute
    return self.state.execute(self)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/state.py", line 36, in execute
    return execution.before_request(self._request)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 56, in before_request
    return self.execute()
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 93, in execute
    return self.state.execute(self)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/state.py", line 105, in execute
    return execution.send(
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 73, in send
    return self._io.invoke(self._client.send, (request,), {}, callback)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 116, in invoke
    return self._io.invoke(func, args, kwargs, callback)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 116, in invoke
    return self._io.invoke(func, args, kwargs, callback)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 116, in invoke
    return self._io.invoke(func, args, kwargs, callback)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/blocking_strategy.py", line 21, in invoke
    return callback.on_success(response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/state.py", line 96, in on_success
    return self._context.execute()
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 93, in execute
    return self.state.execute(self)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/state.py", line 123, in execute
    return execution.after_response(self._request, self._response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 62, in after_response
    return self.execute()
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 93, in execute
    return self.state.execute(self)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/state.py", line 221, in execute
    return execution.finish(self._response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 79, in finish
    return self._io.finish(response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 125, in finish
    return self._io.finish(response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 125, in finish
    return self._io.finish(response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 143, in finish
    return self._invoke(
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 133, in _invoke
    return self._io.invoke(func, args, kwargs, FinishingCallback(self._io))
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/blocking_strategy.py", line 19, in invoke
    return callback.on_failure(type(error), error, tb)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/execution.py", line 108, in on_failure
    return self._io.fail(exc_type, exc_val, exc_tb)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/interfaces.py", line 300, in fail
    compat.reraise(exc_type, exc_val, exc_tb)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/six.py", line 719, in reraise
    raise value
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/io/blocking_strategy.py", line 16, in invoke
    response = func(*arg, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/clients/requests_.py", line 53, in apply_callback
    return callback(response)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/builder.py", line 47, in wrapper
    return func(self._consumer, *args, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/hooks.py", line 21, in wrapper
    return hook(*args, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/returns.py", line 39, in __call__
    return self._strategy(*args, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/converters/typing_.py", line 33, in convert
    return [self._elem_converter(value)]
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/converters/interfaces.py", line 6, in __call__
    return self.convert(*args, **kwargs)
  File "/path/to/python/virtual/env/lib/python3.10/site-packages/uplink/converters/pydantic_.py", line 48, in convert
    return self._model.parse_obj(data)
  File "pydantic/main.py", line 525, in pydantic.main.BaseModel.parse_obj
pydantic.error_wrappers.ValidationError: 1 validation error for Model
__root__
  Model expected dict not list (type=type_error)

Looks like Uplink does not recognise the data being a list and passes it directly to pydantic, which crashes because it expects a dictionary as the root.

For clarity here is example data:

[
  {
    "title": "Title 1",
    "description": "Description 1"
    },
    {
    "title": "Title 2",
    "description": "Description 2"
    }
]

bobojobo commented 1 year ago

@bal-stan, I think you can fix your problem with the returns.json decorator:

import typing as t

import typing_extensions as te
from pydantic import BaseModel
from uplink import Consumer, get, returns

class Model(BaseModel):
    title: str
    description: str

class Client(Consumer):
    @returns.json()
    @get("some/endpoint")
    def get_data(self: te.Self) -> t.List[Model]:
        """List data."""

prkumar / uplink

Uplink crashes on python3.9 when trying to serialize pydantic model #234