search

aws-powertools / powertools-lambda-python

A developer toolkit to implement Serverless best practices and increase developer velocity.
https://docs.powertools.aws.dev/lambda/python/latest/
MIT No Attribution
2.83k stars 391 forks source link

Bug: custom serializer isn't being used by nested dict #4658

Closed LironEr closed 2 months ago

LironEr commented 3 months ago

Expected Behaviour

The custom serializer should be passed to all sub-functions

Current Behaviour

The custom serializer is not being called, it's passed only on the first jsonable_encoder run. https://github.com/aws-powertools/powertools-lambda-python/blob/develop/aws_lambda_powertools/event_handler/openapi/encoders.py#L112

Code snippet

from aws_lambda_powertools.event_handler.openapi.encoders import jsonable_encoder
from bson import Decimal128

def custom_serializer(obj):
    if isinstance(obj, Decimal128):
        return int(obj.to_decimal())

    return obj

# work
print(jsonable_encoder(Decimal128('123'), custom_serializer=custom_serializer))

# doesn't work
print(jsonable_encoder({'dec': Decimal128('123')}, custom_serializer=custom_serializer))

Possible Solution

pass custom_serializer to all functions, as this is a recursive function as mentioned in https://github.com/aws-powertools/powertools-lambda-python/issues/3892#issuecomment-1983683318

Steps to Reproduce

run the code snippet (:

Powertools for AWS Lambda (Python) version

2.40.1

AWS Lambda function runtime

3.11

Packaging format used

PyPi

Debugging logs

Traceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 279, in _dump_other
    data = dict(obj)
           ^^^^^^^^^
TypeError: 'Decimal128' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 283, in _dump_other
    data = vars(obj)
           ^^^^^^^^^
TypeError: vars() argument must have __dict__ attributeTraceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 279, in _dump_other
    data = dict(obj)
           ^^^^^^^^^
TypeError: 'Decimal128' object is not iterable

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 283, in _dump_other
    data = vars(obj)
           ^^^^^^^^^
TypeError: vars() argument must have __dict__ attribute

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 147, in jsonable_encoder
    return _dump_other(
           ^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 286, in _dump_other
    raise ValueError(errors) from e
ValueError: [TypeError("'Decimal128' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "xxxxxxxxxxxxxx/c.py", line 19, in <module>
    print(jsonable_encoder({'dec': Decimal128('123')}, custom_serializer=custom_serializer))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 113, in jsonable_encoder
    return _dump_dict(
           ^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 226, in _dump_dict
    encoded_value = jsonable_encoder(
                    ^^^^^^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 157, in jsonable_encoder
    raise SerializationError(
aws_lambda_powertools.event_handler.openapi.exceptions.SerializationError: ('Unable to serialize the object 123 as it is not a supported type. Error details: [TypeError("\'Decimal128\' object is not iterable"), TypeError(\'vars() argument must have __dict__ attribute\')]', 'See: https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#serializing-objects')

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 147, in jsonable_encoder
    return _dump_other(
           ^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 286, in _dump_other
    raise ValueError(errors) from e
ValueError: [TypeError("'Decimal128' object is not iterable"), TypeError('vars() argument must have __dict__ attribute')]

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "xxxxxxxxxxxxxx/c.py", line 19, in <module>
    print(jsonable_encoder({'dec': Decimal128('123')}, custom_serializer=custom_serializer))
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 113, in jsonable_encoder
    return _dump_dict(
           ^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 226, in _dump_dict
    encoded_value = jsonable_encoder(
                    ^^^^^^^^^^^^^^^^^
  File "xxxxxxxxxxxxxx/.venv/lib/python3.11/site-packages/aws_lambda_powertools/event_handler/openapi/encoders.py", line 157, in jsonable_encoder
    raise SerializationError(
aws_lambda_powertools.event_handler.openapi.exceptions.SerializationError: ('Unable to serialize the object 123 as it is not a supported type. Error details: [TypeError("\'Decimal128\' object is not iterable"), TypeError(\'vars() argument must have __dict__ attribute\')]', 'See: https://docs.powertools.aws.dev/lambda/python/latest/core/event_handler/api_gateway/#serializing-objects')
boring-cyborg[bot] commented 3 months ago

Thanks for opening your first issue here! We'll come back to you as soon as we can. In the meantime, check out the #python channel on our Powertools for AWS Lambda Discord: Invite link

leandrodamascena commented 3 months ago

Hey @LironEr thanks for opening this issue! I see your point here and I agree that we should pass custom_serializer for every function that invokes jsonable_encoder again. Do you want to work in a PR to fix this? We can include this bugfix in our next release 11/07.

heitorlessa commented 3 months ago

Since this is a bug, I've created a draft PR with the fix. At a first glance, we seem to be missing in other data types too so I'll look into those later before getting a review.

4664

PS: To prevent getting lost in translation, by "do you want to work in a PR to fix this?" was meant as a "if can help out we can prioritize this for the next release".. just in case : D

github-actions[bot] commented 2 months ago

⚠️COMMENT VISIBILITY WARNING⚠️

This issue is now closed. Please be mindful that future comments are hard for our team to see.

If you need more assistance, please either tag a team member or open a new issue that references this one.

If you wish to keep having a conversation with other community members under this issue feel free to do so.

LironEr commented 2 months ago

Oops missed the notifications about that Thank you so much for the fast fix (:

github-actions[bot] commented 2 months ago

This is now released under 2.41.0 version!