meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
97 stars 69 forks source link

bug: CustomStreamMap always convert `datetime` field to `date` #2739

Open lngvietthang opened 1 day ago

lngvietthang commented 1 day ago

Singer SDK Version

0.41.0

Is this a regression?

Python Version

3.9

Bug scope

Mapping (stream maps, flattening, etc.)

Operating System

Linux

Description

In my Meltano configuration, I use meltano-map-transformer (latest version with singer-sdk@0.41.0) with config as below:

stream_maps:
  xxx:
    creation: datetime.datetime.strptime(creation, '%Y-%m-%dT%H:%M:%S.%f%z')
...

With new update from singer-sdk v0.41, this field (creation) is mapped with date type in schema message.

class CustomStreamMap(StreamMap):
    def _eval_type(  # noqa: PLR0911
        self,
        expr: str,
        default: th.JSONTypeHelper | None = None,
    ) -> th.JSONTypeHelper:
        ...
        if expr.startswith("datetime.date") or expr.endswith(".date()"):
            return th.DateType()

        if expr.startswith("datetime.datetime"):
            return th.DateTimeType()
        ....

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

Related PR:

2665

Code

class CustomStreamMap(StreamMap): def _eval_type( # noqa: PLR0911 self, expr: str, default: th.JSONTypeHelper | None = None, ) -> th.JSONTypeHelper: ... if expr.startswith("datetime.date") or expr.endswith(".date()"): return th.DateType()

    if expr.startswith("datetime.datetime"):
        return th.DateTimeType()
    ....

Link to Slack/Linen

No response

edgarrmondragon commented 1 day ago

I think we need to swap the position of 2 if clause to check the datetime data type firstly.

That makes sense @lngvietthang. PRs welcome!