MeltanoLabs / target-snowflake

Singer Target for the Snowflake cloud Data Warehouse
https://hub.meltano.com/loaders/target-snowflake--meltanolabs/
Other
10 stars 24 forks source link

Cannot import values as number/decimal #255

Open dluo-sig opened 2 months ago

dluo-sig commented 2 months ago

I'm trying to figure out how to get my data in as a number/decimal, instead of a double. My data type in SQL server is decimal(38,20). Here is my output data type:

"val2":{"inclusion":"available","multipleOf":1e-20,"type":["null","number"]}

However, this seems to come out in snowflake as a double/float and loses the precision. I see that a potential fix for this is to specify singer_decimal on the tap. In that case, I see that the data type is the following:

"val2":{"inclusion":"available","format":"singer.decimal","type":["null","number","string"],"additionalProperties":{"scale_precision":"(38,20)"}}

The problem with this approach is that it ends up in snowflake as a varchar. For what it's worth, regardless of singer_decimal specification, the actual value that I see in the output file of the tap has the correct precision. It's just lost somewhere when pushing it to snowflake.

Not sure if it's related, but I also see this in the code defining the python type as a float:

class NUMBER(sct.NUMBER):
    """Snowflake NUMBER type."""

    def __init__(self, *args: t.Any, **kwargs: t.Any) -> None:
        super().__init__(*args, **kwargs)

    @property
    def python_type(self):
        return float
edgarrmondragon commented 2 months ago

Linking to Slack: https://meltano.slack.com/archives/C069A0GE129/p1725897289536329