search

sodadata / soda-sql

Soda SQL and Soda Spark have been deprecated and replaced by Soda Core. docs.soda.io/soda-core/overview.html
https://docs.soda.io/
Apache License 2.0
59 stars 16 forks source link

[Spark] Column with data type `decimal(38,18)` not showing up correctly in the Soda cloud #148

Closed JCZuurmond closed 2 years ago

JCZuurmond commented 2 years ago

Describe the bug I have a numeric column with data type decimal(38,18) that does not show as a numeric column in the Soda cloud.

To Reproduce

  1. Set-up Spark environment
  2. Create table with a column that has data type decimal(38,18)
  3. Run scan

Context I think we do not detect this data type correctly, see here. We should use a startswith like for the varchar.

OS: OSx Python Version: 3.9.6 Soda SQL Version: 2.1.0b18 Warehouse Type: Spark

Solution: 

    def is_number(self, column_type: str):
        return (
            column_type.upper() in ['TINYINT', 'SMALLINT', 'INT', 'BIGINT', 'FLOAT', 'DOUBLE', 'DOUBLE PRECISION', 'NUMERIC'] 
            or column_type.upper().startswith('DECIMAL')
        )