holoviz / lumen

Illuminate your data.
https://lumen.holoviz.org
BSD 3-Clause "New" or "Revised" License
177 stars 20 forks source link

Speculatively cast columns to determine if they are wrongly typed #689

Closed philippjfr closed 2 months ago

philippjfr commented 2 months ago

Sometimes SQL data will be incorrectly typed as strings rather than the actual type. Since computing enums for float or integer values is extremely expensive and silly we have the get_schema methods speculatively try to cast each column in the subset of columns. If we can successfully cast the subset we will TRY_CAST before computing the min/max values for the schema.

codecov[bot] commented 2 months ago

Codecov Report

Attention: Patch coverage is 69.04762% with 13 lines in your changes missing coverage. Please review.

Project coverage is 56.66%. Comparing base (f8b02ee) to head (a1a4f51). Report is 2 commits behind head on main.

Files Patch % Lines
lumen/transforms/sql.py 41.66% 7 Missing :warning:
lumen/sources/base.py 73.33% 4 Missing :warning:
lumen/sources/duckdb.py 87.50% 1 Missing :warning:
lumen/sources/intake_sql.py 85.71% 1 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #689 +/- ## ========================================== - Coverage 56.66% 56.66% -0.01% ========================================== Files 98 98 Lines 11682 11718 +36 ========================================== + Hits 6620 6640 +20 - Misses 5062 5078 +16 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.