When a new datasource is created from Snowflake all columns aren't showing. This means that LIST columns doesn't show everything. If using the cli someone is prompted to ADD a new column even though it's in there.
Query
I created a datasource using this query:
SELECT BABY_NAME
, NUMBER_OF_BABIES
, YEAR_OF_BIRTH
, GENDER
, SYLLABLES
, STATE
, RANK() OVER (PARTITION BY YEAR_OF_BIRTH, GENDER ORDER BY NUMBER_OF_BABIES DESC) AS ranking
FROM NAMES.PUBLIC.CA_HI
QUALIFY RANK() OVER (PARTITION BY YEAR_OF_BIRTH, GENDER ORDER BY NUMBER_OF_BABIES DESC) <= 100
tdsx
I then added it to saved datasources as a tdsx
List columns
When I run list_columns I don't see all of the columns. I only see "State"
@JustinGrilli Do you think it would be heard to make list_columns show everything even when there isn't a column object? Or perhaps we need some sort of "init" function that would add all the columns in that metadata records section even when there isn't a column
If I remember correctly we solved this in the code for Airflow but not in the CLI. I think we solved it in the base functionality, not the DAG.
When a new datasource is created from Snowflake all columns aren't showing. This means that LIST columns doesn't show everything. If using the cli someone is prompted to ADD a new column even though it's in there.
Query
I created a datasource using this query:
tdsx
I then added it to saved datasources as a tdsx
List columns
When I run list_columns I don't see all of the columns. I only see "State"
@JustinGrilli Do you think it would be heard to make list_columns show everything even when there isn't a column object? Or perhaps we need some sort of "init" function that would add all the columns in that metadata records section even when there isn't a column
If I remember correctly we solved this in the code for Airflow but not in the CLI. I think we solved it in the base functionality, not the DAG.