search

meltano / sdk

Write 70% less code by using the SDK to build custom extractors and loaders that adhere to the Singer standard: https://sdk.meltano.com
https://sdk.meltano.com
Apache License 2.0
92 stars 66 forks source link

feat: define a maxLength limit for varchar at the connector level #1812

Open pnadolny13 opened 1 year ago

pnadolny13 commented 1 year ago

Originally raised in slack https://meltano.slack.com/archives/C01TCRBBJD7/p1688652938423919

The meltanolabs target-snowflake was throwing an error because its accepting the max length 4294967295 from the schema message as the varchar length which is larger than what snowflake accepts 16777216. It looks like currently we just pass that value through to sqlalchemy with no validation to limit it from being too large and ultimately the DDL fails.

I'm going to fix it in target-snowflake but it does feel like something that should be handled by the SDK because its going to be relevant for most ever target. If we had a property method that needed to be implemented or a default class property that could be overridden. Another thing that might be worth checking out is if the sqlalchemy dialect package could tell us this generically somehow.

pnadolny13 commented 1 year ago

This was solved in target-snowflake in https://github.com/MeltanoLabs/target-snowflake/pull/68. That has some logic specific to target-snowflake so using something like Edgar suggested in https://github.com/MeltanoLabs/target-snowflake/pull/68#discussion_r1254674308 would probably be best for the SDK.

stale[bot] commented 2 weeks ago

This has been marked as stale because it is unassigned, and has not had recent activity. It will be closed after 21 days if no further activity occurs. If this should never go stale, please add the evergreen label, or request that it be added.