Tomme / dbt-athena

The athena adapter plugin for dbt (https://getdbt.com)
Apache License 2.0
140 stars 79 forks source link

dbt seed fails when too long content #136

Open plektra opened 1 year ago

plektra commented 1 year ago

When trying to add a seed file that's too long for Athena, the operation fails because it exceeds the query string length hard limit of 262144 bytes. The error message is somewhat like this:

An error occurred (InvalidRequestException) when calling the StartQueryExecution operation: 1 validation error detected: Value 'insert into database.table (col1, col2, ...) values (........) at 'queryString' failed to satisfy constraint: Member must have length less than or equal to 262144

Perhaps it's not a good use of seeds when the table size exceeds single query lenght limits, but at least FYI for those wondering about the same issue. I worked around this issue by introducing my static table as a new Glue table from S3 data.