When using the sql_database source with .with_resources, schema reflection occurs for all tables in the schema, not just the specified ones. This behavior leads to excessive queries and significant performance degradation for schemas with many tables.
This aspect is not clear in the docs.
Workaround
To limit schema reflection to specific tables, use the table_names argument in sql_database instead of .with_resources:
When using the sql_database source with .with_resources, schema reflection occurs for all tables in the schema, not just the specified ones. This behavior leads to excessive queries and significant performance degradation for schemas with many tables.
This aspect is not clear in the docs.
Workaround To limit schema reflection to specific tables, use the table_names argument in sql_database instead of
.with_resources
:This approach ensures that only the specified tables are reflected and avoids unnecessary overhead.