While processing documents using a source.incremental, some causes a lot of errors of this type, this shouldn't happen.
It works without the incremental config.
Edit: Using a filter instead of incremental still causes errors, so this could be a underlying problem with handling data quality issues.
Edit 2: max_table_nesting doesn't work with the verified source, it still fails on the same deeper level than specified parameter. But adding a mongodb.resources["orders"].max_table_nesting = 2 do work partially, for example, still receiving errors for orders__shopping_cartproductskey.
Edit 3: The problem continued even without drop_sources option.
Im using the verified mongodb source, tested many ways, some collections load if are cleaner.
Expected behavior
No errors, documents loaded.
Steps to reproduce
Load documents with nested documents on fields, where some aren't documents, for example, mix array with undefined.
Use refresh = "drop_sources"
And the error is true, for example:
Cannot find the object "orders__shopping_cart" because it does not exist or you do not have permissions.
Doesn't exists on any schema.
dlt version
1.4
Describe the problem
While processing documents using a source.incremental, some causes a lot of errors of this type, this shouldn't happen. It works without the incremental config.
Edit: Using a filter instead of incremental still causes errors, so this could be a underlying problem with handling data quality issues. Edit 2: max_table_nesting doesn't work with the verified source, it still fails on the same deeper level than specified parameter. But adding a mongodb.resources["orders"].max_table_nesting = 2 do work partially, for example, still receiving errors for orders__shopping_cartproductskey. Edit 3: The problem continued even without drop_sources option.
Im using the verified mongodb source, tested many ways, some collections load if are cleaner.
Expected behavior
No errors, documents loaded.
Steps to reproduce
Load documents with nested documents on fields, where some aren't documents, for example, mix array with undefined. Use refresh = "drop_sources" And the error is true, for example: Cannot find the object "orders__shopping_cart" because it does not exist or you do not have permissions. Doesn't exists on any schema.
Operating system
Linux, Windows
Runtime environment
Local
Python version
3.11
dlt data source
dlt verified source mongodb
dlt destination
No response
Other deployment details
sql server destination
Additional information
consistent error, 1.3 version same problem.