On many Operations, Dremio does not return the actual row count of the result sets as "rowCount", but instead a number related to the operation.
A good example is an insert: If you insert 1_000_000 rows into a table, Dremio will show a "rowCount" of 1_000_000 (Dremio 24.2.6). However, the result set of course only has one row - the number of inserted rows.
Is there an existing issue for this?
Current Behavior
On many Operations, Dremio does not return the actual row count of the result sets as "rowCount", but instead a number related to the operation. A good example is an insert: If you insert 1_000_000 rows into a table, Dremio will show a "rowCount" of 1_000_000 (Dremio 24.2.6). However, the result set of course only has one row - the number of inserted rows.
Due to the implementation here: https://github.com/dremio/dbt-dremio/blob/9a9391e8fdfd1d4e03a8d214e5f0647894ecb06d/dbt/adapters/dremio/api/cursor.py#L172
the cursor iterates over hundreds of thousands of empty resultsets.
Expected Behavior
As we cannot count on Dremio returning the number of results in "rowCount" we should probably just
breakwhen we hit the first empty batch of results.Steps To Reproduce
No response
Environment
Relevant log output
No response