microsoft / coe-starter-kit

Other
742 stars 219 forks source link

[CoE Components]: Desktop Flows sync hitting Dataverse/TDS result set limit #419

Closed petepuu closed 8 months ago

petepuu commented 3 years ago

Desktop Flows sync template is hitting the Dataverse TDS buffer/result set limits in "List Desktop Flows" action. In Flow the error is "Cannot write more bytes to the buffer than the configured maximum buffer size: 104857600.'" and when testing the same query in SSMS we receive error "Return records size cannot exceed 83886080. Make sure to filter result set to tailor it to your report." indicating that query hit the TDS 80MB limit.

image

The main issue seems to be clientdata column value returned which in my test returned about 13MB of data only for 1 desktop flow in the table. This would be easy to fix if the List Rows action of Dataverse (legacy) connector used in the flow offers support for selecting columns like the new Dataverse current env connector's List Rows action supports.

image

Component:

JeneferM-MSFT commented 3 years ago

Hello, unfortunately you have discovered a flaw in our system that we cannot fix at this time. Today we use the Legacy connector to Dataverse as it allows us to select different environments to query, and the data for these is stored in each env in the tenant. And the new Dataverse connector does not yet let us pick which envt to query, always current. Unfortunately Legacy connector does not let us select which fields to bring back, and while we do not use this field, and could chose to not use it, we cannot chose to not bring it back with this connector. Connector team is making selection of envt available to the new Dataverse Connector and we will replat the suite to it for this sync flow and others that rely on it. Until then this is a limitation in the toolkit.

petepuu commented 3 years ago

Thanks Jenefer! Yes that is what I was expecting and hope that possibility to select environment will be available soon in new Dataverse connector.

Jenefer-Monroe commented 9 months ago

We have an early release of this giant upcoming version of the kit. Please test for us! See this bug for build and how tos: https://github.com/microsoft/coe-starter-kit/issues/7322

Jenefer-Monroe commented 8 months ago

Fixed with Jan relesae.