search

AzureCosmosDB / data-migration-desktop-tool

MIT License
122 stars 46 forks source link

AzureBlobDataSource to AzureTableApi: Out of Memory #132

Open GuillaumeVadeBe opened 4 months ago

GuillaumeVadeBe commented 4 months ago

We are trying to move data from an AzureBlobDataSource to AzureTableApi and noticed that the tool keeps on increasing in memory. If we look at the source code (AzureTableAPIDataSinkExtension), it turns out that a task is being created for every single dataItem. The completion of the tasks is only awaited after the entire enumeration of the dataItems which causes the memory to increase depending on the amount of items in the source.

Would it be possible to fix this so the memory usage remains stable no matter the blob size?

pdehne-steidle commented 1 month ago

We have the same issue. In our case we are using AzureTableAPI as the data source and data sink. It's currently not possible to copy bigger tables.