search

TheByteStuff / AzureTableUtilities

Backup, Copy, Delete and Restore utilities for use with Azure Tables.
MIT License
4 stars 9 forks source link

System.OverflowException: Array dimensions exceeded supported range. #3

Closed ppongsakorn closed 3 years ago

ppongsakorn commented 3 years ago

I run backup with more than 100 millions records in azure table storage. I got and exception

System.OverflowException: Array dimensions exceeded supported range. at TheByteStuff.AzureTableUtilities.BackupAzureTables.BackupTableToFile(String TableName, String OutFileDirectory, Boolean Compress, Boolean Validate, Int32 TimeoutSeconds, List1 filters) --- End of inner exception stack trace --- at TheByteStuff.AzureTableUtilities.BackupAzureTables.BackupTableToFile(String TableName, String OutFileDirectory, Boolean Compress, Boolean Validate, Int32 TimeoutSeconds, List1 filters) at TheByteStuff.AzureTableUtilities.BackupAzureTables.BackupTableToBlob(String TableName, String BlobRoot, String OutFileDirectory, Boolean Compress, Boolean Validate, Int32 RetentionDays, Int32 TimeoutSeconds, List`1 filters)

How many maximums records that AzureTableUtilities are supported.

Thank you.

TheByteStuff commented 3 years ago

Hello,

Sorry to hear you've run into a problem. I have not tried a backup with 100 million records. It will be a few weeks before I can try to look at it. I'm not sure if I have an environment to test 100 million records or not.

Can you please provide more details on the table - how many fields per row, amount of data (bytes) per row, etc? Also, can you tell how many rows have been pulled to the file before the exception? Are you passing any filters to the process?

In the meantime, can you try BackupTableToBlobDirect to see if that works? I did have another user report problems with large backup to a file; however, he moved on to a different solution as there were other constraints as well before being able to try the direct to blow backup. Assuming it works, you should then be able to download the file to your local environment if that's the end goal you are trying for. Given the record count, I'd recommend having compression on if you don't already.

ppongsakorn commented 3 years ago

Hi,

I have 17 columns each row and 11,433,910 rows in table. Each row size less than 1,000 bytes.

TheByteStuff commented 3 years ago

I did manage to recreate the problem. I ended up with a table that had 66million+ rows with 17 columns as you have. The backup file size is around 60GB. Anyhow, I ended up adding a StreamWriteSizeInBytes property to the BlobBlock for the two backup to Blob methods (original and the direct) which allowed my large table to backup to blob properly.

I published the change under v4.4.1 on NuGet. If you have a chance to try it out and it works, please let me know.

Thank you!