search

Azure / azure-storage-azcopy

The new Azure Storage data transfer utility - AzCopy v10
MIT License
614 stars 222 forks source link

Fix making too many create directory call #2828

Closed dphulkar-msft closed 4 weeks ago

dphulkar-msft commented 1 month ago

Issue: In the folderCreationTracker.go file, we addressed an issue where multiple createDirectory REST calls were being sent for the same folder during the copy operation, particularly with deeply nested directory structures. This was caused by the StopTracking() function prematurely removing folder entries from the contents map, leading to redundant createDirectory requests.

Fix: To resolve this issue while maintaining memory optimization (which was the original reason for removing entries), we have replaced the contents map with a trie data structure. The trie allows us to efficiently track folder creation and ensures that redundant requests are avoided, while also optimizing memory usage.

dphulkar-msft commented 1 month ago

@gapra-msft @adreed-msft Feel free to suggest alternative approaches. This is just the reference MR showcasing a simple fix for the issue.