search

Azure / azure-storage-net-data-movement

Azure Storage Data Movement Library for .Net
MIT License
276 stars 132 forks source link

Percentage progress #203

Open MarcinKowal opened 4 years ago

MarcinKowal commented 4 years ago

Which service(blob, file) does this issue concern?

All

Which version of the SDK was used?

1.2.0

On which platform were you using? (.Net Framework version or .Net Core version, and OS version)

How can the problem be reproduced? It'd be better if the code caused the problem can be shared.

Hi, I would like to ask do you consider percentage progress reported during the transfer. Currently there are only TransferredBytes and TransferredFiles reported in TransferStatus. I realize that it will require total bytes to be known prior the transfer or at least to be calculated during the transfer. It can be achieved as a result from enumeration executed concurrently in different thread

MarcinKowal commented 4 years ago

@jiacfan @EmmaZhu any comments on that ?

jiacfan commented 4 years ago

Hi, @MarcinKowal

Thanks for reaching us, and sorry for the latency due to national holiday.

Below are some insights on how DMLib works now:

Currently, DMLib's listing and transferring are decoupled, and processed in parallel in async pattern. Listing will pause shortly if candidate items(already listed and queued to be transferred) are 3X items being transferred(according to max parallel), and will continue once there're slots for transferring. This is to ensure limited items are journaled, so whatever the transfer scale would be, limited resources are used in any phase of transferring, and pause&resume can be fulfilled for any scale of transferring.

The feature request is good. For in memory journal mode, it might cause additional resource consumption, while there're approaches to discuss. We'll consider this feature request in the further planning.

Thanks, Jiachen