KonradIT / mmt

Media Management Tool - make importing videos/photos from GoPro and other action cameras/drones a little bit more bearable.
http://mmt.camera
Apache License 2.0
109 stars 12 forks source link

RFE: Limit concurrent copies from SD Card to improve performance #116

Open sandymac opened 1 year ago

sandymac commented 1 year ago

RFE: Limit concurrent copies from SD Card

Context:

Trying to import about 50 files from a GoPro 9's SD Card

Expected Behavior:

Copy one file at a time (better for local storage) or limit it to a few (maybe better for network sources).

All files at once has significant overhead, unfortunate failure mode, and likely causes unnecessary disk fragmentation or unnecessary defragmentation effort in destination drives.

Current Behavior:

Currently mmt seems to copy every file it discovers on the SD Card simultaneously. This has negative throughput impact as the SD Card is internally paging between blocks of data to read it. In Windows Task Manager view for the SD Card's drive I noticed the Read Speed almost double as the number of active file copies went from 40+ to a handful.

Also, when a SD Card become unavailable during a copy I got partial data in the destination for many files. I'd rather only 1 file be incomplete than dozens. Also if destination drive were to fill up, I'd prefer some complete copies instead of many partial copies.

Steps to Reproduce:

Have lots of new media to import.

sandymac commented 1 year ago

Here is the SD Card getting 25MB/s towards the start of a 96 file import Screenshot 2023-06-19 175048

and here it is getting 47 MB/s towards the end of the import when many of the files had completed and there was less thrashing Screenshot 2023-06-19 190002