Copy file dialogue closes BEFORE actual copy is complete when copying large files to USB flash memory stick.

[tethragon]

Nemo version (nemo --version) : 5.6.4 Is issue with desktop or windowed nemo? : Both Distribution - (Mint 17.2, Arch, Fedora 25, etc...) : Mint 21.1 Graphics hardware and driver used : system specs 32 or 64 bit : 64 bit

Issue

When copying a large file (i.e. a 5GB iso) to a USB flash memory stick, the file copy dialogue closes too fast (in a couple of seconds), but the actual transfer to USB could take much more depending on the speed of USB stick and the speed of USB port used (i.e. USB 3.0 / 3.1 / 3.2 etc.) The file copy dialogue closes when the file is copied to cache which leads the user into the false assumption that the copy has been completed.

Steps to reproduce

1. copy/paste a large file to a usb flash memory stick
2. open iostat and watch the I/O activity to the same flash memory stick

Expected behaviour File copy dialogue should close when the file has actually been copied to USB flash stick.

Other information Started an issue here but it was closed. I believe there is a simple solution: Nemo should run a sync (or fsync) when the file is copied to the cache and when the sync (or fsync) is complete, then the copy dialogue should close. That way, the user gets a proper visual feedback of the completion of the copy.

[ihatetheantichrist-1]

This, and if you copy large files trough USB 2.0 it will incorrectly calculate speed of copying and then stuck on 70 % with correct speed.

[tethragon]

This, and if you copy large files trough USB 2.0 it will incorrectly calculate speed of copying and then stuck on 70 % with correct speed.

The speed calculation can't be correct because the USB stick can't achieve the same speed depending on how large or how many are the files. This is not a problem, in my opinion, as long as the copy window stays open so you have visual feedback that the copy is still going on.

[ihatetheantichrist-1]

as long as the copy window stays open so you have visual feedback that the copy is still going on.

but it does not and it reports cca 160mb/s for USB 2.0 which can't be correct

[SergiusTheBest]

There is a similar issue for Nautilus: https://bugs.launchpad.net/ubuntu/+source/nautilus/+bug/1871869

[SergiusTheBest]

@tethragon

I believe there is a simple solution: Nemo should run a sync (or fsync) when the file is copied to the cache and when the sync (or fsync) is complete, then the copy dialogue should close. That way, the user gets a proper visual feedback of the completion of the copy.

In such a case the progress will jump quickly to 99% (source file -> cache) and hang there for a long period (cache -> destination file).

[tethragon]

@tethragon

I believe there is a simple solution: Nemo should run a sync (or fsync) when the file is copied to the cache and when the sync (or fsync) is complete, then the copy dialogue should close. That way, the user gets a proper visual feedback of the completion of the copy.

In such a case the progress will jump quickly to 99% (source file -> cache) and hang there for a long period (cache -> destination file).

It is better to have the progress window open, so I know the file copy is still going on and haven't finished, rather than having no visual feedback at all. If somehow the devs could make it show the actual percentage and write speed it would be even better.

[ihatetheantichrist-1]

I hope we all can agree we can not damage BFU's USB stick filesystem just because he decided to copy some photos on USB stick and pull out after dialog told him it is done. Neither we can expect him to fire up iostat and check out if the file operation completed or even understand the concept of IO beyond file copy dialog which should describe reality. For him it would end up at "Linux Mint damages my data on USB stick" and he is not that much un-right. I care about year of the Linux desktop and I hope you too. Not to mention when I copy some music and photos iostat sometimes continues to report activity even 10 minutes after the dialog is closed /that is even after umouting in Nemo. I sincerely believe this is very important problem for user experience. @mtwebster

[tethragon]

I recently watched a video from LTT where he reported that Windows 11 (maybe also 10-I am not sure) disable write cache by default for operation to removable drivers, exactly to solve problems like the one described here. Maybe this is a simple solution to this problem. This or leave the file operation window OPEN (stuck at 99% - I don't care) until the operation is truly finished. I believe this is essential for the best user experience.

[ihatetheantichrist-1]

I recently watched a video from LTT where he reported that Windows 11 (maybe also 10-I am not sure) disable write cache by default for operation to removable drivers, exactly to solve problems like the one described here. Maybe this is a simple solution to this problem. This or leave the file operation window OPEN (stuck at 99% - I don't care) until the operation is truly finished. I believe this is essential for the best user experience.

I tried disabling cache in /dev/hdparm.conf, but it did not help or I did it wrong. If you know how to please let me know.

[tethragon]

I don't think that globally disabling write cache is a proper solution. However, if you wish to go that path:

How to disable the Write Cache on Linux

Press CTRL + ALT + T to start the terminal.
Run Sudo gedit /etc/hdparm.conf.
Find write_cache and remove the # preceding it.
Save and restart your computer.
Write caching is a great performance enhancement, but it has its pros and cons.

[Thoupler]

@tethragon

I might have a fix for ya based on the following article: https://www.baeldung.com/linux/file-system-caching

The problem seems to be not only on the file manager but the kernel itself (correct me, if I'm wrong). There are some variables you can change to make the copy duration more accurate. These are:

• vm.dirty_background_ratio
• vm.dirty_background_bytes
• vm.dirty_ratio
• vm.dirty_bytes

To get the current values of those variables you type in the terminal the following line:

sudo sysctl -a | grep dirty

After you typed in your password (if there is one) you might get values like this (copied from linked article above):

vm.dirty_background_ratio = 10
vm.dirty_background_bytes = 0
vm.dirty_ratio = 20
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_writeback_centisecs = 500

I for myself changed just four values (as mentioned above) by opening the /etc/sysctl.conf file with

sudo nano /etc/sysctl.conf

and added the following entries at the end of the file:

vm.dirty_background_ratio = 0
vm.dirty_background_bytes = 20971520
vm.dirty_ratio = 0
vm.dirty_bytes = 20971520

After that press Crtl + O for saving the entries in the files and press Crtl + X to close the file. Now with the entries added you have to apply them with either

sudo sysctl -p

or rebooting your PC. This should now result into the file manager closing the copy window as expected - when the file copy is acutally done.

I have choosen the value of 20971520 (1024 1024 20) because the average copy speed of a USB 2.0 device is somewhat around 20 MB/s and I tried out lower and higher values, too. But 20MiB seem to be the best value in my testings. I tried it on USB sticks and external hard drives with different speeds over USB 3.0 and they work all pretty good and the estimations are way more accurate then before.

[SergiusTheBest]

@Thoupler

The problem seems to be not only on the file manager but the kernel itself (correct me, if I'm wrong).

You don't want to mess with the kernel as it will affect all file operations. The copy dialog should flush cache only for the specific file. It should be fixed in gnome as Nemo uses its runtime libraries.

[Jeremy7701]

How do generic system changes for slow devices, such as USB, affect the performance of fast devices such as NVME or SATA SSD's?

[Thoupler]

@SergiusTheBest

I can mess with the kernel all I want and you can not stop me! >:) But it looks like that this is not a modification on the kernel itself as @Jeremy7701 claimed. Only just generic system changes so it should be fine as long as you don't use a too small numbers for the above values.

The copy dialog should flush cache only for the specific file.

I don't know what you mean by that. Can you elaborate on that?

It should be fixed in gnome as Nemo uses its runtime libraries.

Does it mean that it is fixed or that it will be fixed in the future? Because if it is fixed then it is not working.

The modification does not harm the copy speed of any device I have tested. When I copy from one SSD to another I get hundreds of MB/s but when I copy to a USB stick the speed will be just around 15 to 20 MB/s which is totally normal because my sticks suck. So there is no downside I can see. At least not yet. Maybe you want to explain exactly why it could be a problem? I'm very interested in that.

[Guldoman]

It should be the job of the distro to disable caching for external drives, not of the file manager.

A solution similar to what is proposed in https://forum.manjaro.org/t/root-tip-how-to-disable-write-cache-for-usb-storage-devices/135566 should be shipped by distros. So an udev rule to either use hdparm, or better yet the "new" sysfs controls, to disable caching for specific devices when they're inserted.

[Jeremy7701]

It doesn't matter how you fiddle with the kernel etc - it will take exactly as long (or possibly longer) to write to a slow USB device. The solution lies with a better USB device!

[SergiusTheBest]

@Thoupler

People invented lazy write (when data are written to the memory first and later they are written to the disk in background) to boost system performance. It's not about write speed, it's more about application responsiveness. In addition the system may reorder and merge small writes into bigger ones to further improve performance and prolong SSD life.

Your original kernel setting allow to use 20% of RAM to cache unwritten data. Your new kernel settings limit it to 20MB. That is very small for nowadays.

There are means to control lazy write behavior per file or per disk while other files/disks will still have benefits from using the default lazy write.

So the fix can be one of:

• Gnome adds functions to detect disk type (external/internal) and disable write cache for a file on an external disk, and those changes are propagated to Nemo
• system disables write cache for external disks (should be done by Debian/Ubuntu/Mint developers)

AFAIK none of these are implemented.

P.S. Windows uses the second approach: it disables write cache for external disks.

[Jeremy7701]

Why would you want to disable the write cache - is this just for those who like to disconnect devices without unmounting them properly? Is it for users who have to shut down the computer and run the instant the copy completes, like in spy films?

[tethragon]

Why would you want to disable the write cache - is this just for those who like to disconnect devices without unmounting them properly? Is it for users who have to shut down the computer and run the instant the copy completes, like in spy films?

Because the user should be informed correctly what is going on with the copy. Simple as that.

[Jeremy7701]

Totally disagree.

[SergiusTheBest]

@Jeremy7701

Why would you want to disable the write cache - is this just for those who like to disconnect devices without unmounting them properly? Is it for users who have to shut down the computer and run the instant the copy completes, like in spy films?

Fast copy and long unmount confuses users. Also there is no progress indicator for unmount, so you don't know how long you should wait.

[Jeremy7701]

I cannot agree with you - I want things to run at full speed, regardless of whether they are run via a GUI or the command line. I don't think this discussion is going to go anywhere, since we have two diametrically opposed views. I'm in favour of the status quo and you are not.

[SergiusTheBest]

@Jeremy7701 The best solution is to have a switch in the setting for removable drives with 2 options:

• optimize for fast removal
• optimize for performance

[leigh123linux]

@Thoupler

People invented lazy write (when data are written to the memory first and later they are written to the disk in background) to boost system performance. It's not about write speed, it's more about application responsiveness. In addition the system may reorder and merge small writes into bigger ones to further improve performance and prolong SSD life.

Your original kernel setting allow to use 20% of RAM to cache unwritten data. Your new kernel settings limit it to 20MB. That is very small for nowadays.

There are means to control lazy write behavior per file or per disk while other files/disks will still have benefits from using the default lazy write.

So the fix can be one of:

• Gnome adds functions to detect disk type (external/internal) and disable write cache for a file on an external disk, and those changes are propagated to Nemo
• system disables write cache for external disks (should be done by Debian/Ubuntu/Mint developers)

AFAIK none of these are implemented.

P.S. Windows uses the second approach: it disables write cache for external disks.

Gnome haven't addressed this in nautilus

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/thread/4YQMQGAVWVHKYC67TKYXRVRZ5QLK6DYD/

[Qronikarz]

The best solution is to have a switch in the setting for removable drives with 2 options:

* optimize for fast removal

* optimize for performance

I agree with that. The first option should be a default for new users since that's what most of them expect. Second can stay for people that want more speed.

I unfortunately made mistake once time of taking the USB stick after the dialog finished, but syncing was not done and in turn none of the files remained on the USB stick.

[Jeremy7701]

It's all OK - when you unmount or eject the device - as you should always do - any outstanding data will be forced onto the hardware. If you just rip the device from its connection, then expect corruption, as you discovered to your cost.

[Qronikarz]

Accidents can happen and new people coming from Windows (In Linux Mint website: "Most of our users come from Windows and they never look back.") may be used to different behavior as was already pointed before. Thankfully we're on Linux where you can customize things to your liking. One additional setting won't hurt anybody (other than developers who have to implement it).

https://discussion.fedoraproject.org/t/larina3315-usb-storage-sync/104272 Here a Fedora package that enables sync mounting for external USB memory. So it won't touch anything internal which is a good compromise in my opinion. The only problem is that it's GitHub repository was lost but the code is available on Fedora Copr.

[Jeremy7701]

Sounds great for doing backups to external disks connected via USB.

[AttilaLL]

This is my personal bane, I have to do iostat. I am seriously thinking about abandoning Linux for windows with all its flaws, at the end of a day I need productive OS-environment not a hobby toy.

[Jeremy7701]

iostat - new to me, (and difficult to interpret) but it provides some interesting stats. I will give it a trial run, next time I run a heavy I/O process. You could just issue a 'sync' command - "Synchronize cached writes to persistent storage".

[tethragon]

I just realized another problem: While copying ~30k files (~25GB of data) the copy was taking for ever so I stopped the process. I then unmounted the USB. The system (linux mint / nemo) never gave me a warning that syncing was still in progress and the USB was unmounted (at least in the user interface). However, only when I opened gnome DISKS app I could see that an "unmounting job was in process" because there were still a ton of data to be written. Nemo should give a WARNING that file syncing is in process and user should not remove USB. This is a critical bug imo.

[leigh123linux]

I just realized another problem: While copying ~30k files (~25GB of data) the copy was taking for ever so I stopped the process. I then unmounted the USB. The system (linux mint / nemo) never gave me a warning that syncing was still in progress and the USB was unmounted (at least in the user interface). However, only when I opened gnome DISKS app I could see that an "unmounting job was in process" because there were still a ton of data to be written. Nemo should give a WARNING that file syncing is in process and user should not remove USB. This is a critical bug imo.

Nemo does warn the data hasn't been written when ejecting the usb media, it also gives a message when it's safe to remove.