Moving files from one directory to another is extremely slow!

[zaps166]

Use this script to generate files and folders for testing (the script will create "Files" directory which will contain "Dest" directory and 50 empty files. It will automatically open Caja):

#!/bin/bash
mkdir -p Files || exit 1
cd Files || exit 1
for ((i=1; $i <= 50; i++))
do
    touch $i || exit 1
done
mkdir -p Dest || exit 1
exec caja --no-desktop .

1. Run the script.
2. Select all generated files (except the "Dest" directory).
3. Cut selected files (Ctrl+X), don't copy.
4. Open "Dest" directory.
5. Paste files (Ctrl+V) - this operation can take 10 sec. up to 40 sec.! Sometimes it pastes files immediately, so remove all files and test it again.

Tested with 1.12.6 (two computers) and master: ce7cc9580809d4017c74b0128f7e82f94eb173d9

--- Want to back this issue? **[Post a bounty on it!](https://www.bountysource.com/issues/32336979-moving-files-from-one-directory-to-another-is-extremely-slow?utm_campaign=plugin&utm_content=tracker%2F651521&utm_medium=issues&utm_source=github)** We accept bounties via [Bountysource](https://www.bountysource.com/?utm_campaign=plugin&utm_content=tracker%2F651521&utm_medium=issues&utm_source=github).

[flexiondotorg]

What distro? What filesystem? Hard disk or SSD? What file manager are you comparing with and how quickly does it perform the same operation? Please provide actual metrics not an estimate.

[zaps166]

Manjaro (Arch Linux), I've compiled Caja from sources (but Mate and GTK and GLib are from Manjaro repo). Tested on HDD and SSD. CPU and HDD/SSD does nothing during this operation (notice that files generated by test script are empty). It doesn't depend on computer speed - I can compare it to random usleep() on every file. The time differs, sometimes it is a few seconds (e.g. 10 sec), but sometimes it is 20, 30 or even 40 sec. It is hard to say, I can only create a movie (should I do it?)...

[raveit65]

hmm, i follow your steps to reproduce, pasting the whole files takes about 0.5-1 sec here. fedora 24 (alpha) with a hardware-raid with 4 ssd using raid0 mode. Same with another hardware-raid with 4 normal disks in raid10 mode. No difference to previous test. With a normal external disk connected via slow usb-2.0, not more than 8 or 12 secs. Well, here i see random differences, but all tests are between those values. I think those bad timings are because of usb-2.0, and with a slower disk you will see larger fluctuations. Is your hardware not defect? Btw. can you confirm your results with another file-manager, ie. nemo, nautilus or thunar?

[zaps166]

Hmm, very strange... All files are empty in my test, so HDD speed does nothing (it only creates 50 empty files (i-nodes?)). OS also caches files, so sometimes moving is done in RAM and later (on during unmounting) on real media. Please notice that it happens mostly, but not always - the time is also different on every attempt (sometimes also I've got about 1 sec).

And my new results for this test case:

• Caja (gtk2) - almost always slow, sometimes very slow,
• Nemo (gtk3) - same as Caja,
• Nautilus (gtk3) - same as Caja,
• Thunar (gtk2) - always instant (correct behavior),
• PCManFM (Qt5) - always instant (correct behavior),
• Dolphin (KF5/Qt5) - always instant (correct behavior).

If you want to test again please don't cut-back files from "Dest" to "Files", but remove "Files" and create the directory again. Please retest e.g. 5-10 times on the same machine.

@raveit65 not more than 8 or 12 secs. - too long for 50 empty files :) Even on USB 1.1 (or floppy disk :smile:) it should be instant. Of course it happens with many small (non-empty) files, e.g. while moving a lot of songs or photos. Please notice that copying (not moving) files from the test are instant, so why so slow on moving?

... it looks like random sleep, because CPU nor HDD does nothing, but moving 50 files can take up to 40 sec. (sometimes 10 sec, 5 sec or < 1 sec.).

[zaps166]

Tested on Ubuntu 15.10 and Ubuntu's standard file browser - the same problem as in Caja... So it is not exactly a Caja bug, but is it possible to do a workaround in Caja? Moving files is a normal, frequently used operation, so it is not convenient to use another file manager like Thunar or write command line every time :) Moving hundreds or thousands files may take a few minutes or even hours!

[ghost]

problem still exist and got worse on linux mint 18, mate 1.14.0.1, caja 1.14.1. moving files should be instant, not take 5-10min to move files. setup : SSD / and HDD /home /swap, i did not have this problem on mate before 1.12.x, also i noticed that "cut and paste" demands additional space, eg cut-->paste (100gb), you need 200GB of space. 100GB + an additional 100GB to paste. which is not normal... that would be true if it was copy-->paste

[kn00tcn]

i dont have to try your example, i run into this sometimes, i thought it happens more often when files are images & figured it's generating some thumbnail cache

i do not see any cpu usage during this time, no hard drive blinking, it just sits there for a few secs halting at random (some of the files moved, then pauses.... then eventually the rest finish), so it's not the hardware if different software works

it also takes a long time to open a folder with thousands of files, should i make a new issue about that? (i didnt search yet, although maybe the problem is it's scanning the amount of files in subfolders to display in the list view column)

[zaps166]

Unfortunately developers doesn't care about old annoying bugs (and also regressions) :(

Still happens on 1.16.0. Moving files by drag and drop works OK, but Cut->Paste is broken.

Now I've killed Caja on moving a lot of photos (after a few minutes, but it should take milliseconds).

[raveit65]

Unfortunately developers doesn't care about old annoying bugs (and also regressions) :(

I answered but i couldn't reproduce it, please stop to tell lies here :unamused:

[zaps166]

@raveit65 I also thought about my PRs which are w/o any response for months :cry: (not for Caja, but for panel/applets).

I couldn't reproduce it

My configuration:

• Arch Linux, Caja 1.14.0 GTK2.

hmm, i follow your steps to reproduce, pasting the whole files takes about 0.5-1 sec here.

Sometimes it can take one second, sometimes 30 sec, etc. Sometimes moving hundreds of photos can lock Caja.

Please try this (1000 files):

#!/bin/bash
mkdir -p Files || exit 1
cd Files || exit 1
for ((i=1; $i <= 1000; i++))
do
    touch $i || exit 1
done
mkdir -p Dest || exit 1
exec caja --no-desktop .

Use Ctrl+X and Ctrl+V for moving :) I've killed Caja after 7 minutes, it moves 698 files in 7 minutes. E.g. Thunar can do it in ~1 second, Dolphin (KDE 5) in ~5 seconds.

[lukefromdc]

I've had the problem myself, and it looks like a very old Nauilus bug that nobody has ever fixed. I see this more on moving files than copying them, my video copy jobs normally go OK, but the really ugly case is cleaning two overlapping directories of Debian files. I do this by moving files from one directory into the other, overwriting any copies found there, then moving back. This is very likely to hang outright and force me to kill Caja. No files will be missing or corrupted, all will be in one directory or the other. Still, if I am cleaning directories it pays to back up both of them first.

Copying or moving a whole directory always goes cleanly, it's with a large number of individual files I sometimes get trouble. Copying a newer version of a file onto an older version with the same name in a flash drive (even ONE file) is so nasty I usually delete the older version first, though I've always blamed the flash drives themselves for that.

[zaps166]

Also sometimes (very rare) I've noticed that after successful copying files the windows used to show progress can't close (only killing Caja can close it).

[filosophem]

Hello everyone,

Totally new here. I came across a very similar problem (not to say exactly the same) yesterday while copying 5G of data from one HD to another.

Since I'm not a developper at all, I won't be looking in the code. But maybe with your help, I can set a cron job to log the problem when it occurs (I can try to reproduce it).

Let me know if It's doable

[kn00tcn]

well i was just copying (sometimes moving) over 500GB across different drives twice without a problem

yet moving a set of 350 jpgs from one folder to another on the same drive resulted in getting stuck... couldnt browse new folders in other caja windows, couldnt open a new caja window, the stop button still took a couple minutes for anything to happen & eventually the multiple new caja windows opened up after the progress window closed

i resumed the move now, let's time it: after 2m10s: 130 items remaining 2m11s: 103 items 4m20s: 102 4m30s: 45 5m00s: 44 5m25s: 43 5m40s: 24 5m41s: done

(to add to this annoyance, i was on an external this time, stupid WD sets it to park after only 6secs of no activity, i was playing loosely encoded videos earlier, but this time i had an mp3 playing, which buffered like 10secs at a time, so the drive kept parking & unparking, i wouldnt have been able to traverse in caja back to the videos folder to stop it from parking)

so maybe one thing more important than this bug is to have file operations in a new process like windows has for explorer (good way to not worry about the desktop/tray going down if one of the explorer windows crashes, if you remember to enable it, off by default)

[zaps166]

@kn00tcn Exactly this bug :)

stupid WD sets it to park after only 6secs of no activity

sudo hdparm -B 255 /dev/sdX (X is the letter of hard drive; Docs)

[kn00tcn]

@zaps166 that doesnt work on usb externals, of course i'm aware of such tweaks (plus wdidle & idle3ctl), but i need to break open the enclosure & plug into sata... or just make constant 0byte load in software

so the caja bug, i seemed to have had it almost every single desktop icon move these days, i would have let's say 20+ files saved over a period of a few hours, attempt to drag them into a folder on the desktop, bug appears for 15 seconds

[Co6aka]

FWIW, the same everlasting "preparing to move" bug happens with Nemo file manager too.

[kn00tcn]

as does the ridiculous 'feature' of the faulty grid https://github.com/linuxmint/nemo/issues/108

do both of these go back to gnome2 or is naut/nemo a fork of old mate?

[lukefromdc]

Nemo is a fork of Nautilus 3.4, and Caja is a fork of Nautilus from the 2.32 era but nobody knows exactly where the divergence from GNOME git master begins it seems. Thus any bug common to all three was introduced in Nautilus prior to 2.32 and thus dates back to Sep 2010 or earlier.

[kn00tcn]

i was looking at system monitor while waiting for the frozen transfer when i noticed cpu usage spike up for gvfsd-metadata the moment the transfers suddenly completed

so i started searching, check this https://ubuntuforums.org/showthread.php?p=10658684 might be on to something

so far it's always partition to same partition, most likely when lots of small images are moved

(ugh half a decade old bugs!)

[thohel]

I've been running Arch linux for years, solely with XFCE and thereby thunar as a file manager. Newer experienced this issue. Figured I'd do an Antergos install with Gnome on my HTPC build, so I have Nautilus (not Caja, but thought I'd chime in since it seems related) as file manager.

It has a 1TB SSHD, so not the fastest of drives, but still decent for small files due to the SSD cache. This evening I've been trying to move multiple collections of small 2-20Kb text files. Between 1500 and 6500 files per collection. On every single one of the "moving files" jobs I've hit this bug. Finally I caved, and did the move by terminal, which I guess I should have done initally. I couldn't even switch back to Nautilus before the move was completed! Moving the files in Nautilus? Multiple minutes of waiting, and two reboots, and I still didn't get all of it moved. No heavy CPU load, or RAM usage, no other applications running.

Coming to think of it I do believe I've experienced this bug before, the last time I tried Gnome. Same happened then, and put me of the whole Gnome experience. Disappointing to see that nothing has happened with the issue, and instead it has possibly gotten even worse :(

[zaps166]

Hmm, it locks in g_file_move() function: https://github.com/mate-desktop/caja/blob/v1.16.1/libcaja-private/caja-file-operations.c#L4717

For most files it takes less than a millisecond (i.e. ~200 µs), but for some files it takes 20 seconds, 60 seconds, 1 second, etc., so the entire moving takes sometimes a few minutes or longer!

Here is the example code (use script from https://github.com/mate-desktop/caja/issues/528#issuecomment-254052023). Both: example code and Caja uses the same g_file_move() function. I can't reproduce a bug with this example code: 500 files can move immediately (also tested with separate thread), weird...

#include <stdio.h>

#include <gio/gio.h>

#define NUM_FILES 500
#define BASE_PATH "/home/user/Desktop/Files/"

int main()
{
    GFile *src[NUM_FILES], *dst[NUM_FILES];
    GError *error;

    GCancellable *cancellable = g_cancellable_new();

    char buff[256];
    for (int i = 0; i < NUM_FILES; ++i)
    {
        snprintf(buff, sizeof buff, BASE_PATH "%d", i + 1);
        src[i] = g_file_new_for_path(buff);

        snprintf(buff, sizeof buff, BASE_PATH "Dest/%d", i + 1);
        dst[i] = g_file_new_for_path(buff);
    }

    for (int i = 0; i < NUM_FILES; ++i)
    {
        error = NULL;
        printf("%s -> %s: %d\n",
               g_file_get_path(src[i]),
               g_file_get_path(dst[i]),
               g_file_move(src[i],
                           dst[i],
                           G_FILE_COPY_NOFOLLOW_SYMLINKS | G_FILE_COPY_NO_FALLBACK_FOR_MOVE,
                           cancellable,
                           NULL,
                           NULL,
                           &error));
    }

    g_object_unref(cancellable);

    for (int i = 0; i < NUM_FILES; ++i)
    {
        g_object_unref(src[i]);
        g_object_unref(dst[i]);
    }

    return 0;
}

Any ideas?

[Co6aka]

"it locks in g_file_move() function"

If it locks IN as in inside the function, then the bug is in the library code. Or, to be clear, is it locking inside the for loop; perhaps something to do with the parameters/variables being passed?

Also, try the sample code with more than 1000 files. Long ago it used to not lock unless there were a lot of files, and that may have been more than 1000. (When you see the "Preparing to move..." you'll have hit upon the problem -- and just where is that message originating and why?) More recently it became only a few files. Also-also, I've noticed that the first time I move a few files they move quickly, but after moving a few more batches it gets very slow. This is with Nemo, not Caja.

Maybe it's time to look at the g_file_move library code.

[zaps166]

Inside the g_file_move() function, so it is possible bug inside glib/gio library, but notice, that the example code works fine using the same function and the same flags. I've tried also to run the loop in my code in separate thread and also I've tried to not use cancellable in Caja code and also I've tried with new GFile objects created by path in Caja - nothing helped, so that's why it is weird... I'm not familiar with GLib/GIO, but maybe debugging those libraries can help with fixing bug or with bug workaround?

[Co6aka]

I edited my previous post, but again, when you see the "Preparing to move..." you'll have hit upon the problem -- and just where is that message originating and why? What's the update history on the library; maybe something can be traced back to the root timewise.

[Co6aka]

A couple more thoughts... How does Thunar move files? Build a debug version of Caja that dumps all of the calling params to g_file_move() out to a text file and see if there's anything interesting in there. Copy the exact Caja file move snippet into the test program and try that.

Sorry I can't be more help; microcontroller/assembly coder here.

[zaps166]

How does Thunar move files?

I can see g_file_move(): https://github.com/xfce-mirror/thunar/blob/thunar-1.6.10/thunar/thunar-transfer-job.c#L984

Maybe it's time to look at the g_file_move library code.

Yes...

"Preparing to move..."

This can be simply debugged in GDB, I can paste a backtrace, but tomorrow.

---

Off topic

microcontroller/assembly coder here.

I'm just curious: AVR, '51, PIC, other...? :)

[zaps166]

Backtrace (caja 1.16, glib 2.50.2):

level | function | file | line | address

1  syscall                                                  0x7fa32808af19 
2  g_cond_wait                  gthread-posix.c        1395 0x7fa3296cb3e6 
3  _g_dbus_worker_flush_sync    gdbusprivate.c         1766 0x7fa32a10173c 
4  g_dbus_connection_flush_sync gdbusconnection.c      1322 0x7fa32a0e478f 
5  ??                                                       0x7fa3203520fa 
6  g_local_file_move            glocalfile.c           2446 0x7fa32a118f05 
7  g_file_move                  gfile.c                3555 0x7fa32a054545 
8  move_file_prepare            caja-file-operations.c 4717 0x5142f4       
9  move_files_prepare           caja-file-operations.c 4904 0x514997       
10 move_job                     caja-file-operations.c 5025 0x514ca2       
11 io_job_thread                gioscheduler.c         85   0x7fa32a074f52 
12 g_task_thread_pool_thread    gtask.c                1304 0x7fa32a0a824a 
13 g_thread_pool_thread_proxy   gthreadpool.c          307  0x7fa3296a54ec 
14 g_thread_proxy               gthread.c              784  0x7fa3296a4f12 
15 start_thread                                             0x7fa32834c454 
16 clone                                                    0x7fa32808f7df 

[zaps166]

So the glib hangs in dbus, it waits for a notification from somewhere (a wait condition) for a long time. Commenting dbus in glib move file code fixes the problem, but it is not a solution. Why caja and similar software has this bug, but thunar and my simple example does not? Maybe caja also uses dbus and it causes a conflict? I don't know.

[protopyte]

I'm using nemo, and abnormally slow cut and pastes had me start looking for other people reporting it.

For those using nemo, it looks like activating the two panes view (F3) and moving files between the panes doesn't show the issue (or at least it feels acceptable). It might help narrowing down the issue if someone else could confirm that, and give a workaround in the meantime for some of us.

[kn00tcn]

i am possibly confirming the F3 trick works

edit: i dragged files from the parent folder on the left side into the opened destination subfolder on the right side

[kn00tcn]

problem not worked around with F3 drag, but i am doing desktop (in caja window left pane) to folder on desktop (in same caja window right pane), so maybe the fact that the icons are... on the desktop is not letting the F3 trick work this time

[ghost]

I'm having the same problem with Nemo, using Linux Mint 18.1. Version 17 never had this problem. When I move or cut/paste a series of files, Nemo gets very slow. With 5 files it's OK, with twenty files it's getting difficult, and moving 50 files or so results in Nemo hanging. This also occurs when using the two panes (F3).

[chrisamow]

Seeing this on Mint 18/caja iotop shows bursts, but typical read/write in hundreds KB/s one of the 4 cpu cores pegged at 100% by caja, harddisk graph on the system monitor near the lows continuously. Was doing a 30GB copy of files from one ssd to another ssd and it feels like it is moving over the internet but it's not :)

[ghost]

Good description chrisamow of how it feels. Files are sometimes moved four at a time, slowly, then four more and then it takes forever to move the rest or Nemo just stops.

[aucampia]

same issue Caja 1.16.0 on ubuntu 16.10 also had it on gentoo.

[ip413]

same thing - Linux Mint, Mate, Caja, Kernel 4.4.0-66-generic; copying 120 files about 1MB each takes 30 sec. on SSD? In terminal fast as hell.

[actinism]

tested myself with midnight commander and caja (shipped with mint 18.4) from ssd to a luks partition. with caja 300 files (250mb images) took literally 8mins + being stucked in the middle (2nd testrun need to kill -9 caja), with midnight commander 4secs.

this is ridiculous

[lukefromdc]

https://github.com/actinism , Can you try this with Nautilus? If Nautilus no longer has this problem than there would have to be one or more Nautilus commits that fixed it. If Nautilus still has this issue, a bug report there would reach a larger dev team and could generate a fix that could then be backported to Caja and Nemo(where I've also seen this problem).

[actinism]

@lukefromdc did some different tests with nautilus. (symlink folder to real folder etc. or native mount to native mount).

setup: 2 nautilus windows, dragging folder/files with shift-hold-click (move) from one to the other

1. moving folder "pics" to other window = almost instantly
2. ctrl-a selecting all files in pics moving to other window/folder = same behaviour/getting stucked with modem speed

System: Release Linux Mint 18 Sarah 64-bit Kernel Linux 4.8.0-39-generic x86_64 MATE 1.14.1

[kn00tcn]

i dont think it's a caja/nautilus bug directly, it's gvfs-metadata most likely

a file manager that doesnt use that is fine

[protopyte]

Trying to help narrowing down the issue... I just transferred over 4.5GB of data (a mix of folders, photos and videos) at a steady 20MB/s from a PPTP device. This is Nemo on Debian, package version 3.2.2-3, likely more recent than the one I was using in December for my previous message.

@actinism aren't you doing two very different things in each case? By moving the "pics" folder, there is just one filesystem entry which has to be changed, whereas in the second case, it has to be done for each file in the folder, so case 1 is by nature faster than case 2.

[actinism]

@sleveque Right, but physically these are two different hardware devices. /home (SSD with ext4) to a harddrive .../_test ... like @kn00tcn saying this is likely metadata or something.

I will test with some other data ...

[zaps166]

@sleveque @actinism It is not HDD/SSD or file size issue. You can move empty (0 bytes) files to activate a bug.

Gio waits for dbus (see https://github.com/mate-desktop/caja/issues/528#issuecomment-262956254). The function g_file_move() behaves weird in Caja, Nemo and similar file browsers. I wrote my tests for this function a few months ago and it worked very fast on stand-alone application. Also Thunar probably calls it (https://github.com/xfce-mirror/thunar/blob/thunar-1.6.10/thunar/thunar-transfer-job.c#L984) an it doesn't have any problems.

[protopyte]

@zaps166 yes, I am aware that even with empty files the problem can show up. My point was to highlight that PPTP, unlike regular filesystems (be it on an external device or not), didn't seem to be affected. I have no idea how PPTP works, though I guess that at some point g_file_move() is also involved.

[jeremy447]

I just stumbled across this issue by reading an unrelated reddit thread and it reminded me this gvfs issue (fixed) that can be the real problem here.

https://bugzilla.gnome.org/show_bug.cgi?id=757747

[lukefromdc]

Here's the synposis of the Nautilus change that went with the GVFS change:

"When copying, moving or linking files, original sources and their destinations are stored in an information structure used for undoing the operation. The sources and destinations were appended at the end of a list. Due to append taking linear time, there was a noticeable decrease in speed when operating with many files. In order to fix this, the sources and destinations are stored in a queue that allows appending in constant time."

Unfortunately the changes to Nautilus for this won't apply directly to Caja unless we update the entire undo/redo structure to match what Nautilus does. I did a quick mate-search-tool search of a Caja source directory for one of the variables in https://bugzilla.gnome.org/attachment.cgi?id=327593&action=diff and it was not present at all. I recall "create an undo manager" as one of the nautilus changes not brought into Caja with the GtkApplication work.

[ghost]

I have this problem for years now, has someone any solution please ?

[lukefromdc]

We could copy the entire "create an undo manager" setup, but when I first did the GtkApplication port we needed to be able to build the same code with GTK3 or GK2, and with or without libunique in GTK3. The port alone was complex enough at the time. We can go back to this item though, and iterate through the relevant commits

[monsta]

@lukefromdc: do you think we need to make any changes in Caja? I thought the fix in gvfs should be enough.