Slow navigation on NAS

[Beep6581]

Originally reported on Google Code with ID 1823

Whenever I browse directories in mapped drive on NAS (Synology DS211+) RT is experiencing
an unreasonably long delay in file browser.
When browsing through windows Explorer, everything is fast - directory navigation/thumbnail
preview.

Using the latest RT of course.

Reported by michaelezra000 on 2013-04-01 16:54:00

[Beep6581]

Build RT with MMAP off, is it better?

Reported by natureh.510 on 2013-04-01 17:25:42

[Beep6581]

What is a mapped drive? SUBST ?

Reported by natureh.510 on 2013-04-01 17:26:34

[Beep6581]

mapped drive in my case:
\\Share1 (\\192.168.0.5) (Y:)
The network path: \\192.168.0.5\Share1

MMAP off did not help

Reported by michaelezra000 on 2013-04-01 17:58:57

[Beep6581]

So you have used Y: and not the \\Share1 path?

Reported by natureh.510 on 2013-04-01 18:52:31

[Beep6581]

yes

Reported by michaelezra000 on 2013-04-01 20:18:28

[Beep6581]

But I just checked that accessing it via \\192.168.0.5\Share1 is slow as well..

Reported by michaelezra000 on 2013-04-01 20:22:11

[Beep6581]

That is because RT is not optimized for disk traffic, and you especially experience
the slowness because of the network inbetween.
See issue 622.
So it's not really a bug I think.

Reported by oduis@hotmail.com on 2013-04-02 06:20:07

[Beep6581]

Hi Olli, thanks for the reference issue. This really sucks:) I bought a pretty fast
NAS with RAID 1 redundancy @ 1Gigabit cabling, moved all family images there, hoping
to be able go through it with RT. 
BTW, do you know where most of the IO is - with the raw files or with the pp3 files?
(My cache is on SSD).

Reported by michaelezra000 on 2013-04-02 11:38:47

[Beep6581]

It's my tiff32 patch that sucks! I wondered why the thumbnail loading was so slow now,
but thanks to you and oliver, the problem is identified and will be solved soon. In
fact, i'm reading and writing the 16/32 bits rtti image line per line through fwrite.
It generates a disk access on each call!

I'll use a mapped file for reading the data, but it would be nice if the dev that created
the myfile.cc/h files could enhance it for writing operations too. In the mean time,
i'll handle a temporary buffer where necessary since i know the length by advance.

If you open a directory with only 8bits images in it, it should be faster. Can you
confirm?

Reported by natureh.510 on 2013-04-02 13:39:28

• Status changed: Started
• Labels added: Priority-High
• Labels removed: Priority-Low

[Beep6581]

Hombre, I am glad you were able to identify where the problem is. I will test with 8-bit
images in the evening (EST) when @my machine&NAS.

Reported by michaelezra000 on 2013-04-02 13:54:56

[Beep6581]

Michael, the problem introduced by my tiff32 patch concerns the cached images (*.rtti
files). In your case, they should be read quite fast since they are on SSD.

So i'm afraid that the upcoming patch won't solve this problem.

Could you quantify the "unreasonably long delay" you're talking about, compared to
the same directory content but on HD?

Reported by natureh.510 on 2013-04-03 01:19:02

[Beep6581]

Hi Hombre, I am at the computer now, testing:)

Reported by michaelezra000 on 2013-04-03 01:27:32

[Beep6581]

"unreasonably long delay" - that is about 30 seconds when I double-click on the folder
to view whats inside. During this time nothing happens, then thumbnails get drawn.

The speed at which thumbnails get drawn is not too bad:) It is the delay when nothing
seems to happen (sort of). 

Only at the end of this delay I hear the hard drives crunching on NAS, so there must
be data reading going at that time, but before that, nothing - feels like a hang.

Delay is the same when navigating to a folder1 with 100 jpgs or to folder2 with 100
developed raws.

Reported by michaelezra000 on 2013-04-03 01:37:30

[Beep6581]

BTW, navigating to an empty NAS folder is fast.
If I click on RT window during that "delay", I get a busy windows cursor, RT is not
responding. After a while it loads thumbs and keeps working.

Reported by michaelezra000 on 2013-04-03 01:41:00

[Beep6581]

Do you have the same kind of delay when trying to open an image with e.g. Gimp?

Reported by natureh.510 on 2013-04-03 01:43:06

[Beep6581]

I don't have GIMP, but when I use Bridge CD6, no delay - it just goes into folder. The
same with windows explorer - no delay in browsing, feels almost like a local drive.

Reported by michaelezra000 on 2013-04-03 01:45:46

[Beep6581]

Hombre, I am in IRC, if you like to chat faster

Reported by michaelezra000 on 2013-04-03 01:47:10

[Beep6581]

It looks like RT is looking for one or more remote files that does not exist, this process
generally takes time. It would be nice to find out which, if that's the problem...

Could you try e.g. "Process Monitor" to listen to the disk & network activity?

Reported by natureh.510 on 2013-04-03 01:53:31

[Beep6581]

Reported by michaelezra000 on 2013-04-03 03:46:10

---

• _Attachment: RT_File_IO_Log01.txt_

[Beep6581]

Michael, I have a NAS (Netgear Ultra 2, 1 Gbps) in Raid 1 where I keep my backups and
documents. All images are normally on a separate ssd. 

I started RT and navigated to my images on the NAS (ie the backups) to a catalogue
with 30 images. I noticed about five seconds of "inactivity" from the double click
on a folder until RT started drawing the thumbs. 

I tried to enter another catalogue where I put some 500 images, the delay now was about
15-20 seconds. 
While building the thumbnails, stat a lot of entries like this:
=================
read(4, "\1\0\0\0\0\0\0\0", 16)         = 8
recvfrom(3, 0x1a241c4, 4096, 0, 0, 0)   = -1 EAGAIN (Resource temporarily unavailable)
poll([{fd=4, events=POLLIN}, {fd=3, events=POLLIN}, {fd=6, events=0}, {fd=5, events=POLLIN}],
4, 2688) = 1 ([{fd=4, revents=POLLIN}])
=================

It seems to me things move faster here, though I can't explain why. The stat-entries,
on the other side, makes me think it's quite a lot of babbling going on on the network?

Johan

Reported by johan@birkagatan.com on 2013-04-03 09:17:04

[Beep6581]

In my case computer and NAS are on the same 1Gb switch with 3 feet Cat6 cables each.
On the same switch I also have a blue-ray player, printer and a wi-fi router, all of
them should not be doing anything at the time of testing.
NAS was configured for Jumbo frames MTU 9000
I retested without Jumbo frames and MTU 1500, without seeing any difference is "inactivity"
delay.

Reported by michaelezra000 on 2013-04-03 11:13:35

[Beep6581]

Googling for "C:\Windows\CSC\" ... CSC is a caching mechanism between the NAS and the
client: i guess it make sure that files are in synch, and eventually transfert them,
hence the delay.

Reported by natureh.510 on 2013-04-03 17:25:22

[Beep6581]

Here is a patch that use a buffer (through IMFILE and std::ofstream) to read and write
the rtti file in a single shot. I'd like to know if you experience a speedup or slow
down with it. I think it'll make no difference with your SSD Michael :).

Test with special chars in the filename too, i'm not sure that safe_locale_from_ut8
is sufficient to handle all cases. Would be nice if a japanese (mattintosh?) could
test it too.

Reported by natureh.510 on 2013-04-03 19:42:30

---

• Attachment: [1823-Less file operations02.patch](https://storage.googleapis.com/google-code-attachments/rawtherapee/issue-1823/comment-25/1823-Less file operations02.patch)

[Beep6581]

Thanks, Hombre, I will test on Friday with access to my system and let you know.

Reported by michaelezra000 on 2013-04-04 03:23:58

[Beep6581]

Hi Hombre, I just got to try this patch. Unfortunately, it does not change the long
delay before appearance of the thumbs from NAS, so it is not the cache issue, but issue
with browsing to the images themselves.

Reported by michaelezra000 on 2013-04-04 23:30:59

[Beep6581]

I tested this patch with a Folder and Filename in Russian language, all files in cache
were written correctly

Reported by michaelezra000 on 2013-04-04 23:47:23

[Beep6581]

That's fine then.

For your delay problem, see comment #24. If possible, try to disable caching for your
images's root folder, if that's help.

Reported by natureh.510 on 2013-04-05 07:27:13

[Beep6581]

I don't have cashing enabled, I verified.

Reported by michaelezra000 on 2013-04-05 15:26:09

[Beep6581]

Just found that there are more logs available on issue 622

Reported by michaelezra000 on 2013-05-01 01:54:12

[Beep6581]

Hi Michael, have you found a solution for this issue?

Re #30:

"C:\Windows\CSC" is the folder used by Windows for its "Offline Files" caching mechanism.
Here is a link that may be interesting: http://social.technet.microsoft.com/Forums/windows/en-US/0578dc8b-1dc8-4380-b3a5-60a59ccc4f74/offline-files-very-slow?forum=w7itproperf

Just to make sure that this mechanism is disabled, have you followed those instructions?:
- WindowsXP: http://offlinefiles.blogspot.fr/2009/11/offline-files-and-folder-redirection-xp.html
- Windows 7: http://www.windowstipspage.com/offline-files-folder-redirection-windows-7/

Reported by natureh.510 on 2013-10-11 14:59:44

[Beep6581]

HI Hombre, yes issue is still there.
In my setup, Offline files and folder redirection is disabled.

The offline files are very slow only via RT, they are fast via file manager, Adobe
Bridge.

About the 1-st link this is interesting: http://blogs.technet.com/b/askds/archive/2009/06/26/debunking-the-vista-remote-differential-compression-myth.aspx

Reported by michaelezra000 on 2013-10-11 16:04:47