Can't read file (over 4G?)

[GoogleCodeExporter]

What steps will reproduce the problem?

1. Create 16GiB blank file with truncate.
# truncate -s 16G test

2. Compress this file with WinRAR 5.00. I used following options:
-cfg- -ep1 -htb -m5 -ma5 -mcD -md256m -ms- -mt1 -oi:0 -qo- -scu -s-

3. Mount archive with rar2fs.

4. Try to read compressed file with dd.
# dd if=(mounted path)/test of=/dev/null bs=8k count=32

What is the expected output? What do you see instead?

 - It must be read data correctly. But, it hangs. dd won't kill even sends SIGKILL. rar2fs process can be kill only with SIGKILL. dd reports 0 bytes readed.

What version of the product are you using? On what operating system?

fuse 2.9.3 (openSUSE 13.1 RC1, x86_64), unrar 5.0.12, rar2fs 1.18.0

Please provide any additional information below.

 - It seems occur with archive which has very large archived file offset. I'll attach example file which is explained above.

p.s. I'm sorry about my poor english.

Original issue reported on code.google.com by jyhpsy...@gmail.com on 4 Nov 2013 at 12:47

Attachments:

• test.rar

[GoogleCodeExporter]

Thanks for the attached file. I will try to reproduce this locally.
In general rar2fs should have no issues with large file support.
Something behaves strange also if you try 'unrar t test.rar'. Testing almost 
immediately progress to 99% but then get stuck for a very long time. It 
completes but it takes a lot longer than expected.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 1:46

• Changed state: Started

[GoogleCodeExporter]

I am having problems reproducing this issue (on SUSE11) :(

>rar2fs issue19 d
>cd d
>ls -l
ls -l
total 16777220
-rw-rw-r-- 1 foobar foobar 17179869184 Nov  4 13:22 test.blank
>dd if=test.blank of=/dev/null bs=8k count=32
32+0 records in
32+0 records out
262144 bytes (262 kB) copied, 0.00018429 s, 1.4 GB/s

Can you try eg. this instead:
>hexdump -C -n 64 test.blank

Just want to see if it makes a difference or not.

I might need some more information about exactly how you mounted it (full set 
of arguments etc.) I do not have a system with fuse 2.9 so currently I am 
unable to test if this could be a fuse issue or not. Do you have some other 
machine with a different configuration?

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 2:04

[GoogleCodeExporter]

Can you also please provide the output from

>rar2fs -V

And a copy of your config.log.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 2:12

[GoogleCodeExporter]

> Something behaves strange also if you try 'unrar t test.rar'. Testing almost 
immediately progress to 99% but then get stuck for a very long time. It 
completes but it takes a lot longer than expected.

It's normal. Because compression ratio(compressed size / original size) is 
almost 0. Criteria of WinRAR's percentage indicator is archive's file 
pointer(including recovery record), not the uncompressed data size. Testing 
archive that have 10% RR, you can see extraction process is done at 90%.

Explanation of whole steps:

Working directory : /dev/shm/test

1. Prepare unrar library. I used manually-made static version of libunrar to 
prevent shared library path problems.
# tar -xf unrarsrc-5.0.12.tar.gz
# cd unrar
# make lib
# ar -r libunrar.a rar.o strlist.o strfn.o pathfn.o smallfn.o global.o file.o 
filefn.o filcreat.o archive.o arcread.o unicode.o system.o isnt.o crypt.o crc.o 
rawread.o encname.o resource.o match.o timefn.o rdwrfn.o consio.o options.o 
errhnd.o rarvm.o secpassword.o rijndael.o getbits.o sha1.o sha256.o blake2s.o 
hash.o extinfo.o extract.o volume.o list.o find.o unpack.o headers.o 
threadpool.o rs16.o cmddata.o filestr.o scantree.o dll.o qopen.o
# rm libunrar.so
# cd ..

 - I copied command line that create shared object printed by make, and just modified command to create static library.

2. Compile rar2fs.
# tar -xf rar2fs-1.18.0.tar.gz
# cd rar2fs-1.18.0
# LIBS='-lstdc++ -lpthread' ./configure --with-unrar=../unrar 
--with-unrar-lib=../unrar
# make
# cd ..

 - LIBS variable is required because currently used unrar library is static. Without this, configure script failed detection of unrar. With shared version of libunrar, it doesn't occur.

3. Mount filesystem and read data

# mkdir a
# rar2fs-1.18.0/rar2fs test.rar a
# dd if=a/test.blank of=/dev/null bs=8k count=32

It hangs. For terminating this process, I tried following:

# ps aux | grep rar
root     26945  0.0  0.0 287612  6604 ?        Ssl  01:23   0:00 
rar2fs-1.18.0/rar2fs test.rar a
root     26951  0.0  0.5 557612 276292 ?       S    01:24   0:00 
rar2fs-1.18.0/rar2fs test.rar a
root     27012  0.0  0.0  14432  1028 pts/5    S+   01:24   0:00 grep 
--color=auto rar
# kill -9 26951

After sending SIGKILL, dd outputs:

0+0 records in
0+0 records out
0 bytes (0 B) copied, 36.4623 s, 0.0 kB/s

# hexdump -C -n 64 test.blank

It hangs, too. After kill rar2fs, it doesn't print anything.

# uname -a
Linux localhost 3.11.3-1-desktop #1 SMP PREEMPT Wed Oct 2 05:54:02 UTC 2013 
(485f0d1) x86_64 x86_64 x86_64 GNU/Linux

# rar2fs-1.18.0/rar2fs -V
rar2fs v1.18.0 (DLL version 6)    Copyright (C) 2009-2013 Hans Beckerus
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it under
certain conditions; see <http://www.gnu.org/licenses/> for details.
FUSE library version: 2.9.3
fusermount version: 2.9.3
using FUSE kernel interface version 7.19

I'll attach my config.log. I'm not have another environment that can be test it 
yet, but it will be possible soon via VM...

Original comment by jyhpsy...@gmail.com on 4 Nov 2013 at 4:36

Attachments:

• config.log

[GoogleCodeExporter]

I used shared version of libunrar, too. Only difference is below:

# LD_LIBRARY_PATH=unrar rar2fs-1.18.0/rar2fs test.rar a

Results are same. It hangs. It causes hang all process which accesses mounted 
path, Even bash and xfce-terminal when press Tab key to use auto-complete!

I'll attach config.log with shared library, too.

Original comment by jyhpsy...@gmail.com on 4 Nov 2013 at 4:51

Attachments:

• config.log

[GoogleCodeExporter]

First of all, why would you need to build statically and why would not doing so 
result in failures in detecting libunrar.so? If you already have an old version 
of libunrar.so installed, all you have to do to override it at 
compile/configure time is by using --with-unrar-lib=<path>. Then, of course, 
you need to be careful when running it since rar2fs will pick up the library as 
provided by the system unless you override it using eg. LD_LIBRARY_PATH.
Secondly, can you please try to not mount the file but rather the folder that 
contains it? I think I managed to reproduce the problem when mounting the file 
only.
So try moving test.rar into eg. a folder named s and try:

>rar2fs s a

I am almost certain that it will work. I need to look into why mounting the 
file rather than the folder does not work in this particular case.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 5:05

[GoogleCodeExporter]

1. Yes, I tested case of shared library, too. It mentioned in #5. And, it 
should be work regardless unrar library is shared or static if it compiled 
sucessfully. Both cases --with-unrar and --with-unrar-lib option is required.

2. What's difference between my mount command and yours?
# rar2fs-1.18.0/rar2fs test.rar a
> rar2fs s a

Only difference is executive binary path, not archive.

Original comment by jyhpsy...@gmail.com on 4 Nov 2013 at 5:29

[GoogleCodeExporter]

l. Yes, --with-unrar tell where to find the header files (source), 
--with-unrar-lib specifies where to search for libunrar.so besides the default 
system paths.

2. It is very different. rar2fs can mount in two different ways, as a file or 
as a directory. What you are doing is mounting one (1) single .rar file. What I 
was asking you to do is to mount a directory (containing test.rar) instead.
To be more precise, try making two folders; a1 and a2. Then copy test.rar to a1 
and mount a1 on a2.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 5:53

[GoogleCodeExporter]

I have done some basic trouble shooting and this looks like a tricky problem :( 
Not something I will manage to resolve quickly I am afraid. For some reason the 
communication between the process extracting data and the thread waiting for it 
does not work when using file mount. But only for this file? I have lots of 
other files for which file mount work without problem. 

Can you create one more *small* file (e.g. 1GB) using the same method and 
attach it here. I need to tell if it is the size of the file or something else 
that is causing this. 

Also, please try the directory mount instead to see if it can be used as a 
temporary workaround.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 6:28

[GoogleCodeExporter]

I know a bit more about the problem now, but still I do not have a solution.
It seems to be related to RAR5 archives only. I can reproduce this problem 
easily using any RAR5 archive. For non-RAR5 archives it works just fine!? So 
you do not need to create another file. It is really weird since mounting a 
folder containing the file instead of mounting the file itself also works 
without problems!?

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 8:36

[GoogleCodeExporter]

Got it (I think)! It was a missing close of a file handle before freeing some 
resources that resulted in some major breakage in libunrar. Only seems to 
affect RAR5 archives though which is most likely the reason why this error has 
passed unnoticed. I will commit a new version to trunk. Can you test latest 
version in SVN to verify if the problem is resolved or not in your environment?

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 10:05

• Changed state: Fixed

[GoogleCodeExporter]

Yes, but I can test svn build 12 hours later, due to work... :-(

I'm sorry about misunderstanding your suggestion. I don't know it is possible 
to mount directory containing RAR file... ;-)

Original comment by jyhpsy...@gmail.com on 4 Nov 2013 at 11:26

[GoogleCodeExporter]

No problem. No need to rush it ;) I am confident that it will work for you this 
time.
And yes, it is possible to mount a directory containing not only one (1) RAR 
file, but many, in addition to normal files as well, completely transparent.

Original comment by hans.bec...@gmail.com on 4 Nov 2013 at 11:44

[GoogleCodeExporter]

Good, svn version works! Thanks.

Original comment by jyhpsy...@gmail.com on 5 Nov 2013 at 11:52

[GoogleCodeExporter]

Great! Thanks for reporting the issue and taking your time testing the patch!

Case closed.

Original comment by hans.bec...@gmail.com on 5 Nov 2013 at 12:18

• Changed state: Verified