search

glandium / vmfs-tools

http://glandium.org/projects/vmfs-tools/
GNU General Public License v2.0
76 stars 29 forks source link

Handling of sparse files #2

Closed gabbe closed 12 years ago

gabbe commented 12 years ago

Since my .vmdk-files where created with thin provisioning, they are stored as sparse files. When I try to use tar with the --sparse option on a filesystem mounted with vmfs-fuse, the compression is impressive - but not realistic. A 100Gb virtual disk (containing about 2.2 Gb data) resulted in a tar file of 10240 bytes, and that is without gzip compression... I now realize that "du" reports that the files uses 0 bytes on disk. If I use tar without the "--sparse" option, tar works fine but takes a very long time since it has to read the whole 100Gb and compress it to reduce the size of all those zeros. Is this a known problem?

I am using vmfs-tools 0.2.1-1 on a ubuntu 11.10 rescue remix.

glandium commented 12 years ago

Can you try current again with current master?

gabbe commented 12 years ago

Of course, thanks for the quick answer.

Sorry, I get the same behavior with the latest version from git. I mount the file system with: ./vmfs-fuse /dev/sda3 /mnt and try du -hs /mnt/* and every single directory is reported to use 0 bytes on disk.

glandium commented 12 years ago

With the master version, can you try and report the output for the following commands: debugvmfs /dev/sda3 show 'inode["thin-provisioned-file-name"]' debugvmfs /dev/sda3 show 'inode["thin-provisioned-file-name"].blocks'

gabbe commented 12 years ago 
root@ubuntu:/mnt# /root/vmfs-tools/debugvmfs/debugvmfs /dev/sda3 show 'inode["OraXP/OraXP-flat.vmdk"]'
         ID: 0x198016c4
        ID2: 0xed
      Links: 1
       Type: 3
      Flags: 0
       Size: 100 GiB
 Block size: 1 MiB
Block count: 5340
        UID: 0
        GID: 0
       Mode: 0600 (-rw-------)
        ZLA: 3
        TBZ: 0
        COW: 0
Access Time: 2011-10-23 05:51:30
Modify Time: 2011-10-21 13:33:53
Change Time: 2011-10-20 17:43:32
     RDM ID: 0x0
root@ubuntu:/mnt# /root/vmfs-tools/debugvmfs/debugvmfs /dev/sda3 show 'inode["OraXP/OraXP-flat.vmdk"].blocks'
0xa0001a03 0xb0001a03 0xc0001a03 0xd0001a03
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0xe0001a03
0x20001fc3 0x30001fc3 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0xf0001a03 0x20001a43 0x40001fc3
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00000000
0x00000000 0x00000000 0x00000000 0x00001a43
gabbe commented 12 years ago

But the "thin-provisioning" might be a red herring. I get the same behavior for every single file that exists on the filesystem. Even small script files that I created with "vi" in VmWare maintenance mode shows 0 bytes used when queried by "du".

gabbe commented 12 years ago 
root@ubuntu:/mnt# ls -l files_frigg.sh 
-rwxr--r-- 1 root root 45601 Oct 21 17:37 files_frigg.sh
root@ubuntu:/mnt# du files_frigg.sh 
0   files_frigg.sh
gabbe commented 12 years ago

In the following example, /tmp is on another filesystem (not vmfs):

gabbe commented 12 years ago 
root@ubuntu:/mnt# file files_frigg.sh 
files_frigg.sh: ASCII C++ program text
root@ubuntu:/mnt# tar cSf /tmp/ff.tar files_frigg.sh 
root@ubuntu:/mnt# cd /tmp
root@ubuntu:/tmp# tar xvf ff.tar 
files_frigg.sh
root@ubuntu:/tmp# file files_frigg.sh 
files_frigg.sh: data
root@ubuntu:/tmp# ls -l files_frigg.sh 
-rwxr--r-- 1 root root 45601 Oct 21 17:37 files_frigg.sh
root@ubuntu:/tmp# du files_frigg.sh 
0   files_frigg.sh
root@ubuntu:/tmp# cp /mnt/files_frigg.sh .
root@ubuntu:/tmp# ls -l files_frigg.sh
-rwxr--r-- 1 root root 45601 Oct 24 13:00 files_frigg.sh
root@ubuntu:/tmp# du files_frigg.sh 
48  files_frigg.sh
glandium commented 12 years ago

What do the following commands say?

(with /mnt being the mounted vmfs-fuse ; don't bother copy/pasting the actual result of cat, I just need to know if there is something or nothing at all)

glandium commented 12 years ago

Also add the following:

This last one should display more than lines of 00 00 00 00 00 (...).

gabbe commented 12 years ago 
root@ubuntu:/mnt# /root/vmfs-tools/debugvmfs/debugvmfs /dev/sda3 show 'inode["files_frigg.sh"]'
         ID: 0x12c016c4
        ID2: 0x121
      Links: 1
       Type: 3
      Flags: 0
       Size: 44.53 KiB
 Block size: 64 KiB
Block count: 1
        UID: 0
        GID: 0
       Mode: 100744-rwxr--r--)
        ZLA: 2
        TBZ: 0
        COW: 0
Access Time: 2011-10-23 21:55:32
Modify Time: 2011-10-21 17:37:57
Change Time: 2011-10-21 17:38:35
     RDM ID: 0x0
root@ubuntu:/mnt# cat /mnt/files_frigg.sh
<loads of text, correct output>
root@ubuntu:/mnt# du /mnt/OraXP/OraXP-flat.vmdk 
0       /mnt/OraXP/OraXP-flat.vmdk
root@ubuntu:/mnt# hd /mnt/OraXP/OraXP-flat.vmdk
00000010  bf 1b 06 50 57 b9 e5 01  f3 a4 cb bd be 07 b1 04  |...PW...........|
00000020  38 6e 00 7c 09 75 13 83  c5 10 e2 f4 cd 18 8b f5  |8n.|.u..........|
00000030  83 c6 10 49 74 19 38 2c  74 f6 a0 b5 07 b4 07 8b  |...It.8,t.......|
00000040  f0 ac 3c 00 74 fc bb 07  00 b4 0e cd 10 eb f2 88  |..<.t...........|
00000050  4e 10 e8 46 00 73 2a fe  46 10 80 7e 04 0b 74 0b  |N..F.s*.F..~..t.|
00000060  80 7e 04 0c 74 05 a0 b6  07 75 d2 80 46 02 06 83  |.~..t....u..F...|
00000070  46 08 06 83 56 0a 00 e8  21 00 73 05 a0 b6 07 eb  |F...V...!.s.....|
00000080  bc 81 3e fe 7d 55 aa 74  0b 80 7e 10 00 74 c8 a0  |..>.}U.t..~..t..|
<snip>
glandium commented 12 years ago

So, there would be two different issues. One is that "du" doesn't return the right value. The other comes from tar probably not being reasonable, and/or fuse not helping. Could you give the output for the following command:

gabbe commented 12 years ago 
root@ubuntu:/mnt# strace -cZf /tmp/foo.tar /mnt/files_frigg.sh
execve("/bin/tar", ["tar", "-cZf", "/tmp/foo.tar", "/mnt/files_frigg.sh"], [/* 26 vars */]) = 0
brk(0)                                  = 0x929a000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77ac000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb77a6000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/librt.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\30\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=30684, ...}) = 0
mmap2(NULL, 33364, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xd2b000
mmap2(0xd32000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0xd32000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\222\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1544392, ...}) = 0
mmap2(NULL, 1554968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x8da000
mmap2(0xa50000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x176) = 0xa50000
mmap2(0xa53000, 10776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xa53000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200[\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=124745, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77a5000
mmap2(NULL, 107016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x1b9000
mmap2(0x1d0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0x1d0000
mmap2(0x1d2000, 4616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x1d2000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77a4000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb77a46c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x1d0000, 4096, PROT_READ)     = 0
mprotect(0xa50000, 8192, PROT_READ)     = 0
mprotect(0xd32000, 4096, PROT_READ)     = 0
mprotect(0x8089000, 4096, PROT_READ)    = 0
mprotect(0x47a000, 4096, PROT_READ)     = 0
munmap(0xb77a6000, 20529)               = 0
set_tid_address(0xb77a4728)             = 15592
set_robust_list(0xb77a4730, 0xc)        = 0
futex(0xbf8cc1fc, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xbf8cc1fc, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b77a46c0) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x1be530, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x1be5b0, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="ubuntu", ...}) = 0
clock_gettime(CLOCK_REALTIME, {1319466566, 237570654}) = 0
brk(0)                                  = 0x929a000
brk(0x92bb000)                          = 0x92bb000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1607664, ...}) = 0
mmap2(NULL, 1607664, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb761b000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=2570, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77ab000
read(3, "# Locale name alias data base.\n#"..., 1024) = 1024
read(3, " entries are case independent.\n\n"..., 1024) = 1024
read(3, "R\nko_KR\t\tko_KR.eucKR\nlithuanian "..., 1024) = 522
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0xb77ab000, 4096)                = 0
open("/usr/lib/locale/sv_SE.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv_SE.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv_SE/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
fcntl64(0, F_GETFD)                     = 0
fcntl64(1, F_GETFD)                     = 0
fcntl64(2, F_GETFD)                     = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [CHLD], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
pipe([3, 4])                            = 0
clone(child_stack=0, flags=CLONE_CHILD_CLEARTID|CLONE_CHILD_SETTID|SIGCHLD, child_tidptr=0xb77a4728) = 15593
close(3)                                = 0
fstat64(4, {st_mode=S_IFIFO|0600, st_size=0, ...}) = 0
clock_gettime(CLOCK_REALTIME, {1319466566, 240572329}) = 0
write(2, "tar: ", 5)                    = 5
write(2, "Removing leading `/' from member"..., 38) = 38
write(2, "\n", 1)                       = 1
fstatat64(AT_FDCWD, "/mnt/files_frigg.sh", {st_mode=S_IFREG|0744, st_size=45601, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/mnt/files_frigg.sh", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW|O_CLOEXEC) = 3
fstat64(3, {st_mode=S_IFREG|0744, st_size=45601, ...}) = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
open("/etc/nsswitch.conf", O_RDONLY)    = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=475, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb77ab000
read(5, "# /etc/nsswitch.conf\n#\n# Example"..., 1024) = 475
read(5, "", 1024)                       = 0
close(5)                                = 0
munmap(0xb77ab000, 4096)                = 0
open("/etc/ld.so.cache", O_RDONLY)      = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 5, 0) = 0xb77a6000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_compat.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\16\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=34648, ...}) = 0
mmap2(NULL, 37460, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x110000
mmap2(0x118000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x7) = 0x118000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnsl.so.1", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2001\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=92020, ...}) = 0
mmap2(NULL, 100328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0xdd0000
mmap2(0xde5000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x15) = 0xde5000
mmap2(0xde7000, 6120, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0xde7000
close(5)                                = 0
mprotect(0xde5000, 4096, PROT_READ)     = 0
mprotect(0x118000, 4096, PROT_READ)     = 0
munmap(0xb77a6000, 20529)               = 0
open("/etc/ld.so.cache", O_RDONLY)      = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 5, 0) = 0xb77a6000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_nis.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\31\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=42656, ...}) = 0
mmap2(NULL, 45632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x151000
mmap2(0x15b000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x9) = 0x15b000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_files.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\32\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=46736, ...}) = 0
mmap2(NULL, 49880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x3a5000
mmap2(0x3b0000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0xa) = 0x3b0000
close(5)                                = 0
mprotect(0x3b0000, 4096, PROT_READ)     = 0
mprotect(0x15b000, 4096, PROT_READ)     = 0
munmap(0xb77a6000, 20529)               = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
fcntl64(5, F_GETFD)                     = 0x1 (flags FD_CLOEXEC)
_llseek(5, 0, [0], SEEK_CUR)            = 0
fstat64(5, {st_mode=S_IFREG|0644, st_size=1106, ...}) = 0
mmap2(NULL, 1106, PROT_READ, MAP_SHARED, 5, 0) = 0xb77ab000
_llseek(5, 1106, [1106], SEEK_SET)      = 0
munmap(0xb77ab000, 1106)                = 0
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 5
_llseek(5, 0, [0], SEEK_CUR)            = 0
fstat64(5, {st_mode=S_IFREG|0644, st_size=743, ...}) = 0
mmap2(NULL, 743, PROT_READ, MAP_SHARED, 5, 0) = 0xb77ab000
_llseek(5, 743, [743], SEEK_SET)        = 0
munmap(0xb77ab000, 743)                 = 0
close(5)                                = 0
read(3, "ftpput -v -u admin -p xxxxxx rac"..., 9728) = 9728
write(4, "mnt/files_frigg.sh\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240
read(3, "cl-000001-delta.vmdk\" \"Seeskn157"..., 10240) = 10240
write(4, "cl-000001-delta.vmdk\" \"Seeskn157"..., 10240) = 10240
read(3, "mdk\" \nftpput -v -u admin -p xxxx"..., 10240) = 10240
write(4, "mdk\" \nftpput -v -u admin -p xxxx"..., 10240) = 10240
read(3, "2 x64 template (cl.vmdk\" \nftpput"..., 10240) = 10240
write(4, "2 x64 template (cl.vmdk\" \nftpput"..., 10240) = 10240
read(3, "v -u admin -p xxxxxx rackstation"..., 5153) = 5153
fstat64(3, {st_mode=S_IFREG|0744, st_size=45601, ...}) = 0
close(3)                                = 0
write(4, "v -u admin -p xxxxxx rackstation"..., 10240) = 10240
clock_gettime(CLOCK_REALTIME, {1319466566, 250676588}) = 0
clock_gettime(CLOCK_REALTIME, {1319466566, 250721074}) = 0
close(4)                                = 0
waitpid(15593, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0) = 15593
--- SIGCHLD (Child exited) @ 0 (0) ---
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
glandium commented 12 years ago

Ah sorry, I typoed. That was meant to be "tar -cSf", not "tar -cZf"

gabbe commented 12 years ago

I was scratching my head but thought that you might want to compare the two results ;-)

gabbe commented 12 years ago 
root@ubuntu:/tmp# strace tar -cSf /tmp/foo.tar /mnt/files_frigg.sh
execve("/bin/tar", ["tar", "-cSf", "/tmp/foo3.tar", "/mnt/files_frigg.sh"], [/* 26 vars */]) = 0
brk(0)                                  = 0x949c000
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
mmap2(NULL, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d5000
access("/etc/ld.so.preload", R_OK)      = -1 ENOENT (No such file or directory)
open("/etc/ld.so.cache", O_RDONLY)      = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb78cf000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/librt.so.1", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\260\30\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0644, st_size=30684, ...}) = 0
mmap2(NULL, 33364, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0xb46000
mmap2(0xb4d000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x6) = 0xb4d000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libc.so.6", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0p\222\1\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=1544392, ...}) = 0
mmap2(NULL, 1554968, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x819000
mmap2(0x98f000, 12288, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x176) = 0x98f000
mmap2(0x992000, 10776, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x992000
close(3)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libpthread.so.0", O_RDONLY) = 3
read(3, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200[\0\0004\0\0\0"..., 512) = 512
fstat64(3, {st_mode=S_IFREG|0755, st_size=124745, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78ce000
mmap2(NULL, 107016, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x110000
mmap2(0x127000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x16) = 0x127000
mmap2(0x129000, 4616, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x129000
close(3)                                = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78cd000
set_thread_area({entry_number:-1 -> 6, base_addr:0xb78cd6c0, limit:1048575, seg_32bit:1, contents:0, read_exec_only:0, limit_in_pages:1, seg_not_present:0, useable:1}) = 0
mprotect(0x127000, 4096, PROT_READ)     = 0
mprotect(0x98f000, 8192, PROT_READ)     = 0
mprotect(0xb4d000, 4096, PROT_READ)     = 0
mprotect(0x8089000, 4096, PROT_READ)    = 0
mprotect(0x384000, 4096, PROT_READ)     = 0
munmap(0xb78cf000, 20529)               = 0
set_tid_address(0xb78cd728)             = 15632
set_robust_list(0xb78cd730, 0xc)        = 0
futex(0xbf9f227c, FUTEX_WAKE_PRIVATE, 1) = 0
futex(0xbf9f227c, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 1, NULL, b78cd6c0) = -1 EAGAIN (Resource temporarily unavailable)
rt_sigaction(SIGRTMIN, {0x115530, [], SA_SIGINFO}, NULL, 8) = 0
rt_sigaction(SIGRT_1, {0x1155b0, [], SA_RESTART|SA_SIGINFO}, NULL, 8) = 0
rt_sigprocmask(SIG_UNBLOCK, [RTMIN RT_1], NULL, 8) = 0
getrlimit(RLIMIT_STACK, {rlim_cur=8192*1024, rlim_max=RLIM_INFINITY}) = 0
uname({sys="Linux", node="ubuntu", ...}) = 0
clock_gettime(CLOCK_REALTIME, {1319467468, 401344319}) = 0
brk(0)                                  = 0x949c000
brk(0x94bd000)                          = 0x94bd000
open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=1607664, ...}) = 0
mmap2(NULL, 1607664, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7744000
close(3)                                = 0
open("/usr/share/locale/locale.alias", O_RDONLY) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=2570, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d4000
read(3, "# Locale name alias data base.\n#"..., 1024) = 1024
read(3, " entries are case independent.\n\n"..., 1024) = 1024
read(3, "R\nko_KR\t\tko_KR.eucKR\nlithuanian "..., 1024) = 522
read(3, "", 1024)                       = 0
close(3)                                = 0
munmap(0xb78d4000, 4096)                = 0
open("/usr/lib/locale/sv_SE.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv_SE.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv_SE/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/lib/locale/sv/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv_SE/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv.UTF-8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv.utf8/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
open("/usr/share/locale-langpack/sv/LC_MESSAGES", O_RDONLY) = -1 ENOENT (No such file or directory)
fcntl64(0, F_GETFD)                     = 0
fcntl64(1, F_GETFD)                     = 0
fcntl64(2, F_GETFD)                     = 0
rt_sigaction(SIGCHLD, {SIG_DFL, [CHLD], SA_RESTART}, {SIG_DFL, [], 0}, 8) = 0
open("/tmp/foo3.tar", O_WRONLY|O_CREAT|O_TRUNC|O_LARGEFILE, 0666) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=0, ...}) = 0
clock_gettime(CLOCK_REALTIME, {1319467468, 404391459}) = 0
write(2, "tar: ", 5)                    = 5
write(2, "Removing leading `/' from member"..., 38) = 38
write(2, "\n", 1)                       = 1
fstatat64(AT_FDCWD, "/mnt/files_frigg.sh", {st_mode=S_IFREG|0744, st_size=45601, ...}, AT_SYMLINK_NOFOLLOW) = 0
openat(AT_FDCWD, "/mnt/files_frigg.sh", O_RDONLY|O_NOCTTY|O_NONBLOCK|O_LARGEFILE|O_NOFOLLOW|O_CLOEXEC) = 4
fstat64(4, {st_mode=S_IFREG|0744, st_size=45601, ...}) = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
open("/etc/nsswitch.conf", O_RDONLY)    = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=475, ...}) = 0
mmap2(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0xb78d4000
read(5, "# /etc/nsswitch.conf\n#\n# Example"..., 1024) = 475
read(5, "", 1024)                       = 0
close(5)                                = 0
munmap(0xb78d4000, 4096)                = 0
open("/etc/ld.so.cache", O_RDONLY)      = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 5, 0) = 0xb78cf000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_compat.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0P\16\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=34648, ...}) = 0
mmap2(NULL, 37460, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x12b000
mmap2(0x133000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x7) = 0x133000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnsl.so.1", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\2001\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=92020, ...}) = 0
mmap2(NULL, 100328, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x135000
mmap2(0x14a000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x15) = 0x14a000
mmap2(0x14c000, 6120, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x14c000
close(5)                                = 0
mprotect(0x14a000, 4096, PROT_READ)     = 0
mprotect(0x133000, 4096, PROT_READ)     = 0
munmap(0xb78cf000, 20529)               = 0
open("/etc/ld.so.cache", O_RDONLY)      = 5
fstat64(5, {st_mode=S_IFREG|0644, st_size=20529, ...}) = 0
mmap2(NULL, 20529, PROT_READ, MAP_PRIVATE, 5, 0) = 0xb78cf000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_nis.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\200\31\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=42656, ...}) = 0
mmap2(NULL, 45632, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x56c000
mmap2(0x576000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0x9) = 0x576000
close(5)                                = 0
access("/etc/ld.so.nohwcap", F_OK)      = -1 ENOENT (No such file or directory)
open("/lib/i386-linux-gnu/libnss_files.so.2", O_RDONLY) = 5
read(5, "\177ELF\1\1\1\0\0\0\0\0\0\0\0\0\3\0\3\0\1\0\0\0\340\32\0\0004\0\0\0"..., 512) = 512
fstat64(5, {st_mode=S_IFREG|0644, st_size=46736, ...}) = 0
mmap2(NULL, 49880, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 5, 0) = 0x58e000
mmap2(0x599000, 8192, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 5, 0xa) = 0x599000
close(5)                                = 0
mprotect(0x599000, 4096, PROT_READ)     = 0
mprotect(0x576000, 4096, PROT_READ)     = 0
munmap(0xb78cf000, 20529)               = 0
open("/etc/passwd", O_RDONLY|O_CLOEXEC) = 5
fcntl64(5, F_GETFD)                     = 0x1 (flags FD_CLOEXEC)
_llseek(5, 0, [0], SEEK_CUR)            = 0
fstat64(5, {st_mode=S_IFREG|0644, st_size=1106, ...}) = 0
mmap2(NULL, 1106, PROT_READ, MAP_SHARED, 5, 0) = 0xb78d4000
_llseek(5, 1106, [1106], SEEK_SET)      = 0
munmap(0xb78d4000, 1106)                = 0
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
socket(PF_FILE, SOCK_STREAM|SOCK_CLOEXEC|SOCK_NONBLOCK, 0) = 5
connect(5, {sa_family=AF_FILE, path="/var/run/nscd/socket"}, 110) = -1 ENOENT (No such file or directory)
close(5)                                = 0
open("/etc/group", O_RDONLY|O_CLOEXEC)  = 5
_llseek(5, 0, [0], SEEK_CUR)            = 0
fstat64(5, {st_mode=S_IFREG|0644, st_size=743, ...}) = 0
mmap2(NULL, 743, PROT_READ, MAP_SHARED, 5, 0) = 0xb78d4000
_llseek(5, 743, [743], SEEK_SET)        = 0
munmap(0xb78d4000, 743)                 = 0
close(5)                                = 0
_llseek(4, 45601, [45601], SEEK_SET)    = 0
fstat64(4, {st_mode=S_IFREG|0744, st_size=45601, ...}) = 0
close(4)                                = 0
write(3, "mnt/files_frigg.sh\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 10240) = 10240
clock_gettime(CLOCK_REALTIME, {1319467468, 412620448}) = 0
clock_gettime(CLOCK_REALTIME, {1319467468, 412664067}) = 0
close(3)                                = 0
close(1)                                = 0
close(2)                                = 0
exit_group(0)                           = ?
glandium commented 12 years ago

Can you try this patch:

diff --git a/libvmfs/vmfs_inode.c b/libvmfs/vmfs_inode.c index 664adda..ffac961 100644 --- a/libvmfs/vmfs_inode.c +++ b/libvmfs/vmfs_inode.c @@ -772,6 +772,8 @@ int vmfs_inode_stat(const vmfs_inode_t inode,struct stat buf) buf->st_atime = inode->atime; buf->st_mtime = inode->mtime; buf->st_ctime = inode->ctime;

I'd expect it to fix du, and maybe to fix tar at least in the .sh file case. Please tell me if that also fixes the problem with the vmdk file.

gabbe commented 12 years ago

Much better! Tar for the .sh file works perfectly, and tar for the biggest file that can fit on my local partition also works. I will have to get back to you on the biggest files since they need to be transferred over network first.

There is still a bit of a problem though. "du" now reports the blocks used by a file and not the file size.

gabbe commented 12 years ago 
root@ubuntu:/mnt# du *
7515    Braas R21
7676    Eskil2 R21 MS
15206   Eskil R21
1       files_frigg.sh
1       ftpput_frigg.sh
20484   ORA_STBRA
37937   ORA STCHA
37937   ORA_STGOT
2674    OraXP
97777   R19 Ora 11g
1       R19 Ora 11gr19ora11g.vmdk
14829   R19 XP SEE
16502   R19 XP x64
15294   R19 XP x64 JE
128672  R21ORA
19996   R21 W7 x64 Client 1
18373   R21 W7 x64 JE
15864   R21 XP x64
159495  Sebran014
32805   Seeskn157
45253   Sees R19 11g
17456   STBRA
110009  STGOT
glandium commented 12 years ago

That's not very surprising. I'm actually surprised it doesn't do that anymore on other file systems. Maybe it has a special rule for 4KB blocks. Anyways, du -h and du -k should work as expected.

gabbe commented 12 years ago

It's not a big thing, but I couldn't help to run some more tests:

gabbe commented 12 years ago 
root@ubuntu:/mnt# du files_frigg.sh 
1       files_frigg.sh
root@ubuntu:/mnt# du -h files_frigg.sh 
512     files_frigg.sh
root@ubuntu:/mnt# du -k files_frigg.sh 
1       files_frigg.sh
root@ubuntu:/mnt# du -b files_frigg.sh 
45601   files_frigg.sh
root@ubuntu:/mnt# du --block-size=1 files_frigg.sh 
512     files_frigg.sh
root@ubuntu:/mnt# du --block-size=4096 files_frigg.sh 
1       files_frigg.sh
root@ubuntu:/mnt# du --apparent-size files_frigg.sh 
45      files_frigg.sh
root@ubuntu:/mnt# du --apparent-size -h files_frigg.sh 
45K     files_frigg.sh
root@ubuntu:/mnt# du /tmp/files_frigg.sh 
48      /tmp/files_frigg.sh
root@ubuntu:/mnt# du -h /tmp/files_frigg.sh 
48K     /tmp/files_frigg.sh
glandium commented 12 years ago

This should be fixed with commit 7c33aa6

gabbe commented 12 years ago

Indeed it is now fixed. Great job, and many thanks for your efforts!