search

ovh / svfs

The Swift Virtual File System
Other
374 stars 54 forks source link

Unable to create a .tar archive bigger than 256MB #145

Open fungiboletus opened 7 years ago

fungiboletus commented 7 years ago

Hei,

I have a folder with 300 000 files and Hubic recommends to have 50 000 files maximum. So I want to create a .tar archive directly in the mount point (the files and the archive don't fit on my system).

It doesn't seem to work 😢

Context

Steps to reproduce this issue :

  1. Have a folder bigger than 256MB
  2. tar --verbose --create --file /svfs_mount/archive.tar /folder
  3. Go to the beach and come back when tar has finished. 🏖

Results you expected :

The .tar archive is created and contains all the files.

Results you observed :

Tar exits with return code 2 :

tar: /svfs_mount/archive.tar: Cannot write: Input/output error
tar: Error is not recoverable: exiting now

The archive.tar file exists and is 256MB (268431360).

Confushion commented 7 years ago

I've got similar issues (on OVH/hubic).

It seems that the maximum segment size (default mount option segment_size=256) doesn't work as expected. Any file larger than the maximum segment size is not properly split into multiple segments. If you mount with option segment_size=10 you will observe the error a lot earlier at an upload of 10Mb.

This currently limits the maximum file size to be about 5Gb (the maximum size of a segment), when mounted with option segment_size=5120.

As I have a lot of files > 5Gb that I'd like to store on Hubic, this is a blocker for me...

pisto commented 6 years ago

Getting the same issue, also with the suggested rsync method. Basically all files greater that the segmententation limit are broken. Endpoint is hubic.

xlucas commented 6 years ago

Can you provide a debug trace ? I suspect this is a duplicate of #100.

pisto commented 6 years ago

How do I generate it?

xlucas commented 6 years ago

Use the debug mount option documented here.

pisto commented 6 years ago 
DEBU[2017-09-22T13:58:29+02:00] <- Lookup [ID=0xa2 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv"  source=fuse
DEBU[2017-09-22T13:58:29+02:00] -> [ID=0xa2] Lookup error=ENOENT     source=fuse
DEBU[2017-09-22T13:58:29+02:00] <- Create [ID=0xa3 Node=0x2 Uid=1000 Gid=1000 Pid=22793] "rick.mkv" fl=OpenWriteOnly+OpenCreate mode=-rw------- umask=--------w-  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa3] Create {0x3 gen=0 valid=1m0s attr={valid=1m0s ino=12768038556851320042 size=0 mode=-rwx------}} {0x1 fl=OpenDirectIO+OpenNonSeekable}  source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @0 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa4] Write 131056            source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @131056 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa5] Write 131072            source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @262128 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa6] Write 16                source=fuse
DEBU[2017-09-22T13:58:30+02:00] <- Write [ID=0xa7 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @262144 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:58:30+02:00] -> [ID=0xa7] Write 131056            source=fuse
[...]
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9c Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @267780080 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9c] Write 131072           source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9d Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @267911152 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9d] Write 16               source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9e Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @267911168 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:20+02:00] -> [ID=0xc9e] Write 131056           source=fuse
DEBU[2017-09-22T13:59:20+02:00] <- Write [ID=0xc9f Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268042224 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xc9f] Write 131072           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca0 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268173296 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca0] Write 16               source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca1 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268173312 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca1] Write 131056           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca2 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131072 @268304368 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca2] Write 131072           source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca3 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 16 @268435440 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:21+02:00] -> [ID=0xca3] Write 16               source=fuse
DEBU[2017-09-22T13:59:21+02:00] <- Write [ID=0xca4 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:39+02:00] -> [ID=0xca4] Write error=EIO: Timeout when reading or writing data  source=fuse
DEBU[2017-09-22T13:59:39+02:00] <- Write [ID=0xca5 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 131056 @268435456 fl=WriteLockOwner lock=6900770311377782222 ffl=OpenWriteOnly  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca5] Write error=EIO: Timeout when reading or writing data  source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Flush [ID=0xca6 Node=0x3 Uid=1000 Gid=1000 Pid=22793] 0x1 fl=0x0 lk=0x5fc476094e3c05ce  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca6] Flush                  source=fuse
DEBU[2017-09-22T13:59:54+02:00] <- Release [ID=0xca7 Node=0x3 Uid=0 Gid=0 Pid=0] 0x1 fl=OpenWriteOnly rfl=0 owner=0x0  source=fuse
DEBU[2017-09-22T13:59:54+02:00] -> [ID=0xca7] Release                source=fuse
xlucas commented 6 years ago

Try setting connect_timeout=5m like suggested in #100

fungiboletus commented 6 years ago

Hei. I will try that, but doesn't that mean the underlying system has a slowness issue ?