problem on Linux Mint: copy to hubic folder creates a folder_segment

[D4void]

Hello

I am using hubicfuse on my Fedora since the beginning without any problem. Thanks for this, it 's just a great job. I recently tested a backup script (which is based on hubicfuse) with another distrib; Linux Mint 17.1 (kernel 3.13.0-37) I have problem here. I tested with the last commit and with hubicfuse v2.0 The same things happen.

If I copy a file to my hubic mount dir , ie: ~/Hubic cp testfile.zip Hubic/default/Backuptest/test for example I have a new directory created: Hubic/default/Backuptest/test_segments If I go in it , I see my file copied ok. Going to Hubic/default/Backuptest/test, I see my file too.

I don't have this problem on my fedora. I don't know what can do this ? Maybe a distro library not in the same version ? Something I didn't set correctly but I don't see what ? Or could it be a bug ?

Thanks for your help

[D4void]

A little precision after making another test: when I do a ls of my mount dir, I don't see this created folder. I see it on the web gui and with the hubic android app too.

[TurboGit]

A bug probably, but I suppose not in hubicfuse but maybe in a dependent lib not working properly. I would check the version of the different package/libs used by hubicfuse.

[maki-chan]

<folder>_segments are folders that hold segments of huge files that are uploaded in several splitted parts to hubic (default: files over 2G are splitted into 1G files)

[D4void]

ok but I copied just a few kb for my test.

[D4void]

I created a VM with a debian wheezy. I compiled the last git version and I have a folder _segment too ... File is 90 bytes. Mint is based on debian and ubuntu. I must not be the only one using Debian with hubifuse. What's wrong ?

[TurboGit]

Nothing wrong. I also have a _segment directory. It is empty. Looks fine to me and I have never bothered with it.

[D4void]

you mean the default_segments at the Hubic file root ? I have one too and yes , empty.

But do you have others one created when you copy a file in a folder (like described in my first post)? In that case, this new segment folder created (and that I don't wan't to see in my files) is not empty and is containing the same file. If I delete this directory and I go in the target folder, I see the file I copied but it is empty. (because I deleted the segment folder)

And i don't have this with my fedora.

[TurboGit]

No I don't have segment files as described in your first post. Just the top level one:

$ cp hh.zip /mnt/hubic/default/ $ ls -al /mnt/hubic/default/ total 4 drwxr-xr-x 2 obry obry 0 May 4 17:28 . drwxr-xr-x 2 obry obry 0 Jan 1 1970 .. drwxr-xr-x 2 obry obry 0 Nov 14 2012 Documents -rw-rw-rw- 1 obry obry 83 May 4 17:29 hh.zip drwxr-xr-x 2 obry obry 0 Nov 17 2012 Images drwxr-xr-x 2 obry obry 0 Nov 15 2012 Music drwxr-xr-x 2 obry obry 0 Nov 14 2012 .ovh drwxr-xr-x 2 obry obry 0 Nov 15 2012 scripts drwxr-xr-x 2 obry obry 0 Nov 14 2012 Videos

And I'm on GNU/Debian sid (very close to Jessie released last week).

[D4void]

If I copy in default, no problem for me too because default_segments already exist I think. Can you try to mkdir /mnt/hubic/default/Test and cp hh.zip /mnt/hubic/default/Test to see if you have a Test_segments created in /mnt/hubic/default/

I am really interested to see if it's coming from me or not. Thank you

I mounted Hubic in my home dir and not with root. Maybe it could be linked.

[TurboGit]

Sure, no segment created:

$ mkdir /mnt/hubic/default/Test $ cp hh.zip /mnt/hubic/default/Test/ $ ls -al /mnt/hubic/default/ total 0 drwxr-xr-x 2 obry obry 0 May 4 17:56 . drwxr-xr-x 2 obry obry 0 Jan 1 1970 .. drwxr-xr-x 2 obry obry 0 Nov 14 2012 Documents drwxr-xr-x 2 obry obry 0 Nov 17 2012 Images drwxr-xr-x 2 obry obry 0 Nov 15 2012 Music drwxr-xr-x 2 obry obry 0 Nov 14 2012 .ovh drwxr-xr-x 2 obry obry 0 Nov 15 2012 scripts drwxr-xr-x 2 obry obry 0 May 4 17:56 Test drwxr-xr-x 2 obry obry 0 Nov 14 2012 Videos

BTW, I'm using hubicfuse compiled from Git master.

[D4void]

Thank you. And can you check with your web browser or Android app ? Because I don't see it too with ls on my mount dir.

[TurboGit]

I don't have a single _segments dir visible from my HubiC Android app. I don't use the Web interface.

[D4void]

ok thx. So it's coming from me... I don't see what it can be. Same things with Mint and Debian Wheezy (with git master compiled too) I tried with package v2.0 on Mint too. I don't think virtual machine can be the reason.

strange. I will check compilation.

edit:

On Mint: $ hubicfuse -V FUSE library version: 2.9.2 fusermount version: 2.9.2 using FUSE kernel interface version 7.19 $ uname -a Linux neutrinos 3.13.0-37-generic #64-Ubuntu SMP Mon Sep 22 21:30:01 UTC 2014 i686 i686 i686 GNU/Linux

On my Fedora: $ hubicfuse -V FUSE library version: 2.9.3 fusermount version: 2.9.3 using FUSE kernel interface version 7.19 $ uname -a Linux Pulsar 3.19.5-200.fc21.x86_64 #1 SMP Mon Apr 20 19:51:56 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

[D4void]

This afternoon, I made some trace with -d option on linux mint and fedora. I've done the same things (and I tested with options direct_io and big_writes, same things) : hubicfuse Hubic/ -d -o noauto_cache,sync_read,direct_io,big_writes mkdir Hubic/default/Test cp dbtest.log Hubic/default/Test

We can see difference. On Mint, the directory Test_segments is created by a PUT. I don't know fuse at all but I see http. I don't know if close connection code could tell something. My only clue is Fuse library... I will try tomorrow if I can "apt-get" the 2.9.3

@Turbogit I would like to know which version you have ? :) I you tell me 2.9.2 I'm lost :p

*\ MINT : LOOKUP /default/Test/dbtest.log getattr /default/Test/dbtest.log unique: 193, error: -2 (No such file or directory), outsize: 16 unique: 194, opcode: CREATE (35), nodeid: 14, insize: 67, pid: 3928 create flags: 0x80c1 /default/Test/dbtest.log 0100644 umask=0022

GET /v1/AUTH_13543ec620463596fd27d4578b6e31db/default?format=xml&delimiter=/&prefix=Test_segments/ HTTP/1.1 < HTTP/1.1 200 OK ....

• Closing connection 20 .... PUT /v1/AUTH_13543ec620463596fd27d4578b6e31db/default/Test_segments HTTP/1.1

*\ FEDORA :

LOOKUP /default/Test/dbtest.log getattr /default/Test/dbtest.log unique: 8, error: -2 (No such file or directory), outsize: 16 unique: 9, opcode: CREATE (35), nodeid: 3, insize: 67, pid: 26057 create flags: 0x80c1 /default/Test/dbtest.log 0100640 umask=0027

GET /v1/AUTH_13543ec620463596fd27d4578b6e31db/default?format=xml&delimiter=/&prefix=Test_segments/ HTTP/1.1 < HTTP/1.1 200 OK ....

• Closing connection 5 .... PUT /v1/AUTH_13543ec620463596fd27d4578b6e31db/default/Test/dbtest.log HTTP/1.1

[TurboGit]

On my GNU/Debian box I have 2.9.3:

$ apt-cache policy libfuse2 libfuse2: Installed: 2.9.3-15+b1 Candidate: 2.9.3-15+b1

So you are not completely lost yet :)

[TurboGit]

BTW, do you have set segment_above in your .hubicfuse? If so what is the value? I see in the code that the segment are created only of above the given limit. Frankly, I don't see why it is created on your side if this is not set...

[D4void]

Hello No I didn't set this in my .hubicfuse I tested with libfuse 2.9.3 and I had the same problem. So I really don't understand what happen. I even tested with debian Jessie. Same thing. I am in holyday this week so I will continue next week.

I Le 7 mai 2015 23:24, "Pascal Obry" notifications@github.com a écrit :

BTW, do you have set segment_above in your .hubicfuse? If so what is the value? I see in the code that the segment are created only of above the given limit. Frankly, I don't see why it is created on your side if this is not set...

— Reply to this email directly or view it on GitHub https://github.com/TurboGit/hubicfuse/issues/64#issuecomment-100023528.

[TurboGit]

I'm using GNU/Debian testing we it was Jessie for some time on my machine. No issue there. Something really fishy...

[D4void]

I tested again at home on a physical PC with a fresh Mint 17.1 installed and updated and this morning at work on a VM with Debian Jessie. I removed all dependancies and reinstalled them

On the Debian: apt-get remove curl libfuse-dev libcurl4-openssl-dev libxml2-dev libssl-dev libjson-c-dev libmagic-dev apt-get remove libpcre3-dev libpcrecpp0 libselinux1-dev libsepol1-dev libssl-doc zlib1g-dev apt-get remove libcurl3

apt-get install curl libfuse-dev libxml2-dev libjson-c-dev libmagic-dev apt-get install libcurl4-openssl-dev apt-get install libssl-dev

I compiled the source again. By the way, I don't know why I have these "no" with ./configure because these files are provided by libfuse and libxml2 and they are present : ... checking fuse.h usability... no checking fuse.h presence... no ... checking libxml/tree.h usability... no checking libxml/tree.h presence... no

make is "ok" , no errors.

And always the same problem :( On the debian and on Mint

I tried to generate again the token for .hubicfuse, no change

I added segment_size=1073741824 and segment_above=2147483648 (the default values) in .hubicfuse, no change

I thought it could be my hubic account ? I created a new one, used hubic_token, no change, same problem.

No I am stuck. I really don't see! I am crazy or there is something I don't do correctly ?

Can we check lib version you have ?

apt-cache policy libfuse-dev libcurl4-openssl-dev libxml2-dev libjson-c-dev libmagic-dev libssl-dev libfuse-dev: Installé : 2.9.3-15+b1 libcurl4-openssl-dev: Installé : 7.38.0-4+deb8u2 libxml2-dev: Installé : 2.9.1+dfsg1-5 libjson-c-dev: Installé : 0.11-4 libmagic-dev: Installé : 1:5.22+15-2 libssl-dev: Installé : 1.0.1k-3

3.16.0-4-686-pae #1 SMP Debian 3.16.7-ckt9-3~deb8u1 (2015-04-24) i686 GNU/Linux

[TurboGit]

My versions:

As I'm using unstable GNU/Debian branch my versions have probably moved recently, here it is:

$ apt-cache policy libfuse-dev libcurl4-openssl-dev libxml2-dev libjson-c-dev libmagic-dev libssl-dev libfuse-dev: Installed: 2.9.3-15+b1 libcurl4-openssl-dev: Installed: 7.42.1-2 libxml2-dev: Installed: 2.9.2+dfsg1-3 libjson-c-dev: Installed: 0.11-4 libmagic-dev: Installed: 1:5.22+15-2 libssl-dev: Installed: 1.0.2a-1

I'm on a 64bits box:

4.0.0-1-amd64 #1 SMP Debian 4.0.2-1 (2015-05-11) x86_64 GNU/Linux

Could it be the issue here? An int going above the limit in the implementation and I don't do that because I'm using a 64bit box.

Worth looking at this... Maybe adding some debug trace in hubicfuse for the segment_* variables?

[D4void]

ok, I installed all the same lib from jessie testing. Same problem.

I quickly installed a VM with Mint 64bits : and no problem !

So there is something when compiling hubicfuse on a 32 bits.

[TurboGit]

Ok, probably just an integer overflow as I said in my previous message. Now which one? Can you debug this on your 32bit box?

[D4void]

euh well i don't really see how to debug this :( I didn't make C since school :)

IMO it must somewhere when there is a check if the directory exist , if we look the debug i put at the beginning (with No such file or directory and the "put" method that creates the _segment)

[LoloVTS]

Hello,

I'm facing same issue on a 32bits system. This is the segment_above default value which is overlapping the long variable. A workaround is to set segment_above to 2147483647

[TurboGit]

Thanks, that's indeed a good candidate for this issue. Just fixed on master. Can you test again on a 32bit system? Thanks.

[D4void]

Great thanks for help ! Tested and it works like a charm :) For my culture, what is segment_above ?

[maki-chan]

segment_above tells the filesystem the size of a file in bytes at which it starts uploading the file in parts rather than in a whole. So if you have segment_above at 2147483647, every file with a size greater than that will be uploaded in parts with a size of segment_size bytes. On 64bit systems, the value can be higher and so I for example use it to upload files in one part with a size up to "4GB -1 byte"

[TurboGit]

Good to hear it is working on your side! Closing this issue then.

[TurboGit]

checking fuse.h usability... no checking fuse.h presence... no ... checking libxml/tree.h usability... no checking libxml/tree.h presence... no

Should be fixed now.