Hard links fail if the host os uses multiple partitions

[stephen304]

Since hard links can't go across partitions. I think this could be solved by tarring to symbolic links instead?

I can't seem to solve this with the current tar archive because there doesn't seem to be an option to convert hard links to symbolic links when untarring.

[fsquillace]

Hi, Thanks for detecting this. Not sure if I got properly the issue though. Can you provide an example? I can see there are correctly hard links on JuJu image. For instance:

[feel@myarch ~]$ ls -l /usr/share/terminfo/t/ti735
-rw-r--r-- 5 feel users 396 Dec  6 17:00 /usr/share/terminfo/t/ti735

[feel@myarch ~]$ find /{usr,lib} -type f -links +1 | wc -l
2632

[stephen304]

Basically, the home directory is a separate partition which is mounted, and that means the root partition is separate as well. Since juju installs to the home directory, the hard links are trying to link from the home partition to the root partition. This makes tar fail. I think switching the hard links for symbolic links will solve it.

for example, home is /home/stephen/ Juju untars /home/stephen/.juju/etc/somefile which hard links to /etc/somefile

/etc/somefile is on the root partition and ~/.juju/etc/somefile is on the user partition, so the hard link fails when untarring.

[fsquillace]

This looks strange since I also have a dedicated partition for the home directory and I cannot reproduce the error. During the setup process (untar of the image to the ~/.juju) there should not be anything that links between files in JuJu environment and files in the host OS. They are completely decoupled systems.

Maybe another reason why is it failing? Can you provide the output so I can try to reproduce the issue.

[stephen304]

Heres the output of typing juju to install. Despite all the errors, it seems to work anyway, but it's a lot of errors: http://pastebin.com/0CCrn1ZL

I'm fairly certain hard links can't go across partitions - everything I read says so. Can you create a hard link with ln ~/auto.misc /etc/auto.misc?

When I try that command, it says

ln: creating hard link `/etc/auto.misc' => `/afs/unity.ncsu.edu/users/s/slsmit12/auto.misc': Invalid cross-device link

Here's the list of mounted things:

eos$ mount
/dev/mapper/Volume00-root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
/dev/mapper/Volume00-tmp on /tmp type ext4 (rw)
/dev/mapper/Volume00-var on /var type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
AFS on /afs type afs (rw)
/afs/eos.ncsu.edu/dist/modelsim10.0c on /ncsu/modelsim10.0c type none (rw,bind)

The AFS one (second from last) is where the home directories are.

[fsquillace]

Thanks Stephen for that.

Yeah, it looks there is an issue with hard links across partitions. From this old thread it seems that sometimes the tar version matters too: http://lists.gnu.org/archive/html/bug-gnu-utils/2001-10/msg00123.html

What is your tar and linux kernel version? Can you run mount and df -h to see if there are mounted partitions inside the JuJu directory?

[stephen304]

The output of mount is in the last comment, and here is df -h. It doesn't look like anything is mounted inside juju.

eos$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/Volume00-root
                       12G  8.0G  3.2G  72% /
tmpfs                 435M  4.0K  435M   1% /dev/shm
/dev/sda1             488M   55M  408M  12% /boot
/dev/mapper/Volume00-tmp
                      7.3G   22M  6.9G   1% /tmp
/dev/mapper/Volume00-var
                      4.8G  1.1G  3.5G  25% /var
AFS                   8.6G     0  8.6G   0% /afs

Tar:

eos$ tar --version
tar (GNU tar) 1.23
Copyright (C) 2010 Free Software Foundation, Inc.

Kernel:

Linux engr-ras-101.eos.ncsu.edu 2.6.32-504.8.1.el6.x86_64 #1 SMP Fri Dec 19 12:09:25 EST 2014 x86_64 x86_64 x86_64 GNU/Linux

[fsquillace]

There should not be problem with that kernel version. It seems the problem is due to the tar command.

The characteristic of your environment are not too far from mine apart for the tar. My tar is 1.28 version and still I cannot see any issue like this.

Could you please install the most recent tar by any chance? Thanks a lot.

[stephen304]

Sorry I can't. It's my school's remote linux server - that's why I'm using juju.

[fsquillace]

If you have access from that host to the gcc compiler you may install it and point the PATH environment variable to the compiled executable.

The sources for a most recent tar are here: http://www.gnu.org/software/tar/

I will try to do the same with version 1.23 to see if i can replicate the issue.

[stephen304]

I compiled tar v1.28 and it doesn't change anything. I still get the invalid cross link device error.

[fsquillace]

Thanks Stephen, I have also tried with older tar (1.15.1) and JuJu still works well for me. Apparentely, it seems a strange edge case in which the tar extraction does not work with hard links.

I will try to investigate more later on.

[aneeshusa]

My school also uses AFS, and I'm also get the hard link errors, same after compiling tarv1.28.

Interestingly, JuJu still seems to work ok - running JuJu again after the first time just works as normal and drops into the shell, although pacman will complain when updating glibc or tzdata, for instance. I'll keep investigating, but I haven't run into anything fatal yet.